Reboots??

[sheldoom]

I have a customer with an IP500v2, that was running fine for years, then about 3 months ago started randomly rebooting. We worked to narrow down a cause and could not. We upgraded to the latest software for their release and that didn't help. So, finally we replaced the Control Unit and upgraded to Rel. 9.0. Well, this fixed the reboots, sort of. What is happening now is the system is showing random reboots in SA and in Monitor, but the customer is not loosing any calls. Their Call History clears, but that is all they are experiencing. I have asked the customer to verify that there is not an incorrect devise plugged into the system and waiting to hear back on that one.

********** contact lost with 172.17.5.5 at 20:34:03 23/11/2013 - reselect = 3 **********
******************************************************************

********** SysMonitor v9.0.0.0 build 829 **********

********** contact made with 172.17.5.5 at 20:35:12 23/11/2013 **********

********** System (172.17.5.5) has been up and running for 41secs(41677mS) **********

********** Warning: TEXT File Logging selected **********


********** Warning: TEXT Logging to C:\Program Files\Avaya\IP Office\Monitor\\TMP4.$$$ File: STARTED on 23/11/2013 20:35:12 **********
41678mS PRN: Monitor Started IP=172.17.5.5 IP 500 V2 9.0.0.0 build 829
(IP Office: Supports Unicode, System Locale is default)
41678mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, VCOMP=32, MDM=0, WAN=0, MODU=5 LANM=0 CkSRC=0 VMAIL=0(VER=0 TYP=1) 1-X=0 CALLS=0(TOT=0)

SA showed 3 reboots for this day, which was a Saturday and the were virtually no calls this day.



"All men are fools. Some the wise, fool others. Others the foolish fool themselves. While a rare few fool both others and themselves. These are the rulers of men" from 'The Darkness That Comes Before' by R Scott Bakker

 

[tlpeter]

Make a monitor trace of the reboot and part it here.


BAZINGA!

I'm not insane, my mother had me tested!

 

[amriddle01]

Sounds like possible config corruption, depending on complexity it may be worth redoing

 

[MankyWarrior]

I have seen this before where someone has plugged a network device (e.g. nic card on a PC) into a port that goes straight back to an extension port on the PABX. An impossible thing to locate easily.

What about power issues? is there a UPS in place?, anything else the PABX is plugged into for power also having similar problems? Earthing on the cabinet / PABX?

 

[Westi]

I had an issue about a month ago and the reason was that 1 user logged into his mailbox (embedded) and he had 245 (or another ridiculous number) of new messages and whenever he logged in the system rebooted. It didn't dawn on that person to tell anyone that this happens EVERY time he logs in and we had to do traces and jump through hoops to find it.
It was the embedded vm that had over 1500 messages in total on it because people kept saving them rather than deleting it so I had to wipe it and had them start from scratch.
Took me a total of 3 weeks until we found the problem as it did not happen every day (the guy was on the road a lot) but then he did it 8 times in a day (AHHHHHHHHH)
The good thing is that the Monitor trace shows clearly the reboot and all you need to do is scroll up from there and see what has happened. It brought a funky error that Avaya could also not decipher btw and looking at the Monitor log was the best way to find it

Joe W.

FHandw, ACSS (SME), ACIS (SME)


“This is the end of the world, make sure to buy your T-shirt before it is too late"
Original expression of my daughter

 

[sheldoom]

Here's Monitor traces from one of the reboots that is fairly typical. No call activity at all. Customer is using Voicemail Pro. We are in the process of verifying all devises. The system in about

********** SysMonitor v9.0.0.0 build 829 [connected to 172.17.5.5 ] **********
13:27:33 259008477mS PRN: Monitor Status IP 500 V2 9.0.0.0 build 829
13:27:33 259008477mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, VCOMP=32, MDM=0, WAN=0, MODU=5 LANM=0 CkSRC=9 VMAIL=1(VER=3 TYP=1) 1-X=0 CALLS=0(TOT=4647)
13:27:35 259010054mS RES: Sat 23/11/2013 13:27:35 FreeMem=45794032 44627324(3) CachedMem=1166708 CMMsg=9(10) Buff=5200 1364 998 7422 4 Links=5883 BTree=11594 CPU=01.75% CPUStats=1/4/4896/13733/14338/0/0 MCR=0 MCW=0
13:27:35 259010054mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=65 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=20(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:27:35 259010054mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)
13:28:04 259038566mS RES: Sat 23/11/2013 13:28:04 FreeMem=45794032 44627324(3) CachedMem=1166708 CMMsg=9(10) Buff=5200 1364 999 7422 4 Links=5883 BTree=11594 CPU=04.77% CPUStats=21/24/4896/13326/14338/0/1 MCR=0 MCW=0
13:28:04 259038566mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=1 CMRTEngine=0 ExRTEngine=0 Timer=67 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=20(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:28:04 259038567mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)
13:28:33 259067074mS RES: Sat 23/11/2013 13:28:32 FreeMem=45794032 44627324(3) CachedMem=1166708 CMMsg=9(10) Buff=5200 1364 998 7422 4 Links=5882 BTree=11594 CPU=02.29% CPUStats=1/4/4896/13734/14338/0/0 MCR=0 MCW=0
13:28:33 259067074mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=66 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=20(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:28:33 259067074mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)

********** SysMonitor v9.0.0.0 build 829 [connected to 172.17.5,5 ] **********
13:28:34 259068472mS PRN: Monitor Status IP 500 V2 9.0.0.0 build 829
13:28:34 259068473mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, VCOMP=32, MDM=0, WAN=0, MODU=5 LANM=0 CkSRC=9 VMAIL=1(VER=3 TYP=1) 1-X=0 CALLS=0(TOT=4647)
13:29:11 259105610mS RES: Sat 23/11/2013 13:29:11 FreeMem=45791440 44627324(3) CachedMem=1164116 CMMsg=9(10) Buff=5200 1364 1000 7423 5 Links=5888 BTree=11594 CPU=03.62% CPUStats=1/5/4896/13525/14338/0/1 MCR=0 MCW=0
13:29:11 259105610mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=68 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=21(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:29:11 259105611mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)
13:29:18 259112614mS RES: Sat 23/11/2013 13:29:18 FreeMem=45789952 44627324(3) CachedMem=1162628 CMMsg=9(10) Buff=5200 1364 999 7422 4 Links=5886 BTree=11594 CPU=05.01% CPUStats=21/26/4896/13295/14338/0/0 MCR=0 MCW=0
13:29:18 259112614mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=1 CMRTEngine=0 ExRTEngine=0 Timer=67 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=21(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:29:18 259112614mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)
13:29:30 259124028mS RES: Sat 23/11/2013 13:29:29 FreeMem=45794032 44627324(3) CachedMem=1166708 CMMsg=9(10) Buff=5200 1364 998 7422 4 Links=5884 BTree=11594 CPU=02.17% CPUStats=1/4/4896/13625/14338/0/0 MCR=0 MCW=0
13:29:30 259124028mS RES2: IP 500 V2 9.0.0.0 build 829 Tasks=83 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=66 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=4 SSA=1 TCP=20(TLS=2) TAPI=2 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
13:29:30 259124028mS RES4: XML MemObjs=56 PoolMem=2097152(1) FreeMem=2081932(1)

********** SysMonitor v9.0.0.0 build 829 [connected to 172.17.5.5 ] **********
13:29:34 259128469mS PRN: Monitor Status IP 500 V2 9.0.0.0 build 829
13:29:34 259128470mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, VCOMP=32, MDM=0, WAN=0, MODU=5 LANM=0 CkSRC=9 VMAIL=1(VER=3 TYP=1) 1-X=0 CALLS=0(TOT=4647)

********** contact lost with 172.17.183.210 at 13:29:54 23/11/2013 - reselect = 1 **********
******************************************************************

********** SysMonitor v9.0.0.0 build 829 **********

********** contact made with 172.17.5.5 at 13:31:00 23/11/2013 **********

********** System (172.17.5.5) has been up and running for 43secs(43189mS) **********

********** Warning: TEXT File Logging selected **********


********** Warning: TEXT Logging to C:\Program Files\Avaya\IP Office\Monitor\\TMP4.$$$ File: STARTED on 23/11/2013 13:31:00 **********
43189mS PRN: Monitor Started IP=172.17.5.5 IP 500 V2 9.0.0.0 build 829
(IP Office: Supports Unicode, System Locale is default)
43190mS PRN: LAW=U PRI=1, BRI=0, ALOG=4, VCOMP=32, MDM=0, WAN=0, MODU=5 LANM=0 CkSRC=0 VMAIL=0(VER=0 TYP=1) 1-X=0 CALLS=0(TOT=0)
13:31:00 43636mS CMExtnEvt: AADKINS:71562 ExtnFault now 0
13:31:00 43644mS CMExtnEvt: 71562 ReportAttachment type=0
13:31:00 43648mS CMExtnTx: v=71562, p1=0

"All men are fools. Some the wise, fool others. Others the foolish fool themselves. While a rare few fool both others and themselves. These are the rulers of men" from 'The Darkness That Comes Before' by R Scott Bakker

 

[tlpeter]

This trace does not show the reboot.
It looks like it just fails.

I would try to replace it and see what happens.


BAZINGA!

I'm not insane, my mother had me tested!