IP Office 5.0.24 - VM Pro 5.0.28 - Abnormal Termination

[xterest]

Hey Guys,

I have a strange problem with an IP Office 500. About 3 months ago, the system started to reboot for no reason every few days. Looking at the logs, I saw the following error "Abnormal Termination" with a reference to a memory address. See attached ss.jpg.

When we investigated it, we noticed the UPS that the IP Office was connected to phone system had a bad battery, so we changed the UPS and brushed it off that the issue was power related, and the UPS was just power cycle the unit causing it to reboot. After we changed the UPS unit, they system was stable for about 2 weeks, until it started rebooting again. So the next step we took was to upgrade the firmware and VM pro to the latest and greatest 5.0.24 on the IP Office and 5.0.28 on VM Pro (without going 6+ and needed additional licenses.)

Asking a few friends in the IP office world, they told me that the issue might be related to a error in the interaction between the control unit and the VM pro box.

What we are currently planning on doing is replacing the whole VM Pro server later today and crossing our figures. The system has rebooted 3 times today so far, and I'm not still exactly sure what is causing it. Kind of working blindly here and hoping for the best.

Anyone have any similar issues to this, or any insight on how to troubleshoot an issue of this nature?

Any help is appreciated.

Below is a system monitor log showing the events leading up to a reboot.

http://www.alspaconsulting.com/IPO/IPOFFICE_Log.txt

http://www.alspaconsulting.com/IPO/ss.jpg

 

[amriddle01]

Only Avaya can make sense of TLB alarms and such errors, escalate it to them if you can, but they may just say you have a corrupted config and to try recreating after exporting/importing what you can

ACSS (SME)
APSS (SME)

http://www.ipoffice.tel

"I'm just off to Hartlepool to buy some exploding trousers

 

[amriddle01]

Just noticed this:

34830mS PRN: FalcLockedChecks FPGA_FIFO_CTRL: 1; FIFO_CTRL: 1
34830mS PRN: WARNING:
34830mS PRN: ****Reset Slip Store Falc 1 0 0

Do you have a PRI and if so what is the distance of cable run between the IPO and the NTE/providers termination point?

ACSS (SME)
APSS (SME)

http://www.ipoffice.tel

"I'm just off to Hartlepool to buy some exploding trousers

 

[hairlessupportmonkey]

did the call park issue turn in 5 or 6?

ACSS - SME

 

[amriddle01]

I didn't see a Park in the trace, wasn't that early R5?

ACSS (SME)
APSS (SME)

http://www.ipoffice.tel

"I'm just off to Hartlepool to buy some exploding trousers

 

[xterest]

There is a PRI on this system, and the cable is about 100 ft from IP Office. The telco left a biscuit / box 2 feet from the IP Office and the other side of that jack is on the other side of the office where the smart jack resides. Think its been like this for about 4 years.

 

[amriddle01]

I suspected a long cable run, that is causing slips in the PRI clocking. It may be totally unrelated to the issue, I just noticed it in the trace. I have not seen it cause a reboot before but there is a first time for everything and every release has it's own fun glitches

ACSS (SME)
APSS (SME)

http://www.ipoffice.tel

"I'm just off to Hartlepool to buy some exploding trousers

 

[tlpeter]

Sync problems will give reboot problems.
I have been there and done that
Put the PRI box next to the IPO and use a short cable.


BAZINGA!

I'm not insane, my mother had me tested!

 

[TheProvider]

If your PRI NTE is miles from the NTE, alway try to get it within 5 metres of the terminating equpment.

Get the line provider to move it (Official statement)

Slips are not good and 100ft your lucky to be working at all, I am suprised you are not getting poor quality calls and cut offs/

 

[hairlessupportmonkey]

or if they dont want to move it port it all over to SIP and ill do you a deal ;-)

ACSS - SME

 

[TomMills]

careful...

http://www.twitter.com/tem1985

ACSS-SME
fo shizzle ma nizzle.

 

[xterest]

So I replaced the hardware of the VM Pro Machine and retired the old one. Also went through the config and fixed a few warning related to some appearance keys not in order and cleaned up some user buttons on 2 users phones that pointed to extensions of users that don't exist anymore. So now there are no more warning in manager anymore.

I will open a ticket with the Telco to have the smart jack relocated into my server room and see what they say.

I have my fingers crossed that the reboots go away, but if they don't, I guess my next step is to rebuild the config ( about 40 users.. o joy..), and then possibly replace the IP Office 500 chassis?

Can this issue be caused by a bad module on the system? I really don't want to start swapping modules out one at a time.

Below is the modules on the system.

http://www.alspaconsulting.com/IPO/modules.jpg

Thanks for all the help so far.

 

[TomMills]

tbh 40 odd users can be rebuilt in an hour.

Just suck it and see. It might have been one of the errors causing it. who knows.

http://www.twitter.com/tem1985

ACSS-SME
fo shizzle ma nizzle.