IPOffice 500V2 7.029 rebooting 6

[makinconversation]

Hello, I am looking for anyone who has seen an issue of a reboot due to ABNORMAL TERMINATION. I turned sysmon on, so I have something to go on, and it looks as though it captured my problem. Now, I can't find any info on this. I am about to open a ticket with Avaya, but wanted to check here first. The last reading states "PRN: Pipe Send too big 583"... then it rebooted. It has done this 3 times in several weeks...middle of day, tv station. Any info would be great. Thanks

 

[CaptainAegis]

I have the same problem with a new IP 500V2. I had the system on 7.0.12 then 7.0.23 (with the patch) then on 7.0.27, and now finally on 8.0.16. We have two sites linked with SCN over a large bandwidth static VPN connection. Works Fine. We have all 9508 phones with firmware R37 on them. One Voicemail pro R8 at each location (no centralized voicemail. No Conflicts in the directories between the two sites, no looping conditions, or funny huntgroup or user names. Been through this with a fine tooth comb, and the bugger still randomly reboots, sometimes 3 times a day. Client is going nuts! Any input would be greatly appreciated. Tonight, I am gong to default the system, format and recreate the SD Card, and reprogram from scratch. I'll let you know how that works out. Thanks

 

[CaptainAegis]

By the Way, When it reboots, I get an Abnormal Termination error
<watchdog> addr=00000000 d=0 pc=f032b060

Very Concerned, been selling AVAYA for 15+ years and this last 7.0 series of sofware has been really substandard at best.

 

[SupportDude]

I had the same issue at one site that was running 7.0(23).
I opened a case with Avaya. They already knew about the issue (and IMHO, 7.0(23) was crap and caused a lot of problems!).
7.0(28) was released later on the same day I raised a case. We installed it that afternoon and it resolved the problem.
If it didn't ... we were downgrading to pre 7.0(23).

Good Luck!

-SD-

http://www.mysupportguy.net

 

[CaptainAegis]

What a nightmare this whole journey has been, and add to that the firmware issues for the 9508 phones.
The 7.0.23 rev, without patch, messed up the speaker phone on all the sets. So we put in the patch, and it fixed the speaker phones, but the reboot issue was still there.
Then, wooohoo, 7.0.27 was on the support site! (never saw a 7.0.28 that you refer to SupportDude) so I upgraded the systems to that, being 7.0.27, and it broke the 9508 Navigation keys, and the when dialing the phone, the digits response lagged, drove the client nuts. No reboots on this Rev, but the phones where about usless.
So up to R8 I go, and it corrected the Navigation Keys and Laggy dialing, but broke visual voicemail, so I upgraded VM Pro to Rev 8 as well. Everything seemed to be working wonderfully, but then, BAM! Reboots started up again.
What a mess! This is not promoting confidence in AVAYA for my client. I have been doing IP Office for over 10 years now and I have never seen such a mess.

 

[SupportDude]

I feel your pain!
I've started saving a copy of each update that is released, just in case I have to go backwards!

-SD-

http://www.mysupportguy.net

 

[makinconversation]

This all started after the 7.023 patch, that included the FW rev 36 (I think that's right). I may have to look into 8+. My client is going crazy as well. Thanks

 

[makinconversation]

Oh, by the way, the only reason we went up to 7.023 was for the 9500r36 patch. I downloaded both and installed at one time. We had a crazy noise out of the 9508 when on speaker. It fixed that, but now unexpected reboots...sigh

 

[CaptainAegis]

@Makingconverstation
Same thing here...man, it's like we are in a parallel universe.
I am at the site now 8pm EST, and I am doing a DTE default, formatting the SD card, recreate card, and manually rebuilding the config. Rev 8.0.16
I'm in for a long night, but I hope this does the trick. I'll let you all know what the results are.

 

[Bas1234]

@CaptainAegis <watchdog> addr=00000000 d=0 pc=f032b060

addr=00000000 = a provider problem not an IPOffice one!!!!

___________________________________________
It works! Now if only I could remember what I did...

Dain Bramaged (Avaya Search tool

http://tinyurl.com/bas1234

)
______________________________________

 

[makinconversation]

Mine has the same Watchdog string, but with f02dd29c at the end. What kind of provider issue?

 

[makinconversation]

I am being told that this abnormal termination issue is related to applications, or other memory items not completing to load. Let's say, you didn't finishing loading an application fully, or it terminated in the middle of loading it. I looked back at my alarms and it shows it tried to load a MOH file and it failed at about the same time the reboot took place. It could be a cause, or a result of the reboot...I will never know.

 

[tlpeter]

Why not?
remove the Moh file and see what happens.


BAZINGA!

I'm not insane, my mother had me tested!

 

[CaptainAegis]

Bas1234,
I have heard that the addr=00000000 = a provider problem.
I am curious about this, and how and why a Flakey T1 PRI would cause the system to REBOOT. (as oppsed to just dropping calls)
It's hard to convince the client that it is a Carrier/Provider problem when the whole IP Office shuts down. Dropped calls would be easier to explain that it is the carrier.
I am also hearing the repetitive statement "well our other system didn't have a problem with it" (it was an NEC 8000)
Kind of crappy that the AVAYA system is so sensitive to the T1 circuit.
What is out there is an XO communications Adtran Unit, and it is less then 5 feet away from the IP Office.
We made sure that the XO Adtran was plugged into the same UPS as the IP Office, and that both the IP Office and the XO Adtran share a common grounding wire, which is grounded to the Telco Copper Dermarcation Ground.
We even replaced the T1 Cross Over cable between the XO Adtran and the IP Office PRI Card.
We also replaced the PRI Card just in case it is a hardware issue with the card.
PRI Config is: NI2 B8ZS ESF 4 Digits 23>1 (pretty basic stuff)
CRC is ON CSU Operation is Off
I was on site till 5am last night, did a DTE Reset, Formatted and Rebuilt the SD card, programmed the whole DA*MN from scratch. (big system so it took a while.
So far...No Reboots....but it remains to be seen.
So, this is my LONG winded way of asking...If the T1 Providers Circuit can Crash the IP Office.... How, Why, and what can be done about it? What would I even tell the Provider to check?
Is it a Physical Layer Issue, or a Provisioning Problem?
I don't really expect anyone to have all the answers, but this will be a nice discussion to get the Brain Juices flowing for any other poor soul with a similar issue.

AVAYA....HELP, your Ruining my Business, my Client want their NEC Back.

 

[Bas1234]

@CaptainAegis, I don't know T1 only E1 (EuroISDN) there might be others here that might know what the problem is. I've seen a lot of providers doing strange things on the line, if you know NEC and could take a look at the T1 settings you might see a settings that is not right. That system could be over 10 years and they might tweaked the PBX so to make it work because the provider wouldn't change it's settings. Also cabling/grouding could cause this so double check that.

If you have a SysMon trace with the all ISDN options ticked and post it here we can have a look at it.

___________________________________________
It works! Now if only I could remember what I did...

Dain Bramaged (Avaya Search tool

http://tinyurl.com/bas1234

)
______________________________________

 

[CaptainAegis]

@Bas1234
Thanks for the suggestions. Will re-check all the wiring again. Also I'll call XO to get some insight on their provisioning.
The SysMon Trace is a great idea. I'll get that going.
So far, after my rebuild, NO REBOOTS - I'll keep my fingers crossed, but I'll start a Trace anyway just in case.
Thanks again.

@makingconversation
how are you comming along?

 

[makinconversation]

My customer had a standard MOH wav he provided. I replaced the default MOH file and rebooted. The "MOH file failed to load" alarm is gone...the sysmon tracing is still on, so we will see what happens. Mine has only booted 3 times in 5-6 weeks, so it will take longer to realize. Now, if I can get the dial by name directory to fully work, I will be better. Not all users are working. I deleted users and did a resync on the VMpro, nada... I need a maintenance window to backup-delete-restore, so I haven't done much with it yet.

 

[CaptainAegis]

@makingconversation
Not sure if you already know this, but the Dial By Name directory will not work until the user has actually recorded their name in the mailbox set up. That might be what is happening.

So far, after my complete teardown and rebuild, I have not had any reboots. Time will tell.

Have a great New Years.

 

[LJDirk]

Hi,

any news about this?
Have the same BUG (IPO 8.0.16, 9508, VMpro, CCR, One-X, Contect Store), but the last Number is 582 -> "pipe is too big 582"

I want a downgrade to 7.0.24, but when read this, i leave it!

Can someone tell me, what a stable release i can take for a CallCenter?
That´s my first IPO and than a bad choice...

 

[tlpeter]

Dirk, it is a knows issue.

CQ42185	Receiving a Pipe Send too big message just before the system restarts	7.0.12	SP_RELEASE_1Q12_7.0	3-Available	7.0.128405 
CQ42202	Pipe Send too big 589 restart	7.0.12	SP_RELEASE_1Q12_7.0	3-Available	7.0.128405

The next release should solve this problem.


BAZINGA!

I'm not insane, my mother had me tested!