Multiple IPO's down at same time

[NLPeterK]

Hi guys,

We just had 17 (!) IPO's go into an 'unknown exception' reboot at the same time. Both 500 V1 and V2 and all SIP trunks (different providers). Our distributor suggested this has to do with the STUN server they use. They indeed all use the same STUN server, except for one. Our demo system also went down but our office system didn't. Both use the same STUN server.

Thoughts?

 

[amriddle01]

Then it must be the STUN server or the SIP provider, what STUN do you use?

 

[NLPeterK]

We use mostly stunserver.org (132.177.123.13).

Funny thing is that not all systems that use that stun server went down. Specifically the ones running 8.0(42) remained up.

Here's the list so far (using stunserver.org unless otherwise specified):
All systems running 8.0(16) or (18) went down.
All systems running 7.0(5) went down.
One system running 7.0(277212) went down, the other did not.
One system running 7.0(27) went down, the other using 216.93.181.86 did not.
System running 6.1(504) did not.
Systems running 6.1(20) and 6.1(11019) went down.
System running 6.0(8) using 216.93.181.86 went down.
System running 6.0(14) using 69.90.168.13 did not, yet system running 6.0(18) using 69.90.168.13 did.
System running 5.0(22) using 69.90.168.13 went down.

I'm trying to find common denominators, but it's all across the board: V1's, V2's, VCM32/64, Combo cards, different stuns. Only thing that really stood out were the systems running 8.0(42). And the fact that all other 8.0 systems went down. Out 7 7.0's, 5 went down. Out of 10 6.0/6.1's , 6 went down. Out of 4 5.0's 3 went down.

I'm still somewhat in shock about 17 IPO's going down simultaneously, almost to the second

 

[intrigrant]

I think you should contact Avaya Tier 3 through your reseller, this is not something to be solved here as it sounds like a software bug.

If it ain't dutch it ain't much

 

[amriddle01]

I had one of ours go down this morning generating an Abnormal Termination error at exactly 09:30 (10:30 to you I guess), that uses SIP trunks but not STUN.

 

[NLPeterK]

Must be a bug across all releases then.

I also had two go down at 10.30 but I thought they were unrelated... Apparently not

 

[SCNIPOffice]

My apologies, I must have bumped the switch.

 

[hairlessupportmonkey]

Actually when you mention it I had a switch fall over due to abnormal termination yesterday too. has SIP but no stun.

ACSS - SME
General Geek

 

[NLPeterK]

Same time?

 

[montyzummer]

me too on sip 9:50am GMT running stun , but check this out ,the IPO lan 1 Ip address changed to the Ip of the G/W on site,never seen anything as bizzar as this before.

APSS (SME)
ACSS (SME)

 

[amriddle01]

Make that 2 for me, one went down at 10:03 too

 

[NLPeterK]

Yes exactly that!

You might wanna check all your SIP IPO's

 

[amriddle01]

Could it be the major web providers making some provisioning/hardware changes for IPv6 and it broadcast/changed something the IPO's didn't like....who knows

 

[Pepp77]

9.30 yesterday morning we had one customer go down with this. SIP via Gamma and Voiceflex, and STUN in place using Voiceflex's .221 server.

| ACSS SME |

 

[amriddle01]

Hipcom/BT trunks in both my cases, it's different providers across different firmware levels and platforms with STUN and some without....only common factor is SIP trunks of some variety. I am fairly sure we will never find out what it was :-0

 

[tlpeter]

I had one last week while it don't use STUN and was not even live yet (did that yesterday)


BAZINGA!

I'm not insane, my mother had me tested!

 

[AndrePa]

Had a 500v1 R6.1(20) that all sip trunks became unregistered. Sip trunks were LES, callwithus. Stun used 69.90.168.13. Server, data network, network printer okay. It was noticed at 8am this Morning, not sure when it happened. I was able to reboot remotly, All is good now. I'm keeping an eye on it.

 

[NLPeterK]

I have that too every now and then, but that's a bit of a different problem. We're talking proper crashes here