IP Phones Rebooting 1

[RobRendle]

Morning all,

This is a very odd issue and is currently being investigated by Avaya, to no avail.

Details ;-

Control Unit - IP500v2 (has been replaced)
Expansion Cards: VCM32 (has been replaced)/BRI8 (has been replaced), CombiATM4 (has been replaced), DS8 (has been replaced)
Releases Attempted: 7.0.36, 8.1 FP1, 8.1 FP1 (build 67)
Handsets have used the firmware from 7.0.36 plus those from 8.1 FP1
SD Card has been replaced and config re-programmed from scratch
We've enabled and also disabled Direct Media Path
Handsets use Call Appearence keys via the user's settings rather then User Rights (I've seen the latter can cause problems)

Issue occurs on 1608i/1616i WITHOUT bbms. The handsets with bbms seem to be recorded.
4 Port VM Pro, with Call Recording set to on, on 4 extensions (On NOT Mandatory) - these handsets haven't rebooted.

One handset at any given time will restart. However, it won't restart in a way that you'd expect an IP Phone would - it's like it loses power for less than a second. The users have base extensions and login users, when the phone restarts the user logged in does not get logged out from the handset.

Phones are connected to the LAN2 port and hence does not share the Cat5e network with the customers PC's/Network Devices. Phones are assigned an IP Address via the IP Office's DHCP setting and are powered using PoE switches (both have been replaced)

Customers network has been proven as OK via an independant Cat5e companies testing.

Wireshark traces show that the handset is "talking" to the IP Office control unit without issue, the IP Office/Handset respond to the keep alive requests just fine and then the handset "Disapears" for a few seconds and then comes back..

SSA shows nothing untorward and a monitor trace show's the following :-

92325983mS H323Evt: GK: Unregister endpoint SystemName_50f7dcdb40364ba2 for extension 322 reason 3
92325983mS CMExtnEvt: GPRoom8:322 ExtnFault now 1

Note I've edited the systemname section for confidentiality reasons! But in it's place was the name of the telephone system

It doesn't really show anything other than that to acknowledge a phone has restarted. If you go in to the h323 phone status section you can see a selection of handsets that have sufferent from the issue.

We have also deployed a few power bricks to see if there was something in the Cat5e network that could interupt the power, however those with power bricks have restarted since then. So really I believe we may have eliminated the following :-

Control Unit
Expansion Cards
PoE Switches
Customers Cat5e Network

Leaving potential issues with :-

IP Handset Firmware
Bugs in 7.0.36, 8.1 FP1, 8.1.67
VoiceMail Pro Call Recording - noted minor congestions here, am trying to get confirmation that the congestion count coincides with a handset restart


If anyone has come across any similar issues or has any suggestions I'd love to hear from you!

Many Thanks,

Rob

 

[RobRendle]

sorry, not sure how to edit my post but the line with :-

The handsets with bbms seem to be recorded

Needs to be :-

The handsets with bbms seem to be fine and haven't restarted

 

[tlpeter]

Do those 16xx with BM32's have a local power brick?
It seems to me like it is a POE issue.


BAZINGA!

I'm not insane, my mother had me tested!

 

[RobRendle]

Hi Peter,

They don't. Our suspicions (and Avaya's) matched yours so in our latest visit we replaced some handsets with new ones and added power bricks and unfortunately those replacements suffered from the same issue, which would suggest the issue is not PoE related nor is it related to a physical issue with the handsets. I'm wondering if the handsets with a bbm attached use a slightly different variant of the IP Handset firmware?

 

[mpisano]

What brand/model is your POE switch?

Do your Avaya bricks have a green or blue LED?

 

[IPGuru]

1616's with BM32 modules should always have an external power brick.
but i would then expect them to play up not the other handsets.


A Maintenance contract is essential, not a Luxury.
Do things on the cheap & it will cost you dear

 

[RobRendle]

My mistake (its been a LONG!! day :/), the 1616's with BBM's DO have power, however they have never restarted.

The PoE are the NetGear FS Series and the few power bricks used have the green LED

 

[tlpeter]

I have seen strange issues on those NetGear switches and too much poe being used.

Also did you turn off poe for the phones with the power brick?
if you can then replace the switch with a proper one.


BAZINGA!

I'm not insane, my mother had me tested!

 

[RobRendle]

I'm not sure if the on site Engineer turned PoE off on those handsets, so I'll check, thanks for that. What PoE Switches would you reccomend? We've always used NetGear and have never come across issues like this, however both switches are pretty much fully loaded with IP Handsets - both have been replaced to rule out any potential hardware issues, however if this is an inherent issue with NetGear then perhaps we're on to something!

Thanks again

 

[CptIPO]

Which Netgear FS units are they. If they are fully laden 24 port switches make sure that the power Wattage usage is not above 170 Watts in total. Also how far away are the phones from system, the further from system the more power they use. Have you checked that the data cables are not cable like telephony cables. Example being blue blue white on pins 1 and 2 than orange orange white on pins 3 and 4. I have had this issue where data company did this and data pairs where twisted with power pairs. Have you tried not using PoE from switch and disable it on port and then use the injector at phone instead?

ACIS - Avaya Certified Implementation Specialist
ACSS - Avaya Certified Support Specialist
APSS - Avaya Professional Sales Specialist
Yes we have VoIP in Cape Town

 

[CptIPO]

Hey did you come right.

ACIS - Avaya Certified Implementation Specialist
ACSS - Avaya Certified Support Specialist
APSS - Avaya Professional Sales Specialist
Yes we have VoIP in Cape Town

 

[RobRendle]

Hey CptIpo,

I've disabled PoE on the ports which serve the handsets with power bricks so that we can truely isolate PoE from being the cause of the issue, it'll take a day or two to get a definitive result

 

[tlpeter]

Rob, can you check how much is in use right now?
I really think you have an issue with the switch.
We use HP procurve switches all the time.
Proven to be good.

BAZINGA!

I'm not insane, my mother had me tested!

 

[RobRendle]

Hi All,

Apologies for the delay - looks as if it was in fact PoE Causing the issue, we've disabled the 2x PoE ports for the handsets that have power bricks and the fault hasn't re-occured. We're going to monitor the issue for another day or so to be completely sure before swapping the PoE Switches for a different brand. It's worth mentioning we replaced both PoE Switches previous to this test, so it goes look like a Manufacturer Problem rather than a duff PoE Switch.

Peter - both switches were just about within the maximum rating.

 

[jhall123]

Just for another data point we've seen issues with netgear switches when you have a full load of POE. They just cannot support a full 24 ports of POE. Check the specs on that model and make sure they support however much power you need. We've used Avaya and Adtran switches without a hitch.

 

[CptIPO]

Glad you managed to figure it out. PoE is always an issue when switches are really full. PoE+ switches are much better if you are not sure of exactly how much power you will need for install. HP procurves and the Avaya newer switches are great. But in my country the wait time from distributer for the Avaya switches is 8 weeks not easy for us.

ACIS - Avaya Certified Implementation Specialist
ACSS - Avaya Certified Support Specialist
APSS - Avaya Professional Sales Specialist
Yes we have VoIP in Cape Town

 

[RobRendle]

Hi All,

Apologies for opening an old case.

The problem has returned - same site, same details as before. All handsets using Power Bricks experience the problem. Also deployed a HP ProCurve to check to see if that had any impact - it doesn't. Deployed a handful of new IP Handsets - they're experiencing the problem too.

The only extra observation is it seems to happen when the user is on a SIP Call. Oddly, we receive the BYE before transmitting it on all logged occassions - thinking it's unlikely the far end has hung up first, every time! Other than that SIP Logs seem normal. This could be a red herring, as if the phone is failling to respond to TCP Keepalive requests then I'm expecting it'll do the same to the SIP Provider - OR the IPO is terminating the call due to the lack of response. Very Odd - any suggestions welcome.

 

[RobRendle]

Excuse the last comment about the IPO terminating the call. Long day :/

 

[amriddle01]

Monitor will show the status of every IP phone, so it will show how may times it has restarted/re registered and also the last time it did it and the reason it did it



"No problem monkey socks

 

[RobRendle]

Yep, thanks for that - that's what I'm using to see when each phone has re-registered. The reason is always the same :-

'TCPclose'