Avaya Appliance Locks Up

[Ray_Rdot]

Hi all! Long time listener, first time caller. I hope this is allowed and in the spirit of the forum - this is a general request for ideas.

My office is currently struggling with an IP Office 500 that locks up periodically. We are not VOIP people specifically, just IT jacks and jills of all trades supporting an SMB. We have a VOIP vendor who has tried several fixes: replaced the SD card, replaced the power supply, replaced the chassis, and upgraded us to 11.4.6. None of these moves had any effect.

This seems like a network issue, and it is beyond their expertise. We are going on six months with no resolution, and after the aforementioned, they've stopped troubleshooting. Trying here for new ideas.

The problem: Periodic lock-ups of the Avaya appliance. Desk phones (9608s and 9611s) do not lose power, but the screens freeze. System Status sessions disconnect. The Avaya appliance does not reboot - it just goes incommunicado.

Observations: The lock-ups coincide with a spike in traffic into the Avaya.

Here is a typical instance:

We call this the split; generally traffic in/out is in sync, but sometimes the traffic diverges. It's okay if the blue line (traffic out FROM the Avaya) is higher than the pink line (traffic INTO the Avaya), but bad if the pink line is higher than the blue line.

The frequency: This happens like clockwork at 3:00 am every Sunday, but it also happens unpredictably throughout the week during business and non-business hours.

Current band-aid: We have found that disabling the uplink port between our VOIP switches and the Avaya appliance allows the appliance to recover.

We mirrored port traffic on this port for over a week. The average traffic volumes are below 5mb/s. I'm on the networking side of things, and this seems so minimal?

Our existing VOIP vendor has been unable to offer any real options for doing analysis on the appliance itself, and it feels like mirroring port traffic etc is second-hand information. Are there any administrative tools on the Avaya appliance that would assist us in seeing what exactly is happening here? Zeroing in on the only predictable part of this issue, does the Avaya do any kind of maintenance tasks that would kick off at 3:00 a.m. on Sundays? How would I locate and audit existing maintenance tasks?

Thank you.

 

[critchey]

Is the system accessible from the internet(is it on a public IP)? I have seen this once in the past that the system was getting hammered by outside sources and would bring the system to it's knees. Finally proved to the customer it was the fact he had it on a public IP and once he properly used a firewall never had the issue again.

The truth is just an excuse for lack of imagination.

 

[Ray_Rdot]

@critchey, thank you for the response. We do have a public IP associated with the Avaya. There is a firewall in between, but it's doing full cone NAT. Is it worth pursuing a more restrictive NAT type? Otherwise, we pulled a list of firewall rules and sent them to our VOIP vendor, but they didn't have any feedback on them.

We haven't been brave enough to wipe out the firewall rules and just see what happens, but this time between Xmas and New Years might be the best time to do it...

 

[critchey]

I guess the question is can you see/access the unit from manager from the public IP? Or can a phone connect directly on the public IP the IPO is on? If so (and you don't have it locked down to specific IPs or MAC addresses) then I would start to suspect that is the issue.

We do have a public IP associated with the Avaya. There is a firewall in between, but it's doing full cone NAT. Is it worth pursuing a more restrictive NAT type?

Based on what you are saying I am not sure if the IPO itself is on a public IP (is the LAN or WAN port entered as the public IP?) because if the firewall is the public IP and its forwarded to the IPO then that is the correct way to do it.

The truth is just an excuse for lack of imagination.

 

[snowman50]

Do you have SIP trunks .

I have seen customers whose firewall was open on the ports 5060,5061
This allowed the system to be swamped with hackers trying to access the system with ip phones to make expensive toll calls.

Using monitor you could see them trying to logon to system.

This would eventually swamp the IPO and system would reboot.