96xx Voice Quality

[bmast]

Thank you in advance for any help you can offer. I have been troubleshooting voice quality on a new IPO 500V2 8.0.42 installation and am currently stumped.

The installation consists of approximately 58 IP extensions; 9650, 1603, and 3725 DECT phones. The IP Telephones are on a dedicated (physically separate) PoE data infrastructure. LAN 1 of the IP Office serves as DHCP server and is connected to the PoE switch stack, LAN 2 is on the customer data network and connected to the data switch stack. There is no connection between the VoIP and Data network (I have checked the MAC tables in the VoIP switches) except through the IP Office.

I have been getting QoS Voice Quality alarms and degraded voice for several weeks now. A reset of the IP office and/or switch stack seems to improve the situation for a period of time, but the quality soon degrades again. I have tested the following:
1) Verified spanning tree and root bridge configuration is the primary switch, verified that no switching loops were detected. Physically inspected cables to verify no broadcast storm.
2) Viewed statistics in the switches to verify that most traffic is unicast, and that there are not broadcast storms or errored packets.
3) Mirrored the LAN1 port on the IP Office and collected flow statistics using a diagnostic appliance. This revealed some interesting and seemingly inexplicable information:
* Several of the RTP ports 49152, 49154, 49158, etc seemed to remain open and continue to generate traffic roughly equivalent to a g.711 media stream despite all devices in the system being idle (even at 2 am on a sunday).
* This behaviour cleared somewhat when the IPO was rebooted, but quickly re-established itself after a few hours in service.
* I reset the entire switch stack and IP office, and the behaviour seems to have cleared (for the last few hours)
4) Tried to run a VoIP assesment against the installed VoIP terminals. I get a 90% packet loss to ICMP pings to all of the 9650 and 1603 terminals but not to the DECT repeaters or IPO LAN1 port...
* are the phones designed to ignore ICMP? Or is this a hint?

 

[bmast]

Jimmy:

Allow me to elaborate, it is always better to have precise details. I performed the following actions, and the cummulative effect appears to have resolved the issue:
1) I relocated the LAN connection from GE port 25 to fast Ethernet port 23
2) I rebooted the switch stack by removing power (2 switches in stack configuration)
3) I rebooted the IPO by removing power ( I should of reset it, but just pulled the power instead)

All of the phones came up, and reported voice quality has been very good for 10 days now. I am uncertain if the IPO hard reboot or switch reboot had the desired effect. I also wonder if it could have been some sort of failed auto negotiation on a gigabit port, I used to see auto negotiation issues between Nortel equipment and Cisco switches, but I forgot to check the speed and duplex on the gigabit port before the reset so I am unsure. Sorry to muddy the issue.

 

[bmast]

Jimmy:

The reports are pretty interesting, you can see that prior to the reset there was a bunch of RTP traffic chatter just pegged up for days at a time and after the fix the chatter disappeared.

 

[amriddle01]

Actually I have been sent a trace recently that showed something similar, the trace showed one side of the RTP stream as the system and the other was a Cisco switch rather than a handset as you would expect, it was a just continuous stream of silence...odd

 

[cormierjimmy]

Had you upgrade to 8.0(44) ? I hear a client who don't have the problem anymore

Still goes well on your system ? no more stupid QOS alarms (Latency and Paquet lost) with 9650 phone?

Jimmy Cormier, Quebec City, Canada
Sorry, I’m french !
15xBCM/SRG,CS1000,23xIP Office,14xVmpro,
10x contactstore,Call Manager,AES,Nice Recording,Callsweet,IEX Totalview,Symposium,Callpilot,Kofax,