Re: jitters, packet losses, sip link alarms

[mitelgear]

I am currently experiencing major problems with a customer which has 3 x 3300 cx/cxi controllers all on version 7.1 software. Im pretty useless at solving networking issues so i would appreciate any responses. Site A(cx controller) is the main site with with an isdn30 and ip trunks to 2 other controllers(cxi's), sites b and c. There is also a live business gateway across all 3 sites and a speech server(6500). There are packet losses, jitters and ip trunk alarms on site A. Every 5 minutes a critical alarm is raised(sip link alarms), One way communication between ip phones which happens erratically. I have posted below software logs of the errors, any help would be greatly appreciated!

Software Logs:

2759 Warning 2007/Dec/20 15:48:27 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2758 Info 2007/Dec/20 15:45:55 SIP/SIPSm "The SIP link LCS has been restored. "
2757 Warning 2007/Dec/20 15:43:55 SSP "Fail to get IP for fqdn[spartan.viroscreen.com].Check on DNS server if all Resource Records related to this fqdn are provisioned correctly.Or check if all IPs provisioned for this fqnd are reachable/in service "
2756 Warning 2007/Dec/20 15:37:55 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2755 Info 2007/Dec/20 15:35:23 SIP/SIPSm "The SIP link LCS has been restored. "
2754 Info 2007/Dec/20 15:33:34 E2tSp "IP Network Congestion AlertJitter from phone 10.10.210.60 > 300msCall duration 1370 sec JB over=0 under=882 maxJtr=394msnumber of Pkts with J > 200ms = 8 "
2753 Info 2007/Dec/20 15:33:34 E2tSp "IP Network Dropped 2.30 percent of Packets from phone 10.10.210.60Total Rx: 68526 Lost=1610 maxLossBurst=7 1 in a row: 1286 2 in a row: 120 3-5 in a row: 23 6-10 in a row: 1 11 or more: 0 "
2752 Warning 2007/Dec/20 15:31:23 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2751 Info 2007/Dec/20 15:29:14 E2tSp "IP Network Dropped 6.32 percent of Packets from phone 10.0.3.39Total Rx: 7132 Lost=481 maxLossBurst=4 1 in a row: 390 2 in a row: 36 3-5 in a row: 6 6-10 in a row: 0 11 or more: 0 "
2750 Info 2007/Dec/20 15:28:51 SIP/SIPSm "The SIP link LCS has been restored. "
2749 Warning 2007/Dec/20 15:24:51 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2748 Info 2007/Dec/20 15:23:02 E2tSp "IP Network Congestion AlertJitter from phone 10.10.210.68 > 300msCall duration 157 sec JB over=0 under=52 maxJtr=399msnumber of Pkts with J > 200ms = 1 "
2747 Info 2007/Dec/20 15:23:02 E2tSp "IP Network Dropped 2.04 percent of Packets from phone 10.10.210.68Total Rx: 7876 Lost=164 maxLossBurst=4 1 in a row: 120 2 in a row: 14 3-5 in a row: 5 6-10 in a row: 0 11 or more: 0 "
2746 Info 2007/Dec/20 15:22:52 Login/Logout Security Audit "Action: login, User: Thund3r, IP Address: 138.37.208.196, Login Access: System Data Synchronization Administration, User Validated: true, Session Available: true, Session ID: 11, Comments: "
2745 Warning 2007/Dec/20 15:22:40 Login/Logout Security Audit "Action: login, User: Thund3r, IP Address: 138.37.208.196, Login Access: ESM Login Page, User Validated: false, Session Available: false, Session ID: 0, Comments: Invalid password. "
2744 Info 2007/Dec/20 15:22:19 SIP/SIPSm "The SIP link LCS has been restored. "
2743 Info 2007/Dec/20 15:14:36 E2tSp "IP Network Dropped 2.61 percent of Packets from phone 10.0.3.29Total Rx: 31660 Lost=850 maxLossBurst=4 1 in a row: 759 2 in a row: 33 3-5 in a row: 8 6-10 in a row: 0 11 or more: 0 "
2742 Warning 2007/Dec/20 15:13:47 SSP "Fail to get IP for fqdn[spartan.viroscreen.com].Check on DNS server if all Resource Records related to this fqdn are provisioned correctly.Or check if all IPs provisioned for this fqnd are reachable/in service "
2741 Info 2007/Dec/20 15:10:11 E2tSp "IP Network Dropped 2.08 percent of Packets from phone 10.10.210.68Total Rx: 471 Lost=10 maxLossBurst=1 1 in a row: 10 2 in a row: 0 3-5 in a row: 0 6-10 in a row: 0 11 or more: 0 "
2740 Info 2007/Dec/20 15:10:03 E2tSp "IP Network Dropped 2.03 percent of Packets from phone 10.10.210.60Total Rx: 145 Lost=3 maxLossBurst=1 1 in a row: 3 2 in a row: 0 3-5 in a row: 0 6-10 in a row: 0 11 or more: 0 "
2739 Warning 2007/Dec/20 15:09:47 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2738 Info 2007/Dec/20 15:05:15 SIP/SIPSm "The SIP link LCS has been restored. "
2737 Warning 2007/Dec/20 14:57:15 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2736 Info 2007/Dec/20 14:54:43 SIP/SIPSm "The SIP link LCS has been restored. "
2735 Warning 2007/Dec/20 14:43:39 SSP "Fail to get IP for fqdn[spartan.viroscreen.com].Check on DNS server if all Resource Records related to this fqdn are provisioned correctly.Or check if all IPs provisioned for this fqnd are reachable/in service "
2734 Warning 2007/Dec/20 14:39:39 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2733 Info 2007/Dec/20 14:39:30 E2tSp "IP Network Dropped 2.08 percent of Packets from phone 10.0.3.32Total Rx: 1459 Lost=31 maxLossBurst=2 1 in a row: 29 2 in a row: 1 3-5 in a row: 0 6-10 in a row: 0 11 or more: 0 "
2732 Info 2007/Dec/20 14:37:07 SIP/SIPSm "The SIP link LCS has been restored. "
2731 Info 2007/Dec/20 14:31:18 E2tSp "IP Network Dropped 3.70 percent of Packets from phone 10.0.3.39Total Rx: 1067 Lost=41 maxLossBurst=3 1 in a row: 32 2 in a row: 3 3-5 in a row: 1 6-10 in a row: 0 11 or more: 0 "
2730 Info 2007/Dec/20 14:31:09 E2tSp "IP Network Dropped 2.01 percent of Packets from phone 10.10.210.68Total Rx: 2099 Lost=43 maxLossBurst=4 1 in a row: 33 2 in a row: 3 3-5 in a row: 1 6-10 in a row: 0 11 or more: 0 "
2729 Info 2007/Dec/20 14:28:43 E2tSp "IP Network Dropped 2.56 percent of Packets from phone 10.10.210.68Total Rx: 1180 Lost=31 maxLossBurst=2 1 in a row: 29 2 in a row: 1 3-5 in a row: 0 6-10 in a row: 0 11 or more: 0 "
2728 Info 2007/Dec/20 14:27:59 E2tSp "IP Network Dropped 2.37 percent of Packets from phone 10.0.3.32Total Rx: 1070 Lost=26 maxLossBurst=2 1 in a row: 24 2 in a row: 1 3-5 in a row: 0 6-10 in a row: 0 11 or more: 0 "
2727 Info 2007/Dec/20 14:27:01 E2tSp "IP Network Dropped 2.16 percent of Packets from phone 10.10.210.69Total Rx: 3988 Lost=88 maxLossBurst=4 1 in a row: 72 2 in a row: 2 3-5 in a row: 3 6-10 in a row: 0 11 or more: 0 "
2726 Warning 2007/Dec/20 14:20:35 SIP/SIPSm "The SIP link LCS has failed with error RequestTimeout - (Options).Please check the connection between the controllers.The link will be brought into service when communication is reestablished. "
2725 Info 2007/Dec/20 14:18:03 SIP/SIPSm "The SIP link LCS has been restored. "
2724 Info 2007/Dec/20 14:15:18 E2tSp "IP Network Congestion AlertJitter from phone 10.10.210.68 > 300msCall duration 30 sec JB over=0 under=36 maxJtr=395msnumber of Pkts with J > 200ms = 1 "

 

[Lundah]

What's the IP network between each of the controllers look like? Link speeds, gear, etc.?

 

[mitelgear]

link speed 100meg sdsl link for the ip trunks. i believe the live business gateway uses sip. Its also using vlans to separate voice and data network with dhcp enabled on the 3300. Im not very clued up with the newtwork side and have been thrown in the deep end on this one.

 

[Lundah]

how stable are the WAN links? Are you using any QoS on the voice VLAN? The log messages you are seeing indicate either links are dropping or at the very least are congested.

 

[mitelgear]

Im not certain about the wan links but there is qos on the voice vlans as this question was asked to the customer. Nothing has changed on the customer network. i will try to get more info on the wan links and reply.

 

[TheMitelGuy]

What are the data switches? How exactly are they configured?

 

[Echoeng1]

I am seeing the same log on one of my controllers. At the same time I have had some phones drop of the network and come up with waiting. The network guys are saying that its not the LAN switches. When I set up various pings either through the phones or directly on the network, I see interuptions i.e destination unreachable. This is proving very dificult to pinpoint and in my view looks to be congestion but we are not sure where.

2706 2008/Jan/10 14:46:04 E2T/E2tSp IP Network Congestion AlertJitter from phone 62.172.72.15 > 300msCall duration 2650 sec JB over=0 under=2744 maxJtr=420msnumber of Pkts with J > 200ms = 1
If anyone else couls shed some light on this, I would be grateful.

 

[Lundah]

Your network guys may be right in saying there's not a problem with the LAN switches, but what about the traffic level? Is there any QoS implemented for the voice traffic?

 

[TheMitelGuy]

Echoeng1... When you say " When I set up various pings either through the phones or directly on the network, I see interuptions i.e destination unreachable", are you pinging the 3300?

Also, during the time when these interuptions happen, do all users experience this at the same time?

What is manufacture and variant are your data switches? Is there routers between switches? Is there more then one switch?

 

[yeahbutno]

When I experienced an almost identical issue as your seeing above It came down to a conflict in configuration between the provider of the WAN links and the config on the LAN network.
Also, the guys that were looking after the network also said that there was nothing wrong on the network which to some extent was true,because they weren't looking at the WHOLE picture. They to also said they were running QoS but unfortunately wasn't configured correctly either!!
So my first point of attack would be is to see if you can minimize the locality of phone calls between each other and go from there.

Remember,even with correctly configured QoS, packets eventually WILL be dropped if the network has too much jitter and congestion.
Suggest that you also check with your network guys to see at what level of bandwidth you're running at.

Maybe the below link will help somewhat,its a video of QoS.

http://www.snapdrive.net/qs/67f8d86771f5

 

[yeahbutno]

If the above link for the video doesn't work then try this one

http://www.snapdrive.net/files/524149/QoSvideo.rar