Webex audio issues (SIP trunk, remote site and 7841 phones)

[drewdown]

Recently brought up a new remote site with an MPLS connection back to CUCM(9.1.1.2).

Remote users are reporting issues of call quality issues but only via webex calls. It seems like audio loss, jitter and digitization are a daily occurrence for them but only on Webex calls. We are using a local SIP trunk so their calls go out via a local gateway (2911). I engaged TAC but the amount of packet captures and spans they want me to configure (while being 3000 miles away) is disheartening and difficult for me to do. To me it looks to be provider related but the provider has been of little help as of late.

Looking at the CUBE and the phone I don't see anything that would point the finger at our local network. I have the logs from the phone at the time (user reported 9:05) but nothing jumps out at me. I dont have CUBE logs as I wasn't logging at the time. Does anyeone know what could cause this? I believe it has nothing to do with with phone < > CUCM at this point and would lean towards the provider?

Having said that I do see Rcvr Lost Packets ad Latency from the 'stream' on this users phone.

Rcvr Lost Packets 211
Severely Conceal Seconds 27
Latency 17
Max Jitter 9

2352 NOT May 24 09:04:03.514742 CDP-gdpReadThrd(): CDP select wakeup - n = 501
2353 NOT May 24 09:04:38.740684 NTP-calling catchAlarm
2354 NOT May 24 09:04:38.741081 NTP- expired: targ=19B7D423 tmr=0xE0E350 func = 0xc37c arg=1B4FC
2355 NOT May 24 09:04:38.741814 NTP-remove timer ok
2356 NOT May 24 09:04:38.742180 NTP-before ptimer
2357 NOT May 24 09:04:56.628736 JAVA-ccsip_messaging: sipSPIAddContactHeader: CFGID_DEVICE_NAME = SEP3820560D014D
2358 NOT May 24 09:04:56.628949 JAVA-ccsip_messaging: sipSPIAddContactHeader: ccb->call_mode = 0, display_name = <localDID>
2359 NOT May 24 09:05:03.560309 CDP-gdpReadThrd(): CDP select wakeup - n = 501
2360 NOT May 24 09:05:32.526157 JAVA-ccsip_messaging: sipSPIAddContactHeader: CFGID_DEVICE_NAME = SEP3820560D014D
2361 NOT May 24 09:05:32.526370 JAVA-ccsip_messaging: sipSPIAddContactHeader: ccb->call_mode = 0, display_name = <localDID>
2362 ERR May 24 09:05:32.563361 JAVA-SIPCC-SIP_CC_PROV: ccappFeatureUpdated: REG_STATE event:line=1,state=1
2363 NOT May 24 09:05:32.565803 JAVA-SIPCC-UI_API: ui_is_server_side_adr: server side application dial rules set = 0
2364 NOT May 24 09:05:42.732749 NTP-calling catchAlarm
2365 NOT May 24 09:05:42.733146 NTP- expired: targ=19B7ED23 tmr=0xE0E350 func = 0xc37c arg=1B4FC
2366 NOT May 24 09:05:42.733878 NTP-remove timer ok
2367 NOT May 24 09:05:42.734245 NTP-before ptimer
2368 NOT May 24 09:06:03.603434 CDP-gdpReadThrd(): CDP select wakeup - n = 501
2369 WRN May 24 09:06:04.821014 kernel: [4315160.935432] dspgfb_ioctl: 1417 callbacks suppressed
2370 DEB May 24 09:06:04.821197 kernel: [4315160.935463] FILL block test for LCD driver
2371 DEB May 24 09:06:04.821289 kernel: [4315160.935463] DSPG LCDC DMA start
2372 DEB May 24 09:06:04.821380 kernel: [4315160.935493] DSPG LCDC DMA end
2373 DEB May 24 09:06:04.890784 kernel: [4315161.010940] FILL block test for LCD driver
2374 DEB May 24 09:06:04.890967 kernel: [4315161.010970] DSPG LCDC DMA start

 

[gnrslash4life]

Yeah, you're stream is definitely proving there is a problem. Have you checked to see if the MPLS circuit is showing errors on your router out there? Is your Webex in house or hosted?

Certifications:
A+
Network+
CCENT
CCNA Voice
TVOICE
CAPPS

 

[drewdown]

Webex is hosted, off-prem. The stream refers to the connection to CUCM? Other than signaling why would the MPLS connection come into play when the calls are going out via the local CUBE > SIP TRUNK > PSTN? AS far as I can tell the MPLS connection is clean (no drops out CRC errors on the MPLS facing interfaces).

 

[gnrslash4life]

Either your SIP is local to that site or is centralized, it still needs to ride a data connection. So either your SIP is on that MPLS or the connection is riding MPLS over to where the SIP is.

Certifications:
A+
Network+
CCENT
CCNA Voice
TVOICE
CAPPS

 

[drewdown]

I may have misled you then. SIP trunk is local to where the issue is (remote office), CUCM is remote (HQ) and accessed via an MPLS connection. Having said that you would expect to the problem to be with the provider's path? And not the path to CUCM?

 

[gnrslash4life]

So the SIP is riding an MPLS connection to the site. So your path would be Phone -> Gateway -> PSTN. I would be looking at the MPLS circuit that the SIP is riding on. But are they having issues with other calls or just Webex?

Certifications:
A+
Network+
CCENT
CCNA Voice
TVOICE
CAPPS

 

[drewdown]

Just webex. Another report and the stream again showed RCVR Lost Packets of 87 but again not sure where its losing them, out towards PSTN or over MPLS to CUCM?

As far as I can tell there are no issue on the MPLS connection, no drops, CRC or anything like that on the interfaces that I can see. And if it were the MPLS connection I would suspect it would be prevalent on all calls and not just WEBEX. Pulling my hair out here trying to figure out what the problem is or where it resides.

 

[gnrslash4life]

Seems like your best bet is to do the packet captures. Otherwise you're chasing ghosts.

Certifications:
A+
Network+
CCENT
CCNA Voice
TVOICE
CAPPS

 

[drewdown]

I think you were right all along but I just wasn't able to see it. I made two changes and more than likely only the first one mattered.

1. Something I hadn't noticed until last week was that if I did an extended ping in either direction it would drop a packet every so often. What I found was the traffic was being routed to a PA 3050 (routing for certain networks) and once I moved the route to bypass the 3050 no packets were dropped. Prior to this I was doing short pings to see latency and RTT, nothing extended so it looked fine to me.
2. Removed the data vlan tag on the end users switchport.

Anyways the end user reported no problems on his webex audio after these changes were made.

Thanks for your help gnrslash4life.

 

[gnrslash4life]

I would bet the first option was the culprit.

Certifications:
A+
Network+
CCENT
CCNA Voice
TVOICE
CAPPS