Cisco IP Phones dropping calls out of the blue

[Hankejp]

Just wondering ifanyone has come across this problem. In one of our buildings we have multiple users who when talking on their 7940's are having their calls drop. There's no noise or anything when the calls drop the phone just goes dead. The user can dial a new number right away and the phone doesn't reset. It can be when talking to a number outside the business or inside, IP to IP, IP to Mitel, it doesn't matter. We had some engineers look at all of our routing logs and they say everything seems to be fine with them. The next step for us is to wait for a user to get a call dropped and to look in the logs as to why it was dropped.

Just wondering if anyone is or has experienced this problem.

 

[jneiberger]

I asked a Cisco engineer about this and here is his suggestion:

"For starters, I would look at the CDR records via the CDR Analysis and
Reporting (CAR) tool. Specifically I would like to look at the
Termination Reason Code and Description for the call. If that were not
enough information, I would then look at the Call Manager Traces to see
specifically what took place with the call in question."

So, if it happens again perhaps this will give you another place to look.

Good luck!

 

[Hankejp]

Thanks for the help.....to all of you. If you were in my area, I'd buy ya all a beer.:-D

 

[telcomwork]

So what was the problem?

 

[jneiberger]

Yeah, it's been nearly a month. Have you made any progress on this issue?

 

[Hankejp]

I've opened a TAC case and have sent log files to the engineer that I'm working with. They were in XML format and he couldn't get any info from them. I switched the log files to txt format and am just waiting for a user to call me saying they had a call drop so I can get that logfile to send.

I'll keep everyone posted on what we find out.

 

[mishkanyc]

Your issue is probably CCM service related, if your service has either memory corruptions or page fault overload s it can start dropping calls, i've seen it.
I would recommend and check the general health of your callmanager system first.
Are you running a cluster? If oyu have more then one subscriber offload some call processing between them and see it it helps. also make sure your ccm service or SQL isnt bogged down. suprisingly slow response from the service can trigger a call drop.
But it may also be a hardware issue. i've had two replace two fones not to long ago which dropped calls on the regular bases.

 

[lui3]

Check your QOS configurations. I have had issues with RTP streams dropping due to to improper QOS configs. Placing dscp 46 in the scavenger queue or not having qos enabled at all. You don't need a lot of bandwidth to mess up the phone. its more of packet per second issue than mbps. I am able to get pretty high jitter rates (>50) with less than a meg of traffic pushed at the phone. The intermittence of your problem seems to lend itself to something bursting on your network, say ghost imaging or some other multicast broadcast issue. By default with QOS disabled everyone gets the same share of bandwidth. Large File Transfer packets can wreak havoc on VOIP RTP streams. If your QOS is enabled correctly then you should be ok. However, i have seen lots of folks not enable QOS on their switches since they assume bandwidth is plentiful and they don't have congestions issues. Try sending a 1000Kbps of icmp at one of the phones while using the ? info key and see what the stats say. If jitter jumps around you know you have a qos issue. I would check there next.

Plus, when your phones drop calls do you see unregistration errors on CCM as well? I was seeing both when i had a qos problem since uplink was getting bogged up and causes phones to lose contact with ccm server.

Lui3
CCNP,CCDA,A+/Net+
Cisco Wireless Specialization

 

[pndscm]

We just recently ran into similar issues with dropped calls, static and one way conversation on phone-gateway G729 calls. G711 phone to gateway calls and G729/G711 phone to phone calls seem to be unaffected. We think it is the 7.2.2 phone load. According to the Release Note Caveats, these are unresolved issues with that load.
We just upgraded 10-15 phones to 7.2.3 and haven't had the one way conversation re-occur in those 7.2.3 phones but we just did this yesterday afternoon. Will let everyone know end of next week if it fixed our problems.

 

[SStieferman]

I've also experienced similar call drops, however they have been related to DSP issues on the PRI voice gateways, multiple 2821's and one 2651 daughtercard.. It would be interesting to know if IP to IP calls drop as well as IP to PSTN.

In the case of the 2821 there is a known bug with the DSP resources going to "sleep" and dropping calls that have been etabilished.

On the 2651 our issue was a failed DSP that only effected every 4th or 5th channel of the PRI. Because of this failure a call would ring the phone once and then disconnect. For that hardware failure the temp solution was to use the "Change B-Channel Maintenance Status" in the CCM service parameters config to disable the offending channels. Ultimatly a dsp card swap resolved that issue.

To determine the number of failures and preempt the user calls. Open up ART and do a CDR search for Call Termination cause 41 "temp fail". If this is a dsp issue you may see quite a few of these that no one reports. Once you gather this information and track down your traces, TAK may be better able to help with the issue.

Ohhh yea, get your users into QRT, that helps me troubleshoot failed / last call issues tremendously!!!

Scott

 

[Hankejp]

Sorry guys,

It's been a while since I updated everyone. The problem is still happeneing. I have sent numerous log files to Cisco. They're telling me that it looks like the phone is in a "on Hook" state. Meaning that the handset is on the phone. To sum it all up, Cisco hasn't been much help. I am going to our local Cisco retailer that help us install the VOIP system. It takes too long for Cisco to respond to emails and the phone dropping doesn't happening with every call. I'll keep everyone posted.

 

[pndscm]

Phone load 7.2.3 fixed our problems of dropped calls and one side of conversation dropping in and out. Wanted to let everyone know. 7.2.2 BAD, 7.2.3 GOOD We are running 4.02sr2b.

 

[rtfmdude]

hankejp - give me your case number. I can't believe this has gone on this long. Ultimately, TAC should be able to fix ANY problem. It's just a matter of escalating an issue until it is resolved. This should be a P2 case, no less(p2 means that you have someone on the phone as long as you wanna be on the phone).

Have you guys looked at the gateway that you're using? Are you using the same gateway for all the problem calls?

Is this problem spread across all types of phones? 7940/60s? 7920s? 7936s? 7912s? VG238s? (I deployed these for dorm room phone usage)

 

[Hankejp]

Hey guys,

I now have some other engineers looking at the problem as well. These engineers are with the companyt that installed the IP Phone system. They are stumped as well. I took a brief poll with my users on that floor and nobody else has had the problem. So it seems to be isolated to one person, on one floor. Can I put a different phone load on 1 phone, or does the load go to all the phones?

 

[rtfmdude]

if you go to the main phone page, you'll see a device load field; just enter the phone load filename without any extension, for example:

filename: foo.bar
entry in phone load: foo

 

[pndscm]

If it's 1 person on 1 floor, phone load is probably not the issue. I'd do these steps 1 at a time and observe the results:
1. Delete the phone and rebuild it.
2. Exchange phone with someone NOT having the problem. (swap MAC's) Complete swap Handset, cords and all.
3. Swap switchports with someone not having the issue.
4. Observe the user's habits. Make sure they aren't setting a book on the phone, etc.
I had someone swear they couldn't tranfer a call, got wrong numbers, etc. Sent me in circles for 2 weeks. They were dialing the "tranfer to" number B4 they got second 'transfer' DT and CCM was only seeing the last 2-3 digits. I stood behind them and watched them to figure it out.
Our jobs would be MUCH easier if we didn't have Users.

 

[techninut]

Quick Fix:

1) Remove Cisco Equipment

2) Replace with a proven voip/pbx product.

Mitel, Avaya, Adtran, or even a Nortel.

All though I will say that Cisco makes GREAT routers.

T

 

[maverick666]

Had same problem with a gateway that was h323 with FXO.
Its your QOS and CODEC's. Make sure your gateway is hard coded to g711uLaw and get some to look at your QOS.

Raplacing with another vender you will have the same problem unless you fix the QOS.

SBD
CCDP,CCNP,CCVP, CCIE Voice written

 

[Hankejp]

It's been a while, so I thought I would update everyone. The engineers that were helping me couldn't find anything wrong with anything, QOS, Callmanagers, switches, routers, etc. Out of no where, the issue has resolved itself. Last I talked to the user which was a few weeks ago, she said it was a couple of weeks since she had a call drop. I just tried calling her again, but she wasn't there. So who the heck knows what's going on.

 

[MG2005]

techninut you must be some great technician

cisco is the pioneer of voip, they have the greatest vision, and the most investment in it...

the others merely adopted it to try and keep SOME market share of Voip customers, who, like you, are stuck in thinking that having an old brand name on your phone will cause you less problems-

equipment is as good as its install and the people managing it, no matter what brans

and cisco voip equipment, and callmanager and unity, are all awesome products-

nortel voip SUCKS
legacy PBX's SUCK

i use them all .... u prob never ever saw the callmanager GUI let alone used it

 

[dpaige10460]

I had a similar problem external calls being dropped. Cisco couldn't find anything wrong either then I noticed a pattern. This would only happen when it rained. I followed up with the Verizon NOC and they also report an influx of tickets regarding dropped on VoIP system calls when it rains.