Ghost calls to 911 - NOT a misdial!

[kalashnikov118]

We are running a call manager 8.5 and CER 8.5. Last week we had an incident where local 911 was calling us back saying that they were getting repeated dead air calls from one of our offices. Site security was getting the emails from CER and responding but the staff member at the extension in question denied mis-dialing 911. I advised her to stop using her phone while I looked into the issue. CER showed several 911 calls from her extension, but CDR did not. I went up to the office and sat at her desk. The phone, a 7965, appeared to be in normal working condition. Directories -> Placed calls showed several calls to internal and external numbers as well as several calls to 911. I dialed out to our helpdesk 91-XXX-XXX-XXXX and spoke with the tech for a few seconds. I hung up and there was a new call to 911 listed in Placed Calls along with my call to the helpdesk! Sure enough the phone rang and it was 911 doing a callback to see if everything was ok. Security then arrived to respond to the call. I power cycled the phone and made another call. No 911 call that time or since. I opened a ticket with TAC and provided them with the traces for the time period that I had made the call. The engineer responded that the logs showed two separate calls both initiated with the "New Call" button. The 911 call FIRST(!) and my call the helpdesk 30 seconds later. I was sitting at the desk for at least 3 minutes before dialing the helpesk and I certainly didn't hit new call and 911. Has anyone ever had this happen?

Relevant log snip:

911 call:

13:49:57.962 |StationInit: (0001493) SoftKeyEvent softKeyEvent=2(NewCall)
lineInstance=0
callReference=0.|2,100,50,1.8589796^10.104.20.155^SEP001D70FCXXXX

91XXXXXXXXXX: (helpdesk call I initiated)

13:50:31.454 |StationInit: (0001493) SoftKeyEvent softKeyEvent=2(NewCall)
lineInstance=0
callReference=0.|2,100,50,1.8590481^10.104.20.155^SEP001D70FCXXXX

 

[trvlr1]

This sounds like something we have going on in one of our remote offices. We ran all similar tests to you, could see no instances of 911 being called, but they were still there. I would hate to have to power cycle 300+ phones to try to cure this, but if it does fix it, thats what will have to happen. Sounds like a bug to me though. If Cisco do put some time in to this and come up with a fix, please let us all know.

 

[kalashnikov118]

I feel like it's a bug in the phone firmware. Maybe the first call to 911 was an actual mis-dial and the subsequent calls were the result of a buffer not clearing. The TAC engineer didn't seem real enthused about chasing this down any further, as the log certainly looked like someone with opposable thumbs was physically dialing the digits.

As far as power cycling the phones I would think that a reset would have the same effect. I've done 700+ all at once in CUCM through Bulk Administration -> Reset Phones. Just leave at least an hour or two of downtime after as that many can take longer than expected to home back in on the call manager.

 

[whykap]

have you made sure there is not PLAR configured on that phone or misconfigured in this case?

 

[kalashnikov118]

The phone in question was in use for over a year without issue. I don't see how a PLAR could have been configured and then go away after a power cycle. The behavior certainly seems like a PLAR though.

 

[whykap]

have you tried replacing the phone? Stuck buttons or bad hookswitch possibly?
Also what about any apps on the pc that would control the phone remotely? Click to call, jabber, etc.

 

[trvlr1]

Whykap: in our case we thought of all those things, but the logs still showed 911 was not being called.
We did go to the telco and ask them to run reports to see if someone else may have had an old number of ours or if one had been entered in the data base with incorrect information, showing our name instead of the other company, but they were less than enthusiastic about pulling those reports, so I am not sure I trust their validity or not.

 

[MitelInMyBlood]

A 911 misdial trap might solve this. Also remove the checkmark beside "Urgent Priority" and then lower your T302 timer down to 5000 (5 seconds) so that a real 911 call is only delayed by 5 seconds before outpulsing.

The 911 misdial trap is built in the route pattern table and looks like this: 911[0-9]
ergo this looks for 911 + any additional digits to follow, diverting the call (along with the remaining errant digits) to whatever RG you define. (an invalid one, for example) 911 dialed all by itself, as in a real emergency, would simply sit there for the duration of your T302 timer before proceeding out the correct route.

We've successfully used this method for years with our old Mitel PBX and migrated the idea onto the CM with equal success. Yes, 911 is slightly delayed, but we have no more accidental misdials.


Original MUG/NAMU Charter Member