Drops in clusters, same callers, is a bad VCM to blame?

[573dawn]

Hi All, I have been trying to find some pattern to my dropped calls. What I have seen in the last 2 days is very curious.

4 drops in a cluster yesterday, all were the same caller. Found out about it too late to see what was showing in monitor, BUT

This morning, 4 drops, all same caller, all on IP Line 40 (for those of you who know the story). Times were 9:08, 9:10, 9:18, & 9:45.

Apparently this is the norm, where all the drops are within a smll timeframe, and all involve the same caller. The caller can be wireless or wireline.

All of these drops can be directly correlated with the error:
169542025mS ERR: EXCEPTION ON MEDIA CONNECTION --- Error from protocol entity! MSDSE --- local timer expired ...

A bad VCM unit has been suggested by intrigrant, anyone else agree?

Thanks, Dawn

 

[tlpeter]

at least you can try to swap the vcm with another one on an other site
if the problems move then you know it for sure
but you can try to switch the phone first
maybe the phone is not that good or it's wire
you don't wantto know how often people step on the cable or the cable gets stuck in the little wheels of the deskchair


ACA - Implement IP Office
ACA - Voice Services Management
______________
Women and cats can do as they please and men and dogs should relax and get used to the idea!

 

[573dawn]

tlpeter, no I probably don't <g>. How does one go about swapping them? Is this something I should have my BP do?

Incidentally, the drops can happen to anyone in the building, so I don't think it's chairs, BUT the building wiring is a nightmare overall. We just got the place properly grounded yesterday, no one knew the building wasn't grounded until all of the phone issues. But if you could see the wiring closet, which it takes a ladder and some bravery to access, you would understand

 

[tlpeter]

but you wrote that it happend with one user so i asumed it is always the same phone but that can be of the list then

if the wiring closet is a big mess than i realy realy realy give the advice to clean up the mess or even better let a company do it who realy know what they are doing

about the vcm ,you probably have a warrenty on your ipo's so yes let the BP do it but then only do the problem site because they bring a new one with them

if you want to do it your self please be carefull and ground yourself (you have a proper ground now )
but better let the BP do it
if they trash it it is there problem, if you do your far from home



ACA - Implement IP Office
ACA - Voice Services Management
______________
Women and cats can do as they please and men and dogs should relax and get used to the idea!

 

[573dawn]

tlpeter, sorry I mis-spoke, it can happen to anyone but generally, let's say that the Service dept gets a call from customer X, customer X drops and calls back, talking to the same service person (or parts, or finance, or sales, etc). They talk for a few minutes and "drop". They call back and the sequence repeats.

I find this very interesting because in each "cluster" that occurs, it almost always seems to be the same caller. Just like this morning, 4 drops, same customer each time talking to the same service writer, but almost all reports of drops come in these clusters.

As for the building, there's a standing joke in the company that someone in management offered $100 to the first person to burn it down <g>. At some point we will either re-wire it or level it and start over, but that is a winter project. The previous owner was quite the skinflint.

 

[intrigrant]

So if i get it right it happens in a short time and mostly with the same caller.
Dou you have data and viop on the same connection?
Can you keep the Avaya monitor running AND log a performance measurement on the data connection?
If these show no anomalies then the next option in my opinion is to replace the VCM Card and yes, i advise you to ask you BP to do it.
Not because you are not capable but if they destroy your IP Office then they must replace it, if you make that mistake then the IPO and you will be replaced.

 

[573dawn]

intrigrant, LOL! I would prefer that they do it anyway, we're still under warranty! Yes, we have voice and data on the same circuit, this particular circuit has allegedly been prioritized for voice, see thread

http://www.tek-tips.com/viewthread.cfm?qid=1394236&page=1

(I posted this thread yesterday but no one has looked at it). When I have monitored the T1, I have seen very few problems, I am not monitoring it right now, but I can set it back up for monitoring.

General usage of the circuit with both data and voice is under 5%. I am expecting a new router (well, dealer refurb) tomorrow, and I am planning on swapping at least one end of the circuit, just so I can say I have.

Meanwhile, I am still a little confused as to exactly what I should be filtering for in Monitor, there are multiple checks on multiple tabs and I don't know which to keep, add, remove, etc. You mentioned the first 4 checkboxes on the system tab, and they are all checked, should I uncheck or check anything else?

 

[573dawn]

Just had another cluster, same thing applies, BUT this person talked to 2 different extensions, one of them dropped the call once, then the customer called back in and talked to another department, same customer dropped 3 times in that department, Media Exceptions each time, all spaced very close together.

I am really starting to see a pattern now! Time to start monitoring the T1 again, but I did have a look at my routers on both ends, no apparent issues, very few errors, etc.

 

[573dawn]

Any suggestions on what I should watch for on my network (other than missed packets & utilization, I'm using Solar Winds Engineering toolset, but I am open to suggestions on monitoring if anyone has any).

Thanks, Dawn

 

[mattKnight]

>but no one has looked at it

I did look at it, but I didn't understand it well enough to comment!

Take Care

Matt
If at first you don't succeed, skydiving is not for you.

 

[ronromano]

Hey, IP line 40 is SCN? What did the call come in on PRI?

Your posts give me a figgin' headache

 

[573dawn]

ronromans, sorry about the headache, I'd stop posting if calls would stop dropping <g>. IP Line 40 is the line number assigned to the IP line at my trouble site. I am going to have to assume that all of these calls come in on the PRI since they all are on 40. They don't appear to have been transferred from another site, but I can't be sure on all of them.

 

[573dawn]

mattKnight, I don't exactly understand it either, I was hoping someone could shed some light on it. I might post it on the cisco forum.

 

[ronromano]

The calls are dropped on 40 right? Its not an IP trunk a carrier so that means it has to be an SCN station to station call or a transfered call from a remote site. If you use PRI did it occur to you that it could be a timing issue on the PRI.

For Christ's sake get a real flippin' vendor!

 

[intrigrant]

If it is a PRI problem then all sites / calls should have the same problem but here it is only one site having problems.
I did look at your previuos threath and i see no traffic on the ef DSCP value, in common this is the Diffserv Expided Forward which is the highest priority.
Are you sure Qos is ok?
The thing to monitor is just the default filter and some of the Media stuff.
The thing to look for are the CMMap events, these are stating whether two parties are connected and the Media stuff for errors.
If you are on 4.0.x then copy the following to a .reg file and merge it with the registry, then he filter is set as needed:

[blue]
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Avaya\IP400\Sys Monitor]
"ipaddr"=""
"ipaddr_idx"=dword:00000003
"key"="$t%$'~ q"
"ip400_distro"=dword:00000001
"filename"="C:\\Program Files\\Avaya\\IP Office\\Monitor\\Monitor.txt"
"daily"=dword:00000000
"onhour"=dword:00000000
"onsize"=dword:00000001
"binarylog"=dword:00000000
"Log2File"=dword:00000001
"fileinterval"=dword:00000001
"statusinterval"=dword:0000000a
"WindowState"=dword:00000000
"isdnport"=dword:00000000
"isdnl1tx"=dword:00000000
"isdnl1rx"=dword:00000000
"isdnl1ev"=dword:00000000
"isdnl2tx"=dword:00000000
"isdnl2rx"=dword:00000000
"isdnl2ev"=dword:00000000
"isdnl3tx"=dword:00000000
"isdnl3rx"=dword:00000000
"isdnl3ev"=dword:00000000
"isdnb1tx"=dword:00000000
"isdnb1rx"=dword:00000000
"isdnb2tx"=dword:00000000
"isdnb2rx"=dword:00000000
"cmport"=dword:00000000
"cm"=dword:00000001
"cmcallpkt"=dword:00000000
"cmcallevt"=dword:00000000
"cmextntx"=dword:00000001
"cmextnrx"=dword:00000001
"cmshortcodemsgs"=dword:00000001
"cmextntxc"=dword:00000000
"cmextnrxc"=dword:00000000
"cmextnevt"=dword:00000000
"cmextncutevt"=dword:00000001
"cmlinetx"=dword:00000001
"cmlinerx"=dword:00000001
"cmlineevt"=dword:00000000
"cmcalldelta"=dword:00000001
"cmextntxp"=dword:00000001
"cmextnrxp"=dword:00000001
"cmmap"=dword:00000001
"cmtargetting"=dword:00000001
"cmlogging"=dword:00000001
"cmembeddedvmail"=dword:00000000
"cmevmaudioresp"=dword:00000000
"cmevmrecorder"=dword:00000000
"cmevmhousekeeping"=dword:00000000
"cmevmflashstorage"=dword:00000000
"cmevmsilence"=dword:00000000
"cmprimitive"=dword:00000001
"cmfaststartinfo"=dword:00000001
"cmars"=dword:00000001
"cmacdevents"=dword:00000000
"cmipdectevents"=dword:00000000
"cmipdectmessages"=dword:00000000
"cmipdecttxextn"=dword:00000000
"cmipdectrxextn"=dword:00000000
"cmipdecttxline"=dword:00000000
"cmipdectrxline"=dword:00000000
"tapievents"=dword:00000000
"igmpevents"=dword:00000000
"tftpevents"=dword:00000000
"tftpwarningevents"=dword:00000000
"tftpdownloadevents"=dword:00000000
"dnsevents"=dword:00000000
"dhcpevents"=dword:00000000
"telnetevents"=dword:00000000
"cstaevents"=dword:00000000
"mhevents"=dword:00000001
"voipevents"=dword:00000001
"primevents"=dword:00000001
"chevents"=dword:00000001
"ssevents"=dword:00000000
"siptxevents"=dword:00000000
"siprxevents"=dword:00000000
"h323summary"=dword:00000000
"cm_delta2"=dword:00000000
"cmcdr"=dword:00000000
"cmcdrextended"=dword:00000000
"dtecmdtx"=dword:00000000
"dtecmdrx"=dword:00000000
"dteflttx"=dword:00000000
"dtefltrx"=dword:00000000
"dteppptx"=dword:00000000
"dteppprx"=dword:00000000
"dtev120tx"=dword:00000000
"dtev120rx"=dword:00000000
"dtev110tx"=dword:00000000
"dtev110rx"=dword:00000000
"dteevents"=dword:00000000
"dtcommsstatus"=dword:00000000
"dtl2"=dword:00000000
"ecdaemon"=dword:00000000
"ecapi"=dword:00000000
"ectargets"=dword:00000000
"ecconf"=dword:00000000
"ecvmail"=dword:00000000
"ecvmtx"=dword:00000000
"ecvmrx"=dword:00000000
"frevenm"=dword:00000000
"frtxm"=dword:00000000
"frrxm"=dword:00000000
"frmgmtm"=dword:00000000
"pppiptx"=dword:00000000
"pppiprx"=dword:00000000
"ppplcptx"=dword:00000000
"ppplcprx"=dword:00000000
"ppplcpecho"=dword:00000000
"pppsectx"=dword:00000000
"pppsecrx"=dword:00000000
"pppipcptx"=dword:00000000
"pppipcprx"=dword:00000000
"pppbacptx"=dword:00000000
"pppbacprx"=dword:00000000
"pppccptx"=dword:00000000
"pppccprx"=dword:00000000
"pppstack"=dword:00000000
"ppplink"=dword:00000000
"ppperrmsg"=dword:00000000
"pppgenmsg"=dword:00000000
"pppmlcptx"=dword:00000000
"pppmlcprx"=dword:00000000
"ppplinktx"=dword:00000000
"ppplinkrx"=dword:00000000
"pppiphctx"=dword:00000000
"pppiphcrx"=dword:00000000
"pppcrtptx"=dword:00000000
"pppcrtprx"=dword:00000000
"ifacename"=""
"ifacetx"=dword:00000000
"ifacerx"=dword:00000000
"ifacerem"=dword:00000000
"natinfail"=dword:00000000
"natoutfail"=dword:00000000
"natin"=dword:00000000
"natout"=dword:00000000
"firewallallowedout"=dword:00000000
"firewallallowedin"=dword:00000000
"firewalltcpallowedout"=dword:00000000
"firewalltcpallowedin"=dword:00000000
"firewalludpallowedout"=dword:00000000
"firewalludpallowedin"=dword:00000000
"firewallgenericout"=dword:00000000
"firewallgenericin"=dword:00000000
"firewallfailout"=dword:00000000
"firewallfailin"=dword:00000000
"ifacequeue"=dword:00000000
"godclienttx"=dword:00000000
"godclientrx"=dword:00000000
"godservertx"=dword:00000000
"godserverrx"=dword:00000000
"wanport"=dword:00000000
"wantx"=dword:00000000
"wanrx"=dword:00000000
"wanev"=dword:00000000
"iproutename"=""
"iprouteev"=dword:00000000
"iproutetable"=dword:00000000
"iproutetableev"=dword:00000000
"iprouteripin"=dword:00000000
"iprouteripout"=dword:00000000
"systemerror"=dword:00000001
"systemprint"=dword:00000001
"systemresource"=dword:00000000
"systemlicencing"=dword:00000000
"systemdevtrace"=dword:00000001
"MonIVRMessage"=dword:00000000
"MonSysOpMessage"=dword:00000000
"MonPerformanceData1"=dword:00000000
"MonPerformanceData2"=dword:00000000
"MonPerformanceData3"=dword:00000000
"MonPerformanceData4"=dword:00000000
"MonPerformanceData5"=dword:00000000
"MonPerformanceData6"=dword:00000000
"MonPerformanceData7"=dword:00000000
"MonPerformanceData8"=dword:00000000
"MonPerformanceData9"=dword:00000000
"MonPerformanceData10"=dword:00000000
"MonPerformanceData11"=dword:00000000
"MonPerformanceData12"=dword:00000000
"MonPerformanceData13"=dword:00000000
"MonPerformanceData14"=dword:00000000
"MonPerformanceData15"=dword:00000000
"MonPerformanceData16"=dword:00000000
"MonPerformanceData17"=dword:00000000
"MonPerformanceData18"=dword:00000000
"MonPerformanceData19"=dword:00000000
"MonPerformanceData20"=dword:00000000
"MonMemoryData1"=dword:00000000
"MonMemoryData2"=dword:00000000
"MonMemoryData3"=dword:00000000
"MonT1Line"=dword:00000000
"MonT1Channel"=dword:00000000
"MonT1Dialler"=dword:00000000
"MonT1DSP"=dword:00000000
"MonT1CAS"=dword:00000000
"MonCSULineLoopback"=dword:00000000
"MonCSUPayloadLoopback"=dword:00000000
"MonCSULoopbackOff"=dword:00000000
"MonCSULoopbackLine1"=dword:00000000
"MonCSULoopbackLine2"=dword:00000000
"MonCSULoopbackLine5"=dword:00000000
"MonCSULoopbackLine6"=dword:00000000
"MonR2Line"=dword:00000000
"MonR2Channel"=dword:00000000
"MonR2Dialler"=dword:00000000
"MonR2DSP"=dword:00000000
"MonR2CAS"=dword:00000000
"atmcmline"=dword:00000000
"atmchannel"=dword:00000000
"atmio"=dword:00000000
"vpnrip"=dword:00000000
"vpnnode"=dword:00000000
"vpnremotes"=dword:00000000
"vpnforwards"=dword:00000000
"vpnriptx"=dword:00000000
"vpnriprx"=dword:00000000
"vpntftptx"=dword:00000000
"vpntftprx"=dword:00000000
"h323tx"=dword:00000000
"h323rx"=dword:00000000
"h323ev"=dword:00000000
"h245tx"=dword:00000000
"h245rx"=dword:00000000
"rastx"=dword:00000000
"rasrx"=dword:00000000
"h323fs"=dword:00000000
"ccmstx"=dword:00000000
"ccmsrx"=dword:00000000
"h323whole"=dword:00000000
"siptrunkdata"=dword:00000000
"siptrunkinfo"=dword:00000000
"stuninfo"=dword:00000000
"ldaptx"=dword:00000000
"ldaprx"=dword:00000000
"ldapev"=dword:00000000
"filtipsource"=""
"filtipdest"=""
"filtmacsource"=""
"filtmacdest"=""
"filtbcast"=dword:00000001
"filtplsize"="1500"
"ifarpallow"=dword:00000001
"ifWAN3pkts"=dword:00000000
"ifmcast"=dword:00000001
"snmprxmsgpro"=dword:00000000
"snmptrapgen"=dword:00000000
"snmpvarbindpro"=dword:00000000
"ipsecevent"=dword:00000000
"ipsecrx"=dword:00000000
"ipsectx"=dword:00000000
"ipsecWarning"=dword:00000000
"ipsecDecode"=dword:00000000
"ipsecDataEv"=dword:00000000
"ipsecDebug"=dword:00000000
"ipsecSubtending"=dword:00000000
"l2tpevent"=dword:00000000
"l2tprx"=dword:00000000
"l2tptx"=dword:00000000
"klAppearance"=dword:00000000
"klAppearanceGroup"=dword:00000000
"klCall"=dword:00000000
"klFacility"=dword:00000000
"klLocalAppearance"=dword:00000000
"klPhoneStateChange"=dword:00000000
"klSwitchHook"=dword:00000000
"klUserConfig"=dword:00000000
"klCallInterface"=dword:00000000
"klKeyResolver"=dword:00000000
"klCallInterfaceLevel2"=dword:00000000
"klKeyResolverLevel2"=dword:00000000
"klExcludeClassAdapter"=dword:00000000
"klExcludeClassPhone"=dword:00000000
"klExcludeClassAppearance"=dword:00000000
"klExcludeClassAppearanceGroup"=dword:00000000
"klExcludeClassAppearanceSelector"=dword:00000000
"klExcludeClassRinger"=dword:00000000
"klExcludeClassSwitchHook"=dword:00000000
"klExcludeClassCall"=dword:00000000
"klExcludeClassLine"=dword:00000000
"klExcludeClassPrimaryCoverageFacility"=dword:00000000
"klT3ApiEvent"=dword:00000000
"klT3ApiMessage"=dword:00000000
"klT3PhoneModel"=dword:00000000
"CtlStreamTx"=dword:00000000
"CtlStreamRx"=dword:00000000
"DHGRoute"=dword:00000000
"HGE"=dword:00000000
"HGLII"=dword:00000000
"HGSC"=dword:00000000
"HGCFG"=dword:00000000
"ssiRequestMessages"=dword:00000000
"ssiReplyAndEventMessages"=dword:00000000
"ssiObjectEventMessages"=dword:00000000
"ScnUsers"=dword:00000000
"ScnDump"=dword:00000000
"pcvmail"=dword:00000000
"voiplevel"=dword:00000002
"primlevel"=dword:00000002
"MonCSULoopbackLine9"=dword:00000000
"MonCSULoopbackLine10"=dword:00000000
"MonCSULoopbackLine13"=dword:00000000
"MonCSULoopbackLine14"=dword:00000000
"ssiStartOfRequestedData"=dword:00000000
"Top"=dword:00000079
"Left"=dword:00000033
"Width"=dword:0000036d
"Height"=dword:0000020f
"triggerlog"=""
"triggerprint"=""
"triggerstringautoro"=dword:00000000
"triggerstringmultiro"=dword:00000000
[/blue]

 

[573dawn]

ronromano, yes, as far as I know it is SCN station to station, it is not an IP trunk from a carrier, but just a line that runs across a PTP T1.

We have looked at the timing on the PRI. I can occasionally see clock slips, but they don't correspond witht the drops. I can also occasionally see "trunk out of service", but when that happens, ALL calls drop, everywhere. ATT has tested this thing over and over (they really hate hearing from me), and we do occasionally see out of service seconds.

In fact, throughout all of this saga, we have IMPROVED the situation, by cleaning up config errors, testing things into oblivion, straightening out the mesh configs on both the IPO and the cisco routers, we are seeing far fewer drops than we have in the past. BUT...

It is the remining drops that have this odd cluster pattern and can be correlated to the Media Exception errors. This week has been the cleanest for configurations, and also has been the most obvious for a pattern of any kind.

 

[573dawn]

intrigrant, maybe I;m not doing this correctly, I copiued and pasted his into notepad, saved with a .reg extension, and tried to merge, it told me that the file isn't a registry script and that I can only import bibary files from within regedit. However, if I try to import, it wants to overwrite all the values in that location, and still gives me an error telling me it's not a registry file. Any assistance would be most appreciated.

And no, I'm not sure the Qos is ok, I don't understand the dscp enough to know what I am seeing on the other thread.

 

[573dawn]

intrigrant, nevermind on the .reg file, I saved it on a W2K machine and tried to merge it into an XP machine, that wasn't goping to work. Saved on XP and merged just fine I forget that my computer is W2K sometimes...

 

[intrigrant]

You can find the .reg file here:

http://intrigrant.atspace.com/Monitor.reg

And a little doc about Qos:

http://intrigrant.atspace.com/Quality

of Service.archive

change the .archive to .zip and unpack the word document.

 

[573dawn]

intrigrant, I can't get to your QoS link, I keep being redirected to a free hosting site. I tried without the /Quality, but didn't see a link to what I need. Any other way to get to it?

Thanks, Dawn