IPO + Chronicall - Missed Packets

[liquidshokk]

Tearing my hair out trying to resolve an issue with Chronicall reporting missed packets in it's tomcat logs. This is causing agent's status across 6 sites to be incorrect and for calls to be shown as being in progress for many hours, as it sometimes doesnt see the hang up message.

Have spoken to Xima, who say it is a network issue, however the customers network is a PWAN so should be completely open between sites! The PWAN provider has provided proof that there isnt any packet loss on the network, however my understanding is that this will only relate to TCP traffic and Chronicall takes the UDP stream from each system. The only hardware at each site is the IPO, a flat data switch and a cisco router for connecting to the PWAN.

Anyone had similar issues before or know of a way to prove that there is UDP packet loss on the network??

 

[amriddle01]

From R10 Xima has to use the CTI licence and the 'DevLink3' API interface, so perhaps that may solve things for you, however R10 will probably kill the IP Office, so weigh up which is worse

 

[Jimmyjoe1975]

I had the same experience and Xima told me the same.
UDP is not checking if the message is delivered, while TCP do it.
So the result is calls with hours of conversation because maybe the drop call message was not delivered to Chronicall.
From my point of view is designed really bad, instead using TCP are working with UDP which is much worse.
In my case lately it happens less and less but I can not advice you if it was because a network improvement or because there is less calls in the Call Center.

 

[AvantiWade]

What proof of no packet loss did the provider provide?

Just because its a private wan connection doesn't inherently mean its any more reliable of a transport. Its still running over the same network on the providers side that normal traffic is, its just prioritized and tunneled differently.

 

[mattKnight]

>however my understanding is that this will only relate to TCP traffic

No, dropped packet counters on routers do not discriminate between UDP and TCP traffic.


>UDP is not checking if the message is delivered, while TCP do it.
UDP is fast,as the connection overhead can slow down traffic. This is why it is used particularly for stream packets, error checking and recovery is at the application layer

>From my point of view is designed really bad, instead using TCP are working with UDP which is much worse.
If UDP is the only thing available, they don't have much choice. The monitor interface is what is says - a monitor interface and not a CTI feed

Take Care

Matt
I have always wished that my computer would be as easy to use as my telephone.
My wish has come true. I no longer know how to use my telephone.

 

[liquidshokk]

Good to know Matt, cheers

The PWAN providers ran some tests and provided some graphs which do seem to show no packet loss. They also connected to one of the routers and sent 300mb across the network and no loss was reported.

We installed Chronicall to replace Tim+ but the customer is fast losing all trust in the information being reported... surely there must be some way of identifying why packets haven't been received. There's plenty of entries in the tomcat logs!

 

[amriddle01]

Only way to know where packets go missing is to log traces at every hop and find at which point a specific packet fails to show up, you still won't know why though just that it didn't come through...... good luck with that

 

[twitchy087]

What version of Chronicall are you running? I am currently having the exact same issue with a 5 site customer. Xima has provided me with the same feedback, and I currently have a TT open with the MPLS provider. I will provide you with any update that i receive however I am not convinced that this is in fact a network issue. We have no issues with voice, voicemail, or rapid pings across the MPLS to any site.

What version of Chronicall are you currently running? I am currently running 3.6.24 at my customer.

 

[hairlessupportmonkey]

Only way to know where packets go missing is to log traces at every hop and find at which point a specific packet fails to show up, you still won't know why though just that it didn't come through...... good luck with tha

Actually thats not a good way to prove packet loss. Simply because routers will not respond to ICMP requests if they are busy. ICMP has the lowest priority on a network.

When I was doing this, we used to see a fair amount of it too. Packet loss on a LAN really isnt an excuse. not for a fairly light UDP stream. eventually Xima fixed the issue. It was more to do with the time packets were sent then arrived from point to point and were sometimes out of order. Xima fixed this in some releases.

ACSS - SME
General Geek

 

[amriddle01]

I didn't mention ping, I'm talking wireshark traces/captures not ping traces

 

[liquidshokk]

Currently on 3.6 (51)...in a way I'm glad I'm not the only one going through this issue! Sorry Be great to hear of any developments your end please. Might point out to xima that there are other people experiencing the same.

Maybe it's specifically an MPLS issue? We don't have enough installations out there to know if this is an issue in smaller setups

 

[Westi]

here the question that occupies me on this, does the local IPO have the same issue or just the remote ones, related to the Xima server as local?
If all of them have the issue it may just be a lousy patch cord to the server or a faulty switch port or NIC. If only the remote (or worse only some of them) have that issue then it is packet loss of some sort.

Joe W.

FHandw, ACSS (SME)


"This is the end of the world, make sure to buy your T-shirt before it is too late"
Original expression of my daughter

 

[twitchy087]

Hi Westi,

I am not seeing missed packets in the logs for the local IPO however, reports are not correct for users working off of the local IPO. You are right in that it could be something silly like a patch cord, but wouldnt there be other issues with voicemail if that were the case?

So far, I have nothing else of substance on my end.

 

[liquidshokk]

Not seeing missed packets for local site either. Am getting a LOT of the following errors though which don't speciy an IP address;

Failed to get list of active calls. Continuing as though there are no active calls.
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [318] from call with following ids: [320, 321, 318, 319]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [321] from call with following ids: [320, 321, 319]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [74] from call with following ids: [76]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [74] from call with following ids: [76]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [76] from call with following ids: [76]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [180] from call with following ids: [180]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [179] from call with following ids: [180]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [179] from call with following ids: [180]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [223] from call with following ids: [223]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [222] from call with following ids: [223]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [222] from call with following ids: [223]]
Fri Feb 05 09:30:43 GMT 2016 [Missed Packet --> dropping switchCallId: [285] from call with following ids: [284, 285]]

 

[Johnhyde]

I'm having the same issue too! Xima claim there are packet loss hence the logging of the calls are duplicated.
Yet I don't have any issues with the network in terms of voice or data, only affecting one site out of three!

 

[liquidshokk]

How long has Chronicall been about? Has it been a stable product for those who use it in the past?

Getting reports that data isnt matching up with call loggers it has replaced, call answered percetages arent correct, i.e 18 out of 20 calls answered reports as 58.1% answered!

Hoping it's just a dodgy software level going about as we are selling lots of licenses at the moment!

 

[sizbut]

Probably better to start a new thread rather than tag a different issue onto the end of an existing one liquidshokk.

Its hard to compare call loggers without a lot more details. Whilst you might think answered calls is a pretty simple definition I can come up with several different definitions, all of which would cause the same set of calls to yield different results (for example 6 of you calls were answered by voicemail, some will call them answered, some might call them missed).

So without knowing the rules by a call logger is applying its hard to make any useful comparisions.

Stuck in a never ending cycle of file copying.

 

[liquidshokk]

Was just a general question about Chronicall whilst thinking about this particular issue. For the same customer the realtime info in realtime statistics isn't updating itself and often shows users as logged out even though their DND timer is increasing when they are in DND or it suddenly changes to ringing/talking when they are presented with a call.. Hoping this is linked to the missed packet issue and not just the realtime features being unreliable!

In the case of the reports not matching up I am waiting for old and new reports to be sent to me so I will start a new thread if there isnt a logical reason for the differences...

 

[sizbut]

Even if your system was working perfectly, its still a different question (well questions 1) is Xima reliable 2) How does Xima measure 'answered calls'.


A basic call logger will be binary - a call is either answered or not. A more sophisticated call logger will start to try to account for things like "did it actually ring for more than a seconds before the caller disconnected" (ie. give the person at the other end a chance to answer), "did it ring for their full answer time before going to voicemail/whatever" (ie. a chance to sprint across the office and answer before the system decided I wasn't going to anyway and routed it elsewhere).

There are plenty of folks here who can probably say what counts as an 'answered' call in Xima reports but we don't know if your other call logger is using the same set of rules.

Stuck in a never ending cycle of file copying.

 

[liquidshokk]

Thanks

In the case of the group call summary showing Calls presented =20 Calls Answered =18 Percentage of calls Answered =58.1% you can see why I think its inacurate, or mis-leading at the very least. Thats without comparing it to other calls loggers (which I appreciate get their information by SMDR and therefore interpret some call events differently)

The reason I brought the other issues up here is because they back up the impression our customers are getting that the software is potentially flawed, certainly in the example I have given above in the fact I have pointed out a couple of other bugs which their developers are going to work on too..

Xima confirmed the changes that are coming with V10, which can't come soon enough for us. Just a shame we will have to convince everyone with Chronicall to pay for the upgrade licenses lose certain features etc just to get chronicall working properly...