IPO & Zeacom - Calls Dump at Handoff

[marcberm]

I'm working on an implementation of an IPO 500v2 at 8.0.443001 with VMPro and heavy Zeacom integration. The problem we're having is that any incoming SIP calls are getting dumped at the handoff to the Zeacom queue.

The caveat, is that this ONLY happens with SIP, not POTS, and even then ONLY when the Zeacom queue involves a recorded announcement. The system is live. As a temporary workaround, we've got an unconditional forward on the SIP lines to point them out SIP and back into the main POTS number taken out of the old PBX. Those lines are still working as long as they keep paying for them. We've maxed the trunk capacity on the IPO using 8 of these lines which are in circular hunt. The extra lines are just shorted-out to bypass them. It's working, but the hairpin forwarding causes an undesireably (if not unacceptably) long lag before hitting the queue.

Zeacom has duplicated the problem in their lab, and we have another IPO with Zeacom implementation scheduled to go live this Saturday. That one is also SIP, and has NO analog lines at all.

One thing we are arranging with the provider (this Friday morning) is to change the SIP setup so it connects to the client's network, vs. separately into the second LAN on the IPO. We're thinking that may solve the problem, but even the engineers involved on all sides of this are unsure.

I am curious if anyone else has encountered anything even remotely similar to this, and what the solution was? Any thoughts or ideas are greatly appreciated!

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[Westi]

how is the Zeacom integrated? SIP? H.323 trunk? analog extensions?
if so you might want to try and make sure that the incoming protocol matches the internal one so that the calls are not stressing the VCM channels.
You could also try and use a different release of 8.0 but then you run into problems with POTS lines (thanks Avaya)
If you run Monitor on SIP do you get any disconnect messages or other funky redirects?
If reinvite is checked on the SIP trunk take it off, if it is not checked then do it. I know it is a pain but going that way is the only way to find a work around. Make sure you have enough coffee around as that will be after hours work.
Just for testing you could use a SIP extension and try if that also causes the same problem, that would show if the trunk has an influence or SIP is to blame somehow. If you don't have any issues with a SIP extension then it is probably some funny signalling or encoding on the trunk side. But Monitor is your good friend for troubeshooting in that case. You are sitting between two entities (Trunk on one side and Zeacom on the other) and people will blame the IPO even if it is not the actual culprit so make sure you have lots of logs. Maybe you want to post a log file here (take the original numbers and passwords out) and we can have a look. Here are some guys that are absolutely fabulous with Monitor.

Joe W.

FHandw, ACSS

http://convergednetworks.ca

 

[marcberm]

The Zeacom queues are SIP extensions and protocols match. We've tried almost every imaginable combination of settings on the IPO (DMP on/off, Re-Invite, codecs, etc.) Currently DMP, Re-Invite, and Reserve 3rd Party EP Lic. are on, and the codec is hard-set to G.711 ULAW which is what Zeacom is expecting.

The only notable thing in the traces is that we seem to lose all SIP header and IP info on the redirect when the announcement should be going to the caller, which is also when the calls drop. We will find out on Friday if it's getting stripped somehow due to SIP and the network (and hence Zeacom) being on the separate LAN interfaces. If that's not it, then maybe the TAPI handoff from Zeacom, but then... We're out of ideas.

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[marcberm]

Also, we're unable to reproduce the problem on any other SIP calls whatsoever, and have tried some SIP softphone programs which also work normally.

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[Westi]

Have you had a close look at your network setup, is the Zeacom able to connect to the SIP provider?
Is the port 5060 routed incoming directly to the IPO?

Just fishing here because I know you guys have tried a lot already and in a live system it is always harder to test anything.

Joe W.

FHandw, ACSS

http://convergednetworks.ca

 

[marcberm]

Unfortunately, we are just fishing at this point. The provider had to cancel for this morning, so we're rescheduled for next Wednesday. It will be a little bit longer before we know of the SIP change has any effect.

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[marcberm]

Our coordinated attempt with the provider did not go well on 10/3, as there were some related but unrelated problems with the provider's core switch that brought SIP down entirely for a good portion of the day. We were able to creatively route around this until we had service restored. We will be attempting the configuration change again tomorrow evening to avoid impact to the business if things go wrong (as happened on last week's early morning attempt). I will keep everyone posted. In the meantime a similar cutover (different provider) with Zeacom and SIP went well - the only difference being the change we're trying to make in this case. We're really hoping that does the trick.

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[Westi]

Thanks for keeping us updated Marc

Joe W.

FHandw, ACSS

http://convergednetworks.ca

 

[Billz66]

We attempted to get Sip handsets working as call centre agents and had similar issues with integration failing.

In the end we gave up on having any sip endpoints and changed them back to analog extensions.

I would make sure that you have the latest version of zeacom software and it will probably need a patch written to make it work.

If I never did anything I'd never done before , I'd never do anything.....

 

[CarGoSki]

Zeacom is a Devconnect partner. If they think it is a IPO issue then they should open a case with Avaya.

 

[marcberm]

It's been a while, and I figured I owe this thread an update.

We finally appear to be getting somewhere, though the problem remains unresolved. Another project manager for another Avaya business partner on the west coast actually reached out to me having the same exact issue. Fast-forward through many, many discussions and conference calls involving almost every conceivable combination of involved parties, and the progress distills down to something like this:

[ul][li]We installed a Dialogic board in the Zeacom server and changed the method of call handoff to analog (removing the ZCC queues-as-SIP-extensions piece from the equation). We STILL saw that SIP calls would drop at the TAPI redirect. Everyone was floored.[/li]
[li]Something, somewhere causes the IPO to try passing bad info (an IP of 0.0.0.0), which the provider never likes in both the working and non-working cases. The difference, we noticed, was that in the working example, the first, bad INVITE was followed by a re-invite, bearing a correct/negotiable IP address.[/li]
[li]Following some knowledge transfer between business partners, the similar cases with Avaya Service and DevConnect, though not technically linked, begin cross-referencing each other.[/li]
[li]Further investigation leads to the revelation that our working example's provider supports re-invite, while our non-working example's does not. Interestingly, our working example and the other business partner's non-working example have the same provider. The trouble is that being on opposite ends of the country, they register to different data centers with differing core equipment.[/li]
[li]So, where we are today? Each of the tickets (our case with Avaya, Zeacom's case and DevConnect ticket, and the tickets related to the other business partner's case) have all reached Tier 4 at Avaya, where R&D has been engaged. We may or may not have an update before Thanksgiving.[/li][/ul]

At this point, it's looking like the ultimate solution will be (in order of likelihood, which also happens to be order of preference):
[ol][li]A patch from Avaya[/li]
[li]Change to PRI[/li]
[li]Provider change[/li][/ol]

As you can imagine, we're really hoping for #1, and I don't think anybody really wants to have to resort to #2 or #3.

On the evening after the unsuccessful switch to analog handoff, my technician remarked: "It was a balmy 80 degree day when I came here to do this rack and stack, now leaving here tonight there's snow on the ground."

Clearly, this has been an open issue for quite some time now, but thanks to a lot of really great people who've put in a lot of time working on this in various capacities, we are hopefully about to see it come to a close.

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries

 

[marcberm]

Earlier this week, Avaya provided a critical patch to address our TAPI redirect issue. Since application of the patch, the IPO is now passing correct IP information to the SIP provider and our issue appears to be fully resolved. As of Wednesday evening, the client was fully transitioned from our creative analog routing to the SIP trunk for all inbound calls.

Thank you to all who offered their thoughts, ideas and insights via this thread, as well as all those who have worked extremely hard to finally bring this issue to resolution!

Marc Berman
Small and Medium Enterprise (SME) Project Manager
Carousel Industries