IPO Small Community Networking Issues

[usfregale]

I've been the proud owner of two IP Offices for about two years. I"ve spent a great deal of money buying them and a small fortune trying to make them work. That being said -- they do not function properly 80% of the time.

There are two primary problems:

1. We have a series of about 40 Avaya 5601 IP telephones at remote sites. Each site has one, possibly two phones (most have one). Periodically, these phones lock down -- that is they simply stop functioning -- like they have lost their connection to the IP Office. The VPN between the sites is functioning perfectly and the low latency queue (QOS) between the sites reveals no tail drop or other comparable issues.

Sometimes a phone will lock down after being up only a few minutes, sometimes a phone will be up for days or weeks and then lock down. A small portion of the phones that do lock down are rendered permanently unusable. That is they are never again able to connect to an IP Office even when they are local to the IP office, even after a RESET command is performed on the phone.

The VPN tunnels between the sites continue to function properly and other data traveling between the sites appears unimpacted.

2. Our second problem pertains to the Small Community Networking connection between the two IP Offices. We have SCN configured between the two sites and it is properly licensed. However, at one site the SCN status in manager always shows as "Enabled" and at the other it always shows as "Up". Obviously they should both show "Enabled". We have checked the ports between the two sites and confirmed that the VPN tunnel between the sites is open. The two sites are unable to communicate using SCN -- the only way to currently route calls between sites is using short codes pointed to the IP lines, which is very restrictive.

Any help that anyone could povide on either of these issues is very helpful. I"m sure you'll want more information, just ask and I'll tell you anything I can. As I mentioned above I'm very frustrated with the entire process and the fact that we're two years into the process and we're still having the same two problems that we've been working on since day one.

Richard

 

[Westi]

question 1: what types and releases are the IPO's
question 2: what firmware are the 5601 phones
question 3: do you have any trouble with the local 5601's and what release are they

Manager doesn't show you status it shows you programming, the system status application (R4.0 and above) will show the status or the monitor application can do it too.

Joe W.

FHandw., ACSS

insanity is just a state of mind

 

[hairlessupportmonkey]

also, the issue where you have to force calls with shortcodes, might suggest some duplication between sites?

do you also have your HG's advertised?

If you have spent this much time and money on your investment, it might be worth asking a good BP to look at it for you - even if its just to consult on programming etc.

 

[kwing112000]

What is the equipment your using to create all the VPN tunnels?

Kevin Wing
ACSS Small and Medium Enterprise (SME) Communications
ACS- Implement IP Office
ACA- Implement IP Office
Carousel Industries

 

[usfregale]

Our VPN is built using a Cisco ASA 5510 at a central location and Cisco ASA 5505s at each of the remote sites.

Both IP Offices at IP500s and run version 5.0.8. We originally started on 4.x, but have upgraded since then in an effort to previously fix this issue.

How do I check the firmware version on the phones themselves?

How do I check if the hunt groups are advertised?

Also, I assume when you mean duplication you're referring to extension and hunt group numbering, at the central IPO the extensions run in the 25xx series and at the secondary IPO they run in the 3xxx series. The hunt groups at the central IPO are 20x-21x and the single hunt group at the secondary IPO is 500.

Richard

 

[jamie77]

You need to check hunt group and extn names and numbers and make sure there are no clashes accros all sites.

It all sounds to point to your VPN to me. Having said that, I have seen a 4602SW lock up on a VPN that a 4602SW+ is fine on. Never used 5601's so can't really comment.

The SCN issue will almost certainly be down to your VPN not allowing traffic through properly. Wireshark each end for the SCN AvRIP packets coming through. You should also see these being sent and recieved on Monitor.

I would also look at the BP that installed this. They have left you a non working system with no steos towards a fix. Unless your did this yourself.

You should not be continually putting your hand in your pocket to get this working.

Jamie Green

Football is not a matter of life and death-It is far more important!!!!

 

[usfregale]

The base underlying problem has been that our Avaya Business Partner always blames our VPN; however, we bring in our Cisco installer who tries to troubleshoot the problem with the BP and with Cisco TAC and always concludes that the VPN is completely open and is not inhibiting passage of SCN packets. It's been all the way up to level 3 at Cisco TAC The Avaya BP then forwards the issue onto Avaya's internal support and it dies for a few weeks until I contact them (the Avaya BP) again and we start the entire vicious circle once again. I've never gotten a straight answer as to what Avaya's internal support concluded on the problem.

Also, I upgraded both IPOs to 5.0.20 this morning hoping that that would somehow help the issue. It did not.

Richard

 

[jamie77]

Getting the wireshark traces will conculde if the VPN is failing the SCN or not. You should be seeing regular traffic to and from each IPO on port 50795 (AvRIP). These packets are the SCN updates. If you wireshark one end and see them leave the IPO but not appear on the wireshark at the other, then the VPN is at fault. If they do, then there is something wrong with the IPO's. I'm suprised Avaya Tier 3/Distributor support didin't ask for this already.

Jamie Green

Football is not a matter of life and death-It is far more important!!!!

 

[hairlessupportmonkey]

It could be your H323 inspect maps. If you have any turn them off.

Also disable the respective TCP Timeouts.

There were changes with the SCN from 4.x to 5.x that seemed to break a lot of SCNs after an upgrade.

Generally any packet "messing" from the firewall was the issue.

You could also, just delete the SCN Line and re-create them too.

 

[Westi]

also make sure you have not the same names on different sites not only numbers. SCN is tricky that way.
You can see in system status what release the firmware is.
If the VPN would fail I would expect that the phones don't register or have voice issues not that they lock up after a while.
the SCN is only between the two systems not between the sites with only 1 or 2 phones and the system so there should be no IP lines going to those sites because these phones register to the IPO from their location and the IPO sends all packets back via your default gateway which then takes over the routing
This is a very nasty problem and to find this you will need to have the wireshark traces otherwise you will not get anywhere really as I suspect that Cisco already has taken the h.323 fixups out.
If you have an Avaya BP ask them to escalate it to Avaya as they are clearly not qualified to troubleshoot this properly.

Joe W.

FHandw., ACSS

insanity is just a state of mind

 

[usfregale]

The IPO and management PC are presently attached to a switch that does not support port mirroring (this was done in a previous attempt to correct this problem -- at one point our Avaya BP blamed it on QOS as they didn't like our application of QOS based on DSCP, so we instead put the IPO and management pc/voicemail server on their own switch and vlan and applied QOS to that entire vlan). I will move them over to a managed switch tomorrow and mirror the ports and see what we get out of wireshark.

Thank you for the suggestions.

Richard

 

[tlpeter]

I also think it is the VPN to be honest.
When you put two ipo's traight on each other it works fine.
And now when this network is in between it does not work.

Can you give us your ip ranges from the network and the ip settings in the ipo's (including iproutes)

Check if H.323 fixup/inspects are turned of.
Also check if there are any H.323 helpers are on (needs to be off) ?

Is "direct media path" on or off on the iplines from the ipo's ?

How are they connected to each other (layout) ?



Homo sapiens non urinat in ventum

honey, i fried the IP Office !!!

Sarcasm, it's only one of the services I offer.

 

[amriddle01]

I would take one of the 5601's and connect it to the same switch that the IP Office is connected to, I would wager it will not freeze up at all.
We had a similar issue with two sites seperated by a "Cisco ASA Both end" VPN, needless to say we proved the VPN at fault by putting both IPO's on the same LAN and no issues ever showed up, back on the VPN only every other call would connect and handsets could not do DMP at all. The Cisco Engineer couldn't see anything getting blocked and never found the issue, he's still looking (months later)

ACSS (SME)
APSS (SME)

http://www.ipoffice.tel

"I'm just off to Hartlepool to buy some exploding trousers

 

[hairlessupportmonkey]

I also highly recommend the latest firmware on the ASA.

I haven't had any issues creating SCN via VPN with it.

 

[usfregale]

After a great deal of analysis of the outbound AVRIP packets. Neither IPO appears to acutally be sending any.

I'm attaching each of the IPO configs in case that might be helpful.

http://www.jtaxsc.com/Florence.cfg

http://www.jtaxsc.com/Columbia.cfg

Richard

 

[usfregale]

I'll add that I am considering taking advantage of the upcoming holiday weekend to bring both IP Offices onto the same network on Sunday for the purpose of precluding a VPN problem. There are a number of reasons this is undesirable:

1. The IPOs are remote from one another, so to bring them together for testing purposes and take the other back will consume an entire day.
2. I really don't want to work Sunday.
3. My wife won't understand.


So if we aren't able to find something wrong in the next few days I suppose I'll do that, but I would prefer to have resolved this problem by the weekend and have moved onto the next problem.

Richard

 

[Westi]

on first glance your Florence system has 3 digit huntgroups that are overlapping with users they are in teh 200 range and your users in the 2000, for example 220 is a huntgroup and 2201 and 2202 are users
also your units are not on the same releases you are varying from 5.0.20 to old 4.0.10 for a removed digital 8 card and the second expansion card (ATM16 is either also removed physically or not upgraded)
Columbia: shortcode 701 has the wrong line ID

what I don't understand is that the IP Office would not send any IVRIP packages at all becasue the lines are set to SCN and the IP route is good on both sides.

I go to bed and have another look tomorrow morning or better later.

Joe W.

FHandw., ACSS

insanity is just a state of mind

 

[tlpeter]

First i would remove those configs from the internet!

Then never use 3 and 4 digits.
Use 3 or 4 digits for groups and extensions.
So change the group numbers.

Also, like Westi said, remove the DS8 if it is not in the system and upgrade the ATM16 (if it is there)

Remove Obsolete licenses, this can cause strange issues.

Incoming call routes to shortcodes which then point to VMPro is not done (no need to do this)

You can send a call to VM:ModuleName instead of shortcodes.
You also do not need to send shortcode over the iptrunk for voicemail.


If this is the only way to get it working then you really have a network problem.
But i think you have that anyway because of the SCN troubles you have.


Homo sapiens non urinat in ventum

honey, i fried the IP Office !!!

Sarcasm, it's only one of the services I offer.

 

[hairlessupportmonkey]

Yikes all the Yellow and Red errors are making my eyes hurt.

Its good practice to also clear ALL your errors BEFORE sending back the configs.

Naughty!

 

[hairlessupportmonkey]

I also notice that one of your SCN lines has Direct media path on and the other has it off.

Turn them both off for now.

also, any reason for using G.723? bandwidth poor?

try using a different codec, like g729 for now, in case the ASA doesn't know what to do with G723? (just a hunch)