SCN on 4.0.7 and 4.0.10 1

[SuperJenks]

I currently have a customer with 7 sites scn'd together. They were using 3.2.57 on all sites and never had any issues with call quality.

Last thursday, we ugraded the equipment at the main site to an IP500 which obviously requires 4.0 or above. So, I went through the motions of upgrading all sites to 4.0.7 (I was to scared to use 4.0.10 because of issues described here on IPO forum). After the upgrade, all site immediatly had call quality problems, so bad that you could only get one way transmittion at times. I began trouble shooting and found that the ping response to all remote IPO systems were very high (in the 1000ms range at timee) but spuratic. I adjusted codec settings to from G711 to G729 to help with the overhead of the packet size. This made no difference. So, I had one option left, Upgrade to 4.0.10. So, I did, I upgraded all sites and the problem was definetly reduced, but not completely. After more trouble shooting, I found a pattern to the madness. It seems everytime the main system would broadcast any SCN relative information to the remote sites, the pings replies would immediatly become very long, up to 4000ms at times (as if the remote sites were using alot of system resources to process the scn data), but then would clear up until the next scn broadcast. I validated this by watching monitor to see when the broadcast took place from the main site, I would then look at my constant ping I had running, and it would then start lagging bad after the SCN broadcast.

The network is supported with 10meg Point to Points (Way more than enough bandwidth) with QOS. Also, The site with IP500 has a brand new config that I basically had to rekey because of the hardware ugrade change.

Has anyone experienced this problem. If so, please explain how you resolved it, a patch perhaps, or is there possibly a setting Im overlooking that could fix this. It seems entirely software related. If I cant get it resolved with avaya, I will be forced to pull the equipment out and return the network to the way it was in the beginning. Thanks in advance for your replies.

 

[VinylDave]

Hi SJ
Sounds very strange. Every IP endpoint at each end I presume has a reply of less than 29ms but the IPO has up to 4000ms?
What is different in the 3.2.57 build on SCNs compared to 4.0.10. It might be an idea to dismantle the SCN and reestablish it from start.

 

[SuperJenks]

According to avaya tier3, the difference between 3.2 and 4.0.7 and up is the H.323 protocol settings are a bit different.

 

[SuperJenks]

We figured it out! The remote sites had backup IP lines incase the main site went down so they could reroute traffic to another facility containing a backup VMPro. This was confusing the scn and causing multiple scn broadcasts which caused the boxes to lag real bad.

 

[iposcn]

SuperJenks, I believe I am having a simlar issue with our SCN. I have 2 x 500's and one 412, though the 500's do not seem to suffer from the ping issues, the 412 does. It is very sporatic, but seems to happen mostly when a user hotdesks into a remote site.

We do not have a backup VMPro server. Was the issue you had related to multiple IP lines specifically or something related to ARS?

 

[SuperJenks]

The problem was all the remote sites had more than one IP line programmed. One IP line was for the primary IP connection to the Centralized site. The other IP line was supposed to be a dormant backup connection to the customers secondary location if the main centralized site went down. The second backup IP line was built out at all remote sites and was confusing the systems as to who the centralized site. If you have more than one IP line built into the config, they system will think its the centralized site. Because that was the case, all the sites were broadcasting to the rest of the systems based on the IP lines that worked fine in 3.2, but as soon as we upgraded to 4.0.10, the networked bogged down because of all the SCN broadcasts.

 

[iposcn]

SuperJenks, thanks. So for example this is how my setup is currently:

412 (Main Site)
500(a) Remote Site 1
500(b) Remote Site 2

Each Site has two IP lines that point to the other two sites, should I remove the IP lines on Remote1 and Remote2 that point to one another? And if I do that, will Remote1 and Remote2 still know about each other? (dial direct)

 

[SuperJenks]

Yes, you should have only have one IP line on the remote systems to the centralized site. Make sure voice networking is checked on and direct media path as well. The centralized system will share the information between all the remote sites with a scn broadcast.

 

[iposcn]

SuperJenks, Thank you very much. Did you have any IPO's that had "odd" response times on the local LAN during that situation? Our 412 (central site) has ping response times that vary between 1-12ms, averaging about 6ms. I have verified the Speed and Duxplex settings on the switch, which is set at 100-Half-Duplex.

 

[SuperJenks]

In my situation, all the sites would respond normally for a minute or so, then the pings would jump from 12ms to 3500ms or more. This is how I coralated the scn broadcast problem. I would watch them monitor, everytime the scn broadcast would go out to the systems, the pings would get real bad at all the sites at the same time. We saw the partern and quickly relized it was directly related.

 

[iposcn]

Thanks SJ, I will try removing the IP Lines tonight!