Jitter on one leg of SCN 2

[eltorodeoro]

Experiencing bad jitter on one leg of a multi-site SCN. the 403 -3.1(48)- is constantly generating the following:

15058mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS
15839mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS
17888mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS
18664mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS
19433mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS
20141mS PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS

we're using G.729, and when pinging over the same leg, its a bit bursty, but really nothing over 30-40ms delay.

Any ideas on what the errors above mean?

 

[eltorodeoro]

I'm submitting this problem to you again... It seems that the Callback messages are just the IPO's way of complaining about delay. After further investigation, these messages are generated when the IPO is putting out a BLF broadcast to port 50799. We've replace the 403, the VCM, changed the port on the Cisco switch, changed the cable. All other hosts on the subnet experience no such packet delay.

This is severely affecting the quality of VoIP calls back to the main 412.

Any suggestions? Should we waste time upgrading to 3.1(56)? Could this be a symptom of a corrupt config? Is there a way to disable the 403 from conducting these BLF broadcasts every second?

Please Help!!!

 

[intrigrant]

Well, the BLF data is send as a UDP broadcast. This implies that IPO will not check if data is received by the hosts ( PhoneManager or SoftConsole ).
So delays on the network are of no concern in relation to your problem.
The message "PRN: Callback ../tftp/tftpsvr.cpp Line 3101 took 195mS" means that a software module within IP Office called tftpsvr.cpp has problems executing a command on line 3101, the message comes most likely from a watchdog mechanisme in the IP Office core software.

Whats the use of this info?
You should check your configs on duplicate names or numbers, each name and number within the SCN must be unique.

 

[eltorodeoro]

Much thanks, bro.

That only goes for users, right? Lines and Hunt Groups don't fall into the unique category, right? Maybe this is a stupid question, but does that include the NoUser and RemoteManager Users? Should they be NoUserA, NoUserB, etc.?

Not to argue with you, but there is a direct correlation between the BLF broadcast, that Callback message, and these "spikes" in packet delay whilst pinging the IPO. Just coincidence?

 

[intrigrant]

The NoUser and RemoteManager are no real phone users so these names don't matter.
Hunt Groups do fall in the same categorie as users, so also unique names and numbers.

The BLF thing is not a coincidence as i think that the tftp server is blocking the network interface for 195 ms wich the users will notice as a "spike" in their conversation and it will casue a delay while pinging.

At first i would make a copy of the config and then erase the whole unit ( config and firmware ) with a serial cable and then reinstall the firmware and the config.
See what happens. Also check if your config does not exceed the 192 kb limit and your SCN does not have more then 500 users.

 

[eltorodeoro]

Excellent. Thanks so much - I finally have an explanation that makes sense!

 

[eltorodeoro]

ummmmm... so what happens if the SCN > 500 users?

 

[h382]

Small Community Networking

When connecting IP Offices together over IP or Packet based networks, Small Community Networking enhances feature transparency. These networks can support up to a maximum of 500 users across 16 sites. The following additional features are available.

*
o Busy Lamp Field.
o Camp-on.
o Call Back When Free.
o Paging.
o Call Pick-up.
o Centralized Voice Mail (Voicemail Pro).

Support for mailboxes, call recording, dial by name and auto attendants. Queuing on remote systems is not supported.

o Internal Directory.
o Absent Text Message.
o Anti-Tromboning.

If larger networks are required QSIG can be used to link multiple Small Community Networks together. Functionality between the communities is governed by the QSIG feature set.

 

[eltorodeoro]

i know i read that somewhere. guess i overlooked the 500 users thing. is this a realistic cap, or is this just one of the arbitrary numbers that avaya throws out? what symptoms will begin to show for an overtaxed SCN? will the less powerful boxes - for instance the 403 in question - be more succeptible to buggy issues?

BTW, we've tried just about everything on this bloody trouble. one thing that stands out is that the problem seems to coincide with one or two users connecting and disconnecting their laptops. any thoughts?

intrigarant?
sizbut?
IPGURU?
anybody?

 

[h382]

Make sure no one is running an anti-spam and/or network analyzers type of software. I've had situations where those type of programs were killin' the IP Office.

 

[JeSTeROCK]

h382 can you let us know what type programs and what symptoms the ip office was having ?

 

[h382]

I don't remember the exact program name, I don't think my business partner actually gave it to me, but he was getting dropped calls, static, BLF Updates weren't updating. You could see the activity lights on the LAN ports going crazy too.

 

[intrigrant]

you can find out with using a network sniffer like ethereal.
Sysmon won't do as it seems the ipo is overloaded with ip messages.

 

[diannacone]

Just out of curiousity, do you have both the Voice Network and Data Network running on the same subnet?

If so this would be one BIG reason why your seeing these issue's.

 

[eltorodeoro]

they are on the same subnet, and i've brought that to the attention of my boss. only my third week on the job - oh, how these boxes can initiate a newbie!

BTW, we isolated it to one user's laptop. we segregated her off to her own private network today, and guess what? no problems! still haven't got concrete proof by plugging it in to watch the errors. once i have more evidence, i will update.

thanks so much for all the insight!

 

[sizbut]

The main problem symptom you will experience with an SCN over 500 users is disconnects, ie. Avaya putting the phone down on support calls.

 

[eltorodeoro]

funny. that's kinda what i figured. we're at 550 now, with another four sites to add. guess its time to upgrade! what a shame!

as an aside - since we're talking practicalities - the documentation specifies that an SCN should be hub and spoke. has anyone successfully accomplished a mesh? does it really hose things, or is it that they don't want to support something so messy?

 

[sizbut]

A small mesh, 3 maybe 4 can work, but the chances of issues and problems increases exponentially with each node added - hence why Avaya quickly switched away from supporting it as a option.

 

[IPGuru]

On laptop causing the probles sugests posibly a virus or other spyware trying to "phone home".
check for aplications & features cush as enhanced emoticons as these are usualy nothing more that a disguse for spyware.
alternativly perform a scan with ad-aware or similar.