Audio problems on SV8500 apparently related to IP PAD cards

[phadobas]

So yes, many users started complaining about audio problems. They all use DT700 series phones. The complains, one-for-one, were about calls that included the IP PAD cards. No problem with IP-to-IP calls, no problems with TDM-to-TDM calls (we have some analog line cards, couple of PRI cards, etc) but always problem that goes between an IP terminal and some sort of TDM resource (like analog line or PRI trunk).

So I started making a bunch of test calls in my office that involved the PAD cards. Every time, I noted down, which card and which port on the cards was involved. Turned out that the problem was always on one of the IPPAD card (we have 3 32-port cards). Ok, lets reset it. Done. Re-test. No problem. Great.

Next day, new complains come in. Same procedure: bunch of test calls, nail the card. It's one of the other cards!!! Reset. Problem gone.

Today it's the 3rd card!!! What on earth???

Anyways, I don't know what's going on. I'm running a constant ping command to the IP PAD card that's currently acting up. No problems for the last 2 hours. No dropped ping. There don't seem to be a duplicate IP address on the network either. There was no major change on the network (we have multiple VLANs, phones on different subnets, routers in between subnets, needed ports open, etc, etc. we have had this setup for many years now, but these IP PAD problems just started showing up maybe a week ago. I can't imagine that all 3 cards just decided to give up...

Can someone think of a good reason this is happening?

 

[belevedere]

I ran into a similar issue on a IP fusion network. Turned out to be more pad channels than licenses. Are you getting dead air or busies?

 

[phadobas]

I get garbled audio and sometime dead air (no audio). Where can I check the license count? And what licenses am I looking for? It is true that I added a few phones.

 

[belevedere]

AACT should show you the licenses and registration numbers.
Garbled audio and dead air may be your network. Check ADPM for the port assignments, 10/100/auto. NEC suggests setting it to 100 fixed. Then check your data switch for the same settings.

 

[phadobas]

DPTR tells me this:

Hard phone Type A installed licenses: 1080, Programmed: 1275, Registered: 670
Softphone type A installed: 112, Programmed: - , Registered: 112
Standard SIP installed: 290, Programmed:-, registered: 174

Everything else is 0 installed, 0 programmed, 0 registered

 

[belevedere]

AACT shows what the system is licensed for. DPTR just shows IP phones.
Try these commands;
FLNK = PAD trace command.
SPTS = PAD use.

 

[phadobas]

This saga continues. Random audio problems related to calls that involve IP PAD cards. I have made like a 100 test calls between a tdm and an IP phone, and used the FLNK command each time to note down data about calls.
I had 3 PAD cards and 96 channels. The calls picked random trunks of course, and in the course of the tests, some calls picked trunks more than once. Sometimes the same trunk was fine, sometimes it sounded garbled and sometime it gave no audio. All 3 card showed random problems.
FLNK always showed the same call characteristics:
G711, 64K, jitter buffer min:10, Max 300. etc.

Im on the verge of SINZ-ing the system, but not sure if a) it will solve the problem, or b) even if it solves it, what's the cause and whether it will come back.

I checked licenses as suggested earlier, and I have more phones programmed, but less in use than available licneses. So in theory, that shouldn't be a problem. I also used to have 4 PAD cards programmed, but one was removed many years ago and stored as spare, since the traffic didn't justify keeping them all in place. So last week I just unprogrammed the unused slot in hopse of freeing up some system resources, but no change, no improvement.
Speaking of resources, the CPU usage graph never goes beyond 50%, and normally it hovers around 20-30%.

 

[belevedere]

Is the system redundant? Have you switched over CPU's and retested? Are your data ports set to fixed at 100mb?

 

[phadobas]

The system is redundant and I haven't switched over to the other CPU. As for data ports, I'm not sure they are set to auto or fixed 100. I suppose there is a command (or maybe a switch setting on the card somewhere) to manage it?

 

[belevedere]

ADTM to set the CPU ports up.

 

[phadobas]

The situation is ongoing still. This is how far I got:

It doesn't seem to be a license issue. I don't know if these IP PAD channels need a specific type of license, but as I described in an earlier post, AACT shows that I have more licenses than what's in use.

I did a SINZ (but not CMOD yet) and it solved the problem for a day. Then it started creeping up on me again.

ADTM shows that LAN1 and LAN2 ports are set to AUTO NEGOTIATION. I believe the cisco network switch is doing a gigabit, so that's what these ports are doing too.

I keep seeing stuck IP PAD channels, where one IP PAD channel is connected to another. I use MA4000 to monitor the in-use IP PAD channels. Once I find one that's connected to another, I use the FLNK command which shows something very peculiar. Say PAD channel 1 is connected to PAD channel 2, and when I click the NEXT button in the FLNK screen, it tells me that channel 2 is connected to station 31000 (just an extension number). But 31000 is idle, when I check the phone. Then, I ask FLNK, what is IP PAD channel 2 doing? It shows me that it's connected to IP PAD channel 1. Then I click NEXT button and it shows that channel 1 in turn, is connected to station 51005 (another station). But then again, when I go look at that phone, it's IDLE.

The only way I could release these PAD channels was to pull out those cards.

 

[belevedere]

I would change it to 100 fixed full duplex. That is the suggestion from NTAC. Set your data switch ports to the same.
Can you perform a DISS on the main memory? List your software level. This could very well be a bug in the main software.

 

[PFD45]

I had a similar situation where the PAD channels would stop passing DTMF. This would happen one channel at a time over a period of several days. We would re-set the PAD cards and then it would begin happening again.

The solution was to set ASYDL, System 1, Index 824 to 02 and re-set the PAD cards.

When Index 824 is set to 00 then the PAD cards will "learn" and adjust their values to "sound better". When bit 1 is flagged (02) then "learning" is turned off and the PAD cards remain stable.

Might work for you?

 

[phadobas]

Ok, pfd45, thank you. I made that change and reset the PAD cards. I guess it will be 2-3 days before I can say it worked or not...

Belevedere, thank you for your suggestion. For now I just want to change one thing at a time, just so I can see (if the problem goes away), which change made it happen.

So if the ASYDL change doesn't fix it, my next action will be to change those ports to 100BaseT, Full Duplex.

Per DISS, I have many listed programs, but the main one is S07, Issue 02.03. Other programs have different issue versions, but all have S07