Bounty offered for handling buzz/hiss on SV8500

[phadobas]

This is a new thread on the same problem I've had for the last several months, that went on without resolution. I offer a $5 bounty if somebody comes up with the resolution or an idea that leads to it.

Description of the problem:
SV8500 phone system, with a mixture of DT700 series IP phones, and TDM digital trunks (PRI and CCIS), and analog TDM phones.
You hear an annoying background hiss (sounds like "shshhshshshshshs") if you make a call from an IP phone, that goes via first an IP PAD card, and then also passes through one of the digital trunk cards (CCIS or PRI). If the other party mutes their phones, and you turn up the volume on your phone, the shshshshs changes to a definite buzzing sound. I attach a recording. This was done by attaching a recorder to the curly cord of the handset of an IP phone. It is bad quality but the problem can be heard. The buzz may sound like some electrical interference and it may be caused by that or maybe something else.

When an IP station calls an analog TDM phone within the same system, there is no buzz. (in this call, an IP PAD card is involved, but no digital trunk card).

When an analog TDM phone calls out of the system via a digital trunk card, there is no noise (in this call, a trunk card is involved, but no IP PAD card).
So it appears that for the buzz to be present, you have to call via an IP PAD card and a digital trunk card. However, an NTAC person said that we may have the buzz always, and it just gets amplified buy the IP PAD card.



What has been done so far:

- Vendor has been out here on site about 5 times, went through all ARTD for all routes and set them up per NEC documentation.
- Vendor implemented QoS (ALOCL command)
- NTAC went into the system, checked / changed Qos, ARTD.
- NTAC was sent one of our IP PADA cards for testing and it was found to be not faulty
- System was upgraded from S7 to S8
- Phones' firmware was upgraded from 3.0.5.0 to 5.0.3.0 throughout the facility.
- Tested a known, working SW10A card in the system. Still had the buzz.
- Inspected the grounding for the phone system against NEC recommendation and a new, direct ground cable was run to the building's grounding rod from the pbx room's grounding plate (although we don't think the original grounding via the electrical panel was bad).
- All the L/T cables going to the back of the phone system were non-shielded 25-pair cables. They all got replaced with shielded cables, with their drain wire sticking out and attached to the frame of the PIR-s which are verified to be grounded.
- In order to eliminate possible noise coming in to the system through some analog line (from within the same building or from nearby buildings that are connected via copper), all L/T cables were disconnected except the digital trunk cables. Test call from IP phone still produced the problem.
- The DC-to-DC converter (which provides 80vDC for analog phone message waiting lights) was disconnected for testing. Noise was still there.
- All ground connections from the phone system were disconnected. Test call had the same noise on the line.
- The 8500 sitting right next to a 2400ICS and they are CCIS-d together (not cable-migrated). The 8500 used to get its clocking from the 2400 via the CCIS. The 2400 gets its clocking from a telco PRI. This was changed and now the 8500 is getting its clocking from a local telco PRI. No change in the buzz.
- Just about every setting has been tested in AIPDL command, and changes have been achieves, and the buzz even went away. But at that time the calls started sounding very unnatural.
- ASYDL index 825 was changed to 00h in order to cancel out all effects of AIPDL.


What handles the problem:
There are 2 things that appear to mask the problem successfully.

1) an IP PAD card from and SV9500 was sent to me (along with an adapter to insert it to the 8500). Calls via that card don't produce the problem. I've had 3 IP PADA cards (that came with the 8500 with it was installed), and I can't imagine they all went bad at the same time. NTAC's and the Vendor's theory is that the new IP PAD card from the 9500 is a better design and handles the ongoing noise better. I'm not about to drop $30,000 to buy 3 new IP PAD cards of the SV9500...

2) on every single IP phone, the noise suppression was enabled and also the "side tone" volume was changed from 4 to 0. These 2 settings pretty much completely suppressed the noise and the shshshshs sound on the calls.


However, the above 2 handlings are just masking the buzz, they don't take it out of the system. I still need to get to the bottom of it.

I don't know when the problem started but very likely in the last 3-5 months. General users are not necessarily bothered by this and have not really been reporting on it, even though the noise is on every single phone call, no exception. However, our executives are extremely bothered (of course) and they have stated that it really started in the last few months. I personally use a SP350 sofrphone with a DECT wireless headset, so I never hear the noise. It can only be heard on the DT700 phones. So I personally don't know when it started. I did have a lightning strike within 1/4 mile, and there is a possibility of some charge coming in to the phone system somehow. The lightning strike occurred 4 days before the 1st report of noise. So that might lead somewhere, however, as I stated, all L/T cables have been disconnected, SW10A card has been replaced and our IP PAD cards were tested good by NTAC. So I'm not sure about these and how else or what else to test.

 

[fredsonline]

Just a thought. How are the DT700's getting power? POE from a switch or bricks?

 

[phadobas]

Oh, yeah,

I forgot to mention. Besides the fact that the IP phones are across many different buildings, with completely disrelated power to the poe switches, I've also tested non-poe switches, by simply using power-brick to a test phone.

 

[OzzieGeorge]

Have you tried changing the codec for the Voip side of things?

 

[phadobas]

I think I tried that, but not sure to what, and I did that for bandwidth saving purposes, and a test call sounded pretty "compressed". I'm using G.711. What's the next best to test with?

 

[OzzieGeorge]

I think the phones support G.711, G.729a, and G722.

This site will give you some idea of which may be next best....

http://voip.about.com/od/voipbasics/a/voipcodecs.htm

 

[phadobas]

So I tried all compatible codecs and the shshsh/buzz sound remained. With compression in some codecs, it sounded a bit different but nevertheless, remained. The only difference it made was that the noise sounded "compressed", like it was over a cell-phone call or something.

As another note, we have a 48vdc rectifier, and it's been giving 51.5vdc to the phone system. I've also turned that down to 50.5vdc, but that also didn't make any difference.

Is there maybe some test I could perform on the rectifier, grounding or even the dc-dc converter to check if they work to standard or introduce some noise into the phone system?

 

[fredsonline]

Have you looked into swapping out the data switch the 8500 is connected to?

 

[phadobas]

Hmm, that's very (VERY) complicated. It's connected to a dual cisco switch (or linked or stacked? Not sure what's the right way of saying. Two cisco switches on top of each other, their backs are connected with 2 cables, and they basically act as one switch).

What I've done is I have LAN 1 and LAN 2 going into this dual cisco switch, and also the 3 IP PAD cards. So I moved all the connections into just one switch, test, and then moved all the connections only to the other switch, and then test. No change.
So I moved the connections back (they were spread around between the 2 switches).

But I can't take that switch completely out of the equation as then all the phones in the business would down and I can't afford that with the 24/7 usage we have.

However, I'm really getting to a point where I will pull all (ALL) cards out from the PIR-s, leaving 1 IP PAD card and 1 PRI card, to see if the noise remains... But that's like a really crazy and last resort test to do.
I'm still looking for other ideas...

 

[OzzieGeorge]

I can't remember the name of it but cisco have a feature for Voip that deliberately introduces noise into silence as it was found that people thought they had been cut off with the dead silence that can happen with voip calls (is it silence suppression). Could it be that this feature is on and getting amplified by the IP Pad?

 

[phadobas]

In cisco, it's called VAD (Voice Activation Detection - or something like that). I looked into that extensively, but in NEC manuals I couldn't find anything equivalent.
The only thing NEC is talking about is Echo Cancellers and PAD settings. I spent literally days on these settings. I got as far as I introduced a 9db PAD between the IP PAD card and the PRI cards and the noise actually DID go away. But then what happened was (as you say above), people started to think they got disconnected in the middle of a call. Also, while that setting was there, any small background noise at the far end (not where the IP phone was) was being picked up and played back to the IP phone user as some sort of compressed version of background crackle. It was awful. Worse experience than just the constant buz/shshshshs.

The only promising thing along this line was this:
In the Circuit Card Manual, under the IP PAD A card, the last page does talk about how do you configure the PRI(!) card when you have this IP PAD card in the system. You have to set the PRI card to 8dB PAD in one direction (don't remember if receive or send), and 0dB the other direction. But even with the best of my efforts, trying to get that implemented on the PRI card, I could not make a difference (unless I was doing something wrong).

So with APDIL command, I was able to make some sort of change, even made the buzz go away, but introduced other problems as described above.

Oh yes! The firmware on the IP PAD card was also updated to the latest version. Didn't make a difference...

 

[Ragnorak]

It sounds to me like the background "silence" is being amplified on calls that bridge from your PRI to your IP phones. Have you played with the gain and loss in Program 84-16? Your PRI trunks might also show up in 81-07 (really something for analog lines) and you might be able to adjust settings there. Since they are meant to apply to analog trunks I don't even know if this would have any effect. Basically you seem to have too much gain from your PRI to your IP phones.

 

[phadobas]

While your theory might be right on track, the system is an SV8500 and so different commands apply. Would you know which ones?

 

[phadobas]

Can anyone try to re-produce this problem to see if I'm alone or is it happening elsewhere?

The system is SV8500.
It has IP PAD A cards (not the new, smaller one that came out with the SV9500, but the one that still fits in the original SP3835).

Go to a DT700 series IP phone and call out of the system via a digital trunk card, like a PRI that's TDM. (ensure the phone doesn't have it's noise cancellation feature turned on. This can be seen if you get into the admin menus of the phone).

Then have the other person mute their phones, and you turn your volume all the way up. Do you hear any sort of buzz? Or if you blow into or speak into your handset, you probably start self-generating some sort of hiss/crackle sounding sound. That's what I get.

And then go to a TDM phone, like an analog phone or a Dterm that's hanging off of a line card, do the same test and probably you won't hear the same buzz, hiss, crackle. It's totally quiet when the other person puts you on mute.

I'd like to see if I'm unique and have a problem to handle, or it's just the system I have (and anyone else who has the same setup) that has this phenomenon.
I already know that if I put in a new IP PAD card from the SV9500 (the smaller card), the noise goes away, either because the card masks the noise that's coming from somewhere or because the noise IS generated by the IP PAD cards I have, but not by the new PAD card...

 

[phadobas]

This is now resolved. We purchased the IP PAD cards that come nowadays with the 9500, with the proper adapters and they nicely cancel out all noise, buzz and echo.
Expensive but worked.