IP Phones and PRI and DTMF, Oh my!

[BigPoppaMo]

I have this same problem happening at 5 different sites.
4 of the sites are running 8.1.73 and one site is running 7.0.23, all on IP500v2.
All sites use 96xx telephones.
The sites are geographically dispersed ... Chicago, Minneapolis, New York, New Jersey, Philly.

Problem:
(at the 8.1.73 sites)
The four 8.1 sites have been installed for approximately 60 days and all are served PRI by the same provider.
When dialing into a conference bridge, or anything that requires DTMF input, the digits pressed by the 96xx user are either not sent, or incorrectly sent to the conference bridge. For example, if the user dials into a Webex conference bridge and inputs 123456 for the conference ID, the conference bridge will respond that the ID was incorrect and read back the digits that were input as a completely different set of digits. Inputting 123456 may be read back one time as 234667 or 122456. It is very random.
I have used SSA and System Monitor and verified the 96xx phones/IPO is sending the correct digits to the line.
When using an analog set over the same PRI, there are no DTMF issues.

(at the 7.0.23 site)
This site worked fine for two years. Within the last 2 days it appears that DTMF is not being passed at all from any of the IP sets when connected to a conference bridge or IVR.
There have been no changes in the IP Office and they are using the same provider they've been using since the IPO was installed.
While on a call with the customer troubleshooting this morning I had him press the digits 1 - 9. I could see the digits being sent to the line, but I heard nothing on my end, no tones whatsoever.
The 7.0 site uses a different PRI provider then the 8.1 sites.

Questions:
Where do the tones come from in the IP Office? Is there a tone generator?
How are the digit presses on the IP Phones converted to DTMF and what components are involved in the process?
Anybody familiar with this problem?

I'm leaning toward a provider issue, maybe caused by some updates, but of course they never have a problem.
Any help/ideas greatly appreciated!

 

[amriddle01]

You need to look for the common factors and other occurrences of this issue, I can say I don't recall others mentioning this issue across several releases and your systems only share 2 things in common once you rule the system itself out, you and the provider, I very much it's you as you aren't involved in low level settings like DTMF tones as it's hard coded so that just leaves the provider and I'm willing to bet that's where the issue lies

 

[BigPoppaMo]

Thanks amriddle01.
I've been thinking it's the provider since day one of this issue.
I have too many other sites with the same setup, 96xx phones and PRI, that don't have any issues.

Trying to get the provider to take a more active role in troubleshooting is like spitting into the wind.
It just keeps coming back on me.

 

[Gunnaro]

Good morning!

To sum it all up:
You have two different ISDN providers, five systems, two software levels.
Your analogue phones sends DTMF just fine. IP's don't, (they send, but you get random results).
Other systems on same providers works fine.

Leads me to think that your signal output from the IPO's are too weak.

- If your codec is G.729, change it to G.711 (Out of band checked or not?)
- Try raising the gain a bit on IP extn.
- Faulty upgrade of IPO's.

Maybe upgrade the R7 to latest version?
Can they send good DTMF tones internally?

Are these troubled systems just one customer, connected by SCN?


Kind regards

Gunnar
__________________________________________________________________
Hippos have bad eyesight, but considering their weight, it’s hardly their problem

 

[amriddle01]

I didn't notice the analogue set part, perhaps this isn't the provider, by default the IP Office uses G729 for some reason, try changing to G711 as suggested

 

[Gunnaro]

Yeah, what are the odds of having two different providers simultaneously mess up their DTMF in 5 different cities?

Default should have been G.711 IMO

Kind regards

Gunnar
__________________________________________________________________
Hippos have bad eyesight, but considering their weight, it’s hardly their problem

 

[BigPoppaMo]

UPDATE:
The issue at the 7.0 site was resolved by rebooting the IP Office.
Throwing that one out of the mix.

The problem with the other (4) systems remains.

The four systems are on an SCN.
The systems physically reside in New York, New Jersy, Philly, and Chicago.
We have been concentrating our troubleshooting efforts to the New Jersey location (highest number of complaints).
Analog phones do not seem to have any issues sending DTMF tones at any of the sites (but of course the phone generates its own tones).
New York has a different PRI provider and does NOT report any DTMF problems (coincidence???).
We have added transmit gain at the NJ site. While it seemed to improve our success, it did not resolve the problem ... additionally the customer now reports that some people say they sound too loud.
We have changed/tried different codec combinations with no change to the problem.
We have connected a phone directly to the WAN/LAN2 port and tested, to eliminate the customer network, no change.

Recently we did test calls across the SCN and out the PRI in New York to a conference bridge number the NJ site has issues with every time they call.
Once connected to the bridge, we input the incorrect access code of 456789 to force the system to read the digits back to use. By pressing the digit 2 we could do this same test scenario multiple times on the same connection.
We made four of these calls and each time input the incorrect access code 10 times. No errors.

We made ten more calls to the conference bridge, each time inputting the incorrect access code once, to verify the bridge was seeing the digits we input, and then disconnecting and calling back.
The 10th called failed and was off by one digit, we pressed 456789 and the bridge read back 446789.(always possible the tech fat fingered the buttons, but says he didn't)
I would consider this a fairly good success rate.

We did raise a case with Avaya Support and provided all the information requested as well as access to the IPO.
As we expected, they say everything is working as designed on the IP Office side.

Still trying to get the carrier to participate in further testing.

Still trying to determine how the IP Office sends the tones.
I know the IP set sends a message to the IP Office that a digit key has been pressed and the associated DTMF tone needs to be sent ... the question is where is that tone generated?
Is there a tone generator in the IP Office or does it play out something like a WAV file?

I'm still leaning toward a provider issue of some type since the SCN test procedure.

Input appreciated.

 

[Gunnaro]

If I remember correctly, analogue phones have a built-in DTMF, so they confirm that the line actually passes the tone correctly
Still you could be right about the lines, but some providers like to blame everyone else

Have you tried calling your cellular and press some buttons? How does the DTMF sound?

Try this on your IP phone:

Go to

http://www.audiocheck.net/audiocheck_dtmf.php

and make your DTMF tone string. (middle of the page).
Download the wav and blast that out on your PC speakers with the phone handset on top of the speaker (like a 1980's modem) .

Do you get correct numbers read back? If so, there is something wrong with your IPO's.

Kind regards

Gunnar
__________________________________________________________________
Hippos have bad eyesight, but considering their weight, it’s hardly their problem