Perfect. So we know it's about how your SIP conference phone makes vs receives calls.
Now...here's where the SIP stuff gets complicated. You're interested in the signaling group that your phone takes when dialing out vs when being called.
When being called is easy - in the last page of the station form you have 99. I suspect you're on CM6.3. Just 99 = trunk group. On 6.3 you can say 'aar' or 'trunkgroup#". On 7.x they added route patterns, so you need to say rp99 for route 99 or tg99 for trunk 99. They did that so you can have a route with choices to SM1 and SM2 and still hard code something at the station level but have some ability to failover.
Anyway, anything hitting the PBX for your conf phone goes out trunk 99. Trunk 99 has a "sig group" defined in the bottom right of the first page. You're saying that particular sig group is set to rtp-payload.
How CM picks your conf phone's sig group when you're dialing out:
SIP phone goes off hook, dials a number, whatever.
SM receives that and invokes the application sequence in the user's SM Profile
That app sequence has a SIP entity as destination (you'd see that in SMGR-->Session Manager-->Network Configuration-->Applications)
Notice that the Application Sequence specifies ONLY the entity of CM, not a specific entity link or port or anything
SM then sends the request to CM.
When CM gets a SIP request, it needs to figure out which signaling group its associated to. Think of the whole system like a tree. SM is the tree trunk, the branches of SIP phones are on top, and CM is the root system underneath. Yes, it's true that many aspects of the system design are in the roots and the branches, but it's also true that messages from the branches need to feed through the trunk to get to the right roots.
In terms of PBX design, it would be perfectly normal to have 1 CM in a data center with 1 SM feeding many different sites and applications. It would also be normal for CM to have many sig/trunk groups to SM, but SM still only has 1 entity/link to CM. Remember - CM has a concept of channels/trunks - SM doesn't. If you need 1000 channels up to SM for something, you're building 4x trunk/sig groups for it while on the SM side, it still only has 1 entity/link to CM.
Now, take that example of 1000 channels on 4 trunk groups. Suppose they're on the same IP/port for the CM and SM side. How does CM pick a sig group when SM sends a call inward?
The rule is:
CM will look at the near/far IP/port and list all sig groups that are a match.
Next, it will compare the domain of the request - 1234@pbx.company.com.
It will consider a match ANY signaling group that has, as far end domain, something that fits into pbx.company.com
That means all sig groups with far end domain pbx.something.com AND all sig groups with FE domain something.com are matches.
CM lists the matches and the winner is the sig group with the lowest number in the system.
What that means is if you're going to have elaborate subdomains is that ivr.east.company.com should be sig group 1 and company.com should be sig group 999. To say, your most elaborate subdomains should be the lowest numbered signaling groups so they would explicitly not match 1234@pbx.company.com.
If you happen to have a sig group 1 with far end domain company.com, then what you're doing is forcing all traffic from SM to CM thru that sig group, be damned if its a SIP phone calling out, a PSTN SIP trunk on a SBC, or AAM calling out to use a reach me feature, and you have no segregation of handling of call types coming to CM via SM
So, back to your problem, you should try to find out what sig group you're using when dialing OUT from the conference phone and compare those DTMF settings to the sig group it receives calls on.
And, don't kill yourself - what kind of conf phone is it? Their passwords are usually 123 or 456 for polycoms, they all have a web page for the most part, and you can see what its DTMF settings are.
I still find it odd! If you're in-band on the phone side, then Avaya be damned, even if Avaya's going to you with out of band settings, if the phone is in band, the tone should make it in the speech path across the B channel to the AT&T box listening. And if your conference phone is RTP too, then inbound calls using the sig group with RTP payload shouldn't even be a problem.
Anyway, there's a little crash course in why a lot of Avaya SIP networks are goofy. They really didn't socialize these concepts when people were putting them out for the first time, so there's a lot of less-than-ideal config out there that's harder to untangle than 'just doing it right'. Doing it right 5 years ago wasn't easy!
Go easy.