I ran into an issue fairly recently where a fax machine couldn't connect. It involved an S8300 with a PRI, analog ports (for the fax) and DID. Turns out that there was a very weird software bug that had to do with PRI syncronization that prevented any analog fax or modem traveling through that PRI from connecting properly. Avaya ultimately resolved the issue.
That being said, I still wouldn't want to jump to the conclusion that we have an issue with the S8720.
What is the history here? Was your current setup working just fine, and then suddenly went awry? Or has something changed or been moved recently?
Also, you mention that the AS/400 dials out to "retail server modems". What are the modems at the retail end talking to...a terminal controller? One thing about AS/400 serial connections is that they are synchronous, rather than asynchronous.
Synchronous data connections send entire blocks of data along with timing signals, which is used to synchronize both ends. What that means in English is that it is somewhat harder to establish and maintain such a modem connection (at least compared to asynchronous connections). Soooo, a marginal voice line might allow both modems to handshake / negotiate fine, but still pass contaminated data. It's also possible that one of the DTE devices are giving up prematurely because data throughput is too slow (also possibly due to dirty lines).
Lastly, there's a possibility that someone changed the modem setup string on the AS/400 side. Also, some modems are front-panel configurable, which means that a curious IT guy could be to blame too.
Hopefully some of my post will make sense, or at least inspire you to look at a couple of areas that you haven't yet explored. Post back and I'll help any way I can.
Tim Alberstein