To recover the ports, I had to disable the line side T-1s and re-enable them. The IVR ports are built as 500's with ACD agent POS IDs. So, after the line sides were back up, I had to log the agents back in.
A little research has shown that my TAPI server (in my case it is a Genesys T-Server) received an unsolicited message from the "switch" warning that all of the agents were being placed into Not Ready. A little note here, Genesys cannot distinguish between the PBX and Symposium, it just says switch.
Any way, this all seems to prove that Symposium issued this go into Not Ready to all of my IVR agents and the same time.
I am using a Call Presentation Class to LOG OUT an IVR agent if it doesn't answer the phone in 45 seconds; however, that would only take effect one port at a time. Log files indicate this message went to all 720 agents at the exact same instant.
This is the first location where I am using Not Ready Reason Codes. But I don't see how that could have such an effect.
Here is a piece of my T-Server (TAPI) log file.
**********************************************************
@11:04:47.4860 [0] 7.0.202.00 distribute_event: message EventAgentNotReady
AttributeCustomerID 'Resources'
AttributeTimeinuSecs 486000
AttributeTimeinSecs 1114445087 (11:04:47)
AttributeThisQueue '7903'
AttributeAgentID '4480'
AttributeThisDN '54480'
11:04:47.486 Trc 04542 EventAgentNotReady sent to 672 (01a4 I-Server)
@ 11:04:47.5640
<< 1 FF 0A 00 22 02 00 00 11 05 00 48 06 37 39 30 33 36 09 00 1E 35 34 34 38 31 3F 04 64 01 5F 05 0B 05 29
s< 1 Agent Not Ready Notification
ACD DN:7903
This Device DN:ACD Pos ID,54481
Origination TN:6401
Enhanced Time Stamp:11:05:41