PRI strangeness on Cisco IAD 2431

[ncc63725]

I've got a client running a Merlin Legend R7 V14.2 proc with a newly installed 100D. This is a PRI via a 3 MB Dynamic setup that Paetec (now Windstream I suppose) is providing via this Cisco IAD.

What they are handing off to me is a T1 interface configured for ISDN-PRI with the switch type set to 5ESS. I set this up like any other PRI install I've done and once they set it up with 5ESS in the IAD all my errors (PRI-D Channel Inoperative and/or PRI SVC Timeout) disappeared. The issue we then had was the IAD would not pass a call without the switch providing an outbound CPN. This was a new one on me as in my exerience providers are generally happy to override your CPN anyway, but I asked them to provide the CPN line via the router. They complied and then the real fun started.

The issue we're having is this - with them providing the CPN, about 2 out of every 3 calls rings back to the main line (the same number they are providing for CPN). It's the most bizarre thing, you can see an outbound call seize trunk 23 and then ring right back in on trunk 1 no matter what number was dialed. Then sometimes the call will go through as dialed. The provider claims they can setup a trace and watch when these calls are made and "loop" back in and they claim there is no ISDN or SIP communication taking place even though they do see both of those trunks in use when this happen. Basically, they are blaming this on the switch, but I know of no programming that would cause this kind of behavior to occur.

I got the bright idea that I would just provide CPN in hopes of alleviating this issue. To that end, I asked them to remove their line(s) of code in the router providing the CPN and I went into LinesTrunks-->PRI-->NmbrToSnd and assigned their 10-digit number to all 23 of the b-channel lines. Now for a whole new level of strange.

With this setup, you hit 9 to grab a line (which gets you a dial tone), dial your number, and then are presented with a SECOND dial-tone. Now if you dial your full number again, the call will go through, and the calls never "loop" back into the main number, but that's hardly a fix considering they would have to dial every number twice. I asked them to trace this behavior as well and they see the numbers come through both times but that according to their traces, the switch disconnects after dialing the first time and then only connects after dialing the second time.

For now this is the setup and what I have them doing is just dialing their own area code during the first dial tone; since it's going to disconnect anyway no matter what they put into the first dial tone, it seems to be irrelevant whatever they throw in there, and then dial their real number when they are presented with the second dial tone. My thought was that the first dial tone is system generated and then the second was being generated by the IAD but they are insistent that their setup has no ability to generate any dialtone at all and so therefore again, this is a problem with the switch.

Has anyone ran into any of these issues? Any ideas/suggestions for things to try or to ask Paetec to try? They are bringing out a T-BERD tomorrow to do some of their own test calls and if they do not run into these issues they are going to call it a switch issue.

The layout of the switch is:

CKE4 Legend R7 V14.2
800MLX
800MLX
012TRR
Merlin Messaging R1
100D
------
008MLX
008MLX

Thanks guys!!!

 

[merlinman]

If the T-Berd doesn't find anything, I would suggest printing and posting your PRI and T1 translations here.

http://home.comcast.net/~merlinman

-merlinmansblog.blogspot.com

 

[SYQUEST]

You should also ask Paetec for the DCH call setup and cause codes for all the calls placed, and maybe some call traces too. This might reveal some useful info. If you ask me, I think the IAD has lost its mind and can't figure out if it is a switch or a router...

....JIM....

 

[ncc63725]

They sent out their tech with a T-BERD and initially he couldn't get calls to work either (ran into the CPN problem we originally had) but once setting an outbound number string on his unit he was able to make calls using Custom protocol. That said, they must not have been that confident in it as they initally blamed it on the age of the system but decided to swap it out with an Adtran TA-908E. I know Legends are old but to my knowledge, Custom protocol is Custom protocol; if they are truly setup to emulate it well then we shouldn't be having the issues. Nonetheless, they are bringing out an Adtran which I've got more faith in since I've seen it work with an Adtran.

Merlinman - I unfortunately cannot get dialed in long enough to do a print of the T1/PRI translations as the voice channels are so compressed I can't even connect long enough to get the password typed in. I'm going to throw an old laptop in there and attach it to the Admin port with gotomypc on it until this all gets resolved and I'll post the translations as soon as I get that set up.

Syquest - they refused to give me the DCH call setup info or the config as they said it contained proprietary information. I imagine they are just tired of dealing with it as they are replacing this router and aren't really concerned with getting this to work anymore. From what they told me, I was the first person to ever ask their Cisco team to emulate custom so I doubt they think they'll be doing it again any time soon.

 

[merlinman]

Compressed....

Yes, that's an issue these days. I have had a bit of success getting them to REMOVE COMPRESSION and then I am able to Dial In.

It's always a good idea to keep 1 analog line, if for no other reason than to be able to trouble shoot when the T1 is down. But, it's a hard sell to most clients.

http://home.comcast.net/~merlinman

-merlinmansblog.blogspot.com

 

[ncc63725]

merlinman - here are the T1 and PRI translations. Pretty basic but if you see anything in here that would facilitate any of these issues please let me know. I replaced the actual number to send with Xs for the customer's privacy but their actual 10 digit main number is what's really in there. They have yet to bring out the Adtran but keep saying soon:

DS1 INFORMATION

DS1 SLOT ATTRIBUTES

Slot Type Format Supp Signal LineComp
5 PRI ESF B8ZS DMI-MOS 1


PRI INFORMATION

Slot 5 Switch: 5ESS

System: By line

BchnlGrp #: Slot: TestTelNum: NtwkServ: Incoming Routing:
1 5 CallbyCall By Dial Plan

Channel ID: 23 22 21 20 19 18 17 16 15 14
13 12 11 10 9 8 7 6 5 4
3 2 1

Line PhoneNumber NumberToSend
801 XXXXXXXXXX
802 XXXXXXXXXX
803 XXXXXXXXXX
804 XXXXXXXXXX
805 XXXXXXXXXX
806 XXXXXXXXXX
807 XXXXXXXXXX
808 XXXXXXXXXX
809 XXXXXXXXXX
810 XXXXXXXXXX
811 XXXXXXXXXX
812 XXXXXXXXXX
813 XXXXXXXXXX
814 XXXXXXXXXX
815 XXXXXXXXXX
816 XXXXXXXXXX
817 XXXXXXXXXX
818 XXXXXXXXXX
819 XXXXXXXXXX
820 XXXXXXXXXX
821 XXXXXXXXXX
822 XXXXXXXXXX
823 XXXXXXXXXX

Network Selection Table

Entry Number: 0 1 2 3
Pattern to Match: 101**** 10***

Special Service Table

Entry Number: 0 1 2 3 4 5 6 7
Pattern to Match: 011 010 01 00 0 1
Operator: none OP OP OP/P OP none none none
Type of Number: I I I N N N N N
Digits to Delete: 0 0 0 0 0 0 0 0

Call-By-Call Service Table

Entry Number: 0 1 2 3 4
Pattern 0: 0
Pattern 1: 1
PRI INFORMATION


Pattern 2: 2
Pattern 3: 3
Pattern 4: 4
Pattern 5: 5
Pattern 6: 6
Pattern 7: 7
Pattern 8: 8
Pattern 9: 9
Call Type: BOTH BOTH BOTH BOTH BOTH
NtwkServ: No Service
DeleteDigits: 0 0 0 0 0

Entry Number: 5 6 7 8 9
Call Type: BOTH BOTH BOTH BOTH BOTH
NtwkServ:
DeleteDigits: 0 0 0 0 0

Dial Plan Routing Table

Entry Number: 0 1 2 3
NtwkServ: Any service
Expected Digits: 0 0 0 0
Pattern to Match:
Digits to Delete: 0 0 0 0
Digits to Add:

Entry Number: 4 5 6 7
NtwkServ:
Expected Digits: 0 0 0 0
Pattern to Match:
Digits to Delete: 0 0 0 0
Digits to Add:

Entry Number: 8 9 10 11
NtwkServ:
Expected Digits: 0 0 0 0
Pattern to Match:
Digits to Delete: 0 0 0 0
Digits to Add:

Entry Number: 12 13 14 15
NtwkServ:
Expected Digits: 0 0 0 0
Pattern to Match:
Digits to Delete: 0 0 0 0
Digits to Add:

 

[merlinman]

I deem the PRI Programming looks correct. I compared it to a system I service that's been working with a similar set up for over 10 years.

Hello Mr. Carrier!

http://home.comcast.net/~merlinman

-merlinmansblog.blogspot.com

 

[ncc63725]

They are bringing out an Adtran 908e on the 17th to swap with that Cisco IAD. They are suggesting that I also replace the 100D with the latest board which I am assuming they mean to be a 100 DCD. To do that I would need to upgrade the processor to Magix software I believe with the DCD is a Magix card. Do you think it's worth doing that or are they just preparing excuses? To me I would think that the 100D can talk ISDN PRI just as well as a DCD.

 

[ncc63725]

I finally got a working method of remoting in and found that if I dial the unrestricted remote access number DID that I setup and then dial 9 and a number I get through without the second dialtone. This made me think more about the switch playing a part in this so I setup some traces with monitor. Here are the findings. I had someone there get on an analog line so I could actually watch the digits be dialed and see what happens when. This trace is trying to just dial the number as one should (9+number) and he got the secondary dialtone. Interestingly enough, the switch believes it has connected the call and thinks it's done. He then gets the siren noise if he waits too long at the secondary dialtone which is what I directed him to do. Here is the trace from that call (I scrubbed the actual numbers):

** 04/10/12 14:47:41 sid: 3038, button: 247 (9)
** 04/10/12 14:47:42 sid: 3038, button: 255 (1)
** 04/10/12 14:47:43 sid: 3038, button: 252 (4)
FM_S_orig: fid=809
CD_F_sack: fid = 809
** 04/10/12 14:47:43 sid: 3038, button: 255 (1)
** 04/10/12 14:47:43 sid: 3038, button: 249 (2)
** 04/10/12 14:47:43 sid: 3038, button: 248 (5)
** 04/10/12 14:47:44 sid: 3038, button: 252 (5)
** 04/10/12 14:47:44 sid: 3038, button: 252 (5)
** 04/10/12 14:47:44 sid: 3038, button: 248 (1)
** 04/10/12 14:47:44 sid: 3038, button: 254 (2)
** 04/10/12 14:47:45 sid: 3038, button: 249 (3)
** 04/10/12 14:47:45 sid: 3038, button: 255 (4)
** 04/10/12 14:47:50 fid: 809 SND PRI msg 0005 SETUP
Prot Disc: 8 Orig crv: 73
SETUP
IE: 04 Bearer Capability
Coding Std: CCITT
Xfr Cap: 3.1 Khz Audio
Xfr Mode: Circuit
Xfer Rate: 64Kbit
User Inf. Layer 1
Mu Law

IE: 18 Channel ID
Pref Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 16

IE: 6C Calling Party
Type: National
Plan: ISDN Rec. E.164
Presentation Ind: Allowed
Screening Ind: User Prov. Not Network Scr.
Number: 3125551
234
IE: 70 Called Party
Type Sub-Address: User Specified
Number: 14125551234
** 04/10/12 14:47:50 fid: 809 REC PRI msg 0002 CPROC
Prot Disc: 8 Dest crv: 73
CALL PROCeding
IE: 18 Channel ID
Excl Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 16

IA_ch_conf: loss for normal (fid=809) = 0x0006
** 04/10/12 14:47:50 fid: 809 REC PRI msg 0007 CONN
Prot Disc: 8 Dest crv: 73
CONNect
** 04/10/12 14:47:50 fid: 809 SND PRI msg 000F CACK
Prot Disc: 8 Orig crv: 73
CONNect ACKnowledge
** 04/10/12 14:48:16 sid: 3038 now on-hook
** 04/10/12 14:48:16 fid: 809 SND PRI msg 0045 DISC
Prot Disc: 8 Orig crv: 73
DISConnect
IE: 08 Cause
Cod. Std: CCITT
Location: User
Class: Normal - Normal Clearing
CD_rm_fid: fid= 809
FM_F_rel: fid=809
** 04/10/12 14:48:16 fid: 809 REC PRI msg 004D REL
Prot Disc: 8 Dest crv: 73
RELease
** 04/10/12 14:48:16 fid: 809 SND PRI msg 005A RCOMP
Prot Disc: 8 Orig crv: 73
RELease COMplete
Ready




Then I had him call again using the "trick" we've developed of dialing a few digits after the 9 and hitting # to generate the secondary dialtone and then dial the real number. What's interesting here is that the switch believes it has passed only the first few "trick" digits and yet the call actually goes through using this method. This makes me think that the IAD must certainly be generating the secondary dialtone:

Ready
** 04/10/12 14:47:08 sid: 3038 now off-hook
** 04/10/12 14:47:10 sid: 3038, button: 247 (9)
** 04/10/12 14:47:10 sid: 3038, button: 255 (1)
** 04/10/12 14:47:11 sid: 3038, button: 253 (3)
FM_S_orig: fid=808
CD_F_sack: fid = 808
** 04/10/12 14:47:11 sid: 3038, button: 255 (1)
** 04/10/12 14:47:11 sid: 3038, button: 252 (2)
** 04/10/12 14:47:12 sid: 3038, button: 244 (#)
** 04/10/12 14:47:12 fid: 808 SND PRI msg 0005 SETUP
Prot Disc: 8 Orig crv: 72
SETUP
IE: 04 Bearer Capability
Coding Std: CCITT
Xfr Cap: 3.1 Khz Audio
Xfr Mode: Circuit
Xfer Rate: 64Kbit
User Inf. Layer 1
Mu Law

IE: 18 Channel ID
Pref Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 17

IE: 6C Calling Party
Type: National
Plan: ISDN Rec. E.164
Presentation Ind: Allowed
Screening Ind: User Prov. Not Network Scr.
Number: 3125551
234
IE: 70 Called Party
Type Sub-Address: User Specified
Number: 1312
** 04/10/12 14:47:12 fid: 808 REC PRI msg 0002 CPROC
Prot Disc: 8 Dest crv: 72
CALL PROCeding
IE: 18 Channel ID
Excl Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 17

IA_ch_conf: loss for normal (fid=808) = 0x0006
** 04/10/12 14:47:12 fid: 808 REC PRI msg 0007 CONN
Prot Disc: 8 Dest crv: 72
CONNect
** 04/10/12 14:47:12 fid: 808 SND PRI msg 000F CACK
Prot Disc: 8 Orig crv: 72
CONNect ACKnowledge
** 04/10/12 14:47:13 sid: 3038, button: 252 (4)
** 04/10/12 14:47:13 sid: 3038, button: 255 (1)
** 04/10/12 14:47:13 sid: 3038, button: 249 (2)
** 04/10/12 14:47:14 sid: 3038, button: 248 (5)
** 04/10/12 14:47:14 sid: 3038, button: 252 (5)
** 04/10/12 14:47:14 sid: 3038, button: 252 (5)
** 04/10/12 14:47:14 sid: 3038, button: 248 (1)
** 04/10/12 14:47:15 sid: 3038, button: 254 (2)
** 04/10/12 14:47:15 sid: 3038, button: 249 (3)
** 04/10/12 14:47:15 sid: 3038, button: 255 (4)
IA_Conf(gain=2): loss (fid=0x0328) = 0x0006

Thoughts?

 

[ncc63725]

Forgot to include the final trace - this is where I dial into remote access and then dial back out using remote access. This goes through everytime without dialing any extra digits. There interesting thing about this trace is that there is NO calling party information passed at all by the PBX and the calls always go through. If I blank out the numbrtosend piece of the PRI programming it still sends calling party info, it just sends it as blank and then the IAD will stop the call, but for some reason providing no calling party information at all seems to make it the happiest. Any ideas about stopping the switch from sending any calling party info altogether or should I just stop at this point and wait for the Adtran? I can only include the PRI trace as there doesn't seem to be a method for tracing the remote access port:

Ready
FM_S_orig: fid=806
CD_F_sack: fid = 806
** 04/10/12 15:51:22 fid: 806 SND PRI msg 0005 SETUP
Prot Disc: 8 Orig crv: 93
SETUP
IE: 04 Bearer Capability
Coding Std: CCITT
Xfr Cap: Voice
Xfr Mode: Circuit
Xfer Rate: 64Kbit
User Inf. Layer 1
Mu Law

IE: 18 Channel ID
Pref Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 17

IE: 70 Called Party
Type Sub-Address: User Specified
Number: 14125551234
** 04/10/12 15:51:22 fid: 806 REC PRI msg 0002 CPROC
Prot Disc: 8 Dest crv: 93
CALL PROCeding
IE: 18 Channel ID
Excl Not_D-Ch B1 Chan
Coding Std: CCITT
Type: B-Channel Units
Numbe: 17

IA_ch_conf: loss for normal (fid=806) = 0x0000
** 04/10/12 15:51:24 fid: 806 REC PRI msg 0003 PROG
Prot Disc: 8 Dest crv: 93
PROGress
IE: 1E Progress Indicator
Coding Std: CCITT
Location: Prvt Ntwk Srvng Loc Usr
Prog Desc: Inband Treatment Applied

** 04/10/12 15:51:54 fid: 806 REC PRI msg 0007 CONN
Prot Disc: 8 Dest crv: 93
CONNect
** 04/10/12 15:51:54 fid: 806 SND PRI msg 000F CACK
Prot Disc: 8 Orig crv: 93
CONNect ACKnowledge
** 04/10/12 15:52:03 fid: 806 SND PRI msg 0045 DISC
Prot Disc: 8 Orig crv: 93
DISConnect
IE: 08 Cause
Cod. Std: CCITT
Location: User
Class: Normal - Normal Clearing
CD_rm_call: dial_fid = 806
FM_F_rel: fid=806
** 04/10/12 15:52:03 fid: 806 REC PRI msg 004D REL
Prot Disc: 8 Dest crv: 93
RELease
** 04/10/12 15:52:03 fid: 806 SND PRI msg 005A RCOMP
Prot Disc: 8 Orig crv: 93
RELease COMplete
Ready

 

[ncc63725]

This has finally been resolved. After forcing escalation to a higher level engineering group at Paetec, the problem was found to be within the IAD programming of dial peers; basically outbound calls were hitting a misprogrammed dial peer that was stripping the first 6 digits from outbound calls, leaving only four which made the IAD think the call was actually inbound as we receive inbound calls with only the last four digits in order to match internal extensions.

No Adtran swap-out was needed and I can finally say that Merlins will definitely work with Cisco IAD boxes, although CAREFUL attention must be paid to the programming therein.

Thanks for everyone's help with this!