3300 MXe with dual T1 embedded trunks - state change and no recovery

[JSilverberg]

New to the 3300 (coming from a BCM, ugh) and I have an odd issue... twice in the past two weeks my D-channel flapped, and when it comes back up calls still ring busy to those trunks (from outside in). I've tried "returning to service" the trunk lines with no success, but a restart of the system seems to solve it.

As I'm quite new to this sytem, the only logs I see of importance simply state "D-Channel is down!" repeatedly until I restart the system.

Any thoughts? or questions you'd like answered before attempting a shot? Thanks all!

 

[JSilverberg]

Looking closer at the situation, the D-Channel (for SOME reason) goes down, and once it comes up it flaps until I restart the system. It did this for 6 hours straight it would appear before the system was restarted and the issue cleared. What could lead to this behaviour?

 

[MitelInMyBlood]

Determine which hybrid is down using the DT stats command, then try using the DT Clear [PLID] to clear it, example, DT CL 6 1 7 1

Now, as to why it is going down in the first place is another matter.

I have a similar situation with a T1 from an SX2K system going to an external application (911 emergency reporting system from Xtend). If the Xtend server reboots, ie to pick up a critical patch, the T1 drops (as would be expected). However, when the server comes back up and the T1 clears, the D channel may or may not restore itself, or may eventually come back on its own anywhere from 5 minutes to 15 minutes later or may require human intervention (DT CL command). Fortunately this is only a once a month event as we only allow OS patches to get applied on the last Thursday of the month. It usually works OK too, just once in a while we'll need to execute the DT clear command. Strange, makes no sense.

 

[Lundah]

What release are you running? I've got a few controllers that were running 7.1.3.4_2 (just upgraded to 8.0.7.16_2) that are doing the same thing, always on the 2nd PRI of a dual embedded T1/E1 framer card. I've been working with the folks in Kanata on it, would be interesting to get some details on someone else having the same issue.

 

[casnev]

I have a 3300 MX that has been in service since April of 2007, running software version 7.1.3_4, and it has had random D Channel drops where the D channel drops, and resets (takes about 4mins). sometimes it happens daily, other times it happens maybe once a week. Our telecom carrier (TelePacific) is stumped(they have done all sorts of tests, etc), and my Vendor took it to Mitel (response to follow). We are about to try changing PRI vendors...know one can solve the problem. I don't know if this is anyway related, but here is where we are at with this so far.

Any ideas of other things to try would be much appreciated!

casnev


From Mitel:

From the logs you collected we have determined the following:

- When the problem happens, it is caused by the Mitel system receiving a BLUE alarm on the link
- This will always cause the Mitel to reset the link.
- Further more when we send the disconnect to start the link reset the CO does not respond until the fourth message. ( in this example )
- Once the CO responds the process to bring the link back up moves forward.
- The full process can complete in seconds or at times it takes 2-4 minutes.


Summary: There is something outside the Mitel gear that is causing the problem. It could be anything from the D Mark to external cabling to the CO.



Here is the explanation from a design level:

The link was "quiet", only RR polling was happening every 10 secs.
Then we received blue alarm, which gathered by this message, 0x51 is the blue alarm :
MsgType = PRI_Card_Alert_List(31a), PortID = a1188e0 (link 0), CallID = 0, TotLen = 4(Exp + Raw), ExpLen = 4
, CES = 0 Expanded(private IE's):
1, 1, 1, 51,

Mitel disconnected L2 (4 times before the far end responded, Last one was at 15:15:17)
15:15:51 farend sent UI, Then Mitel exchanged SAMBE, UA with the farend.
This brought the L2 backup.
At 15:19:52, The farend restarted all the individual channels one by one.

Below is a capture of the event extracted from the log attached.
-------------------------------------------------------------------------------------------------------------------------------
2007-11-26 15:14:57 Link 2 - From Mitel - HLen 4, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 119 NS 0 RR

2007-11-26 15:14:57 Link 2 - To Mitel - HLen 4, ILen 0, IType 5
SAPI 0 TEI 0 C/R 0 P/F 1 NR 35 NS 0 RR

2007-11-26 15:15:07 Link 2 - From Mitel - HLen 4, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 119 NS 0 RR

2007-11-26 15:15:07 Link 2 - To Mitel - HLen 4, ILen 0, IType 5
SAPI 0 TEI 0 C/R 0 P/F 1 NR 35 NS 0 RR


VDSU-2 - MON NOV 26 15:15:14 2007
UAPI message received(MtceCallback), ExpandLen - 4, RawLen 0 ExL2 4, RaL2 0:
PD = 8, MsgType = PRI_Card_Alert_List(31a), PortID = a1188e0 (link 0), CallID = 0, TotLen = 4(Exp + Raw), ExpLen = 4
, CES = 0 Expanded(private IE's):
1, 1, 1, 51,
Raw(public IE's)



2007-11-26 15:15:14 Link 2 - From Mitel - HLen 3, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 0 NS 0 DISC

2007-11-26 15:15:15 Link 2 - From Mitel - HLen 3, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 0 NS 0 DISC

2007-11-26 15:15:16 Link 2 - From Mitel - HLen 3, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 0 NS 0 DISC

2007-11-26 15:15:17 Link 2 - From Mitel - HLen 3, ILen 0, IType 4
SAPI 0 TEI 0 C/R 0 P/F 1 NR 0 NS 0 DISC


VDSU-2 - MON NOV 26 15:15:18 2007
UAPI message received(CpCallback), ExpandLen - 0, RawLen 0 ExL2 0, RaL2 0:
PD = 8, MsgType = DISCONNECT_CONF(131), PortID = a118890, CallID = 80bb, TotLen = 0(Exp + Raw)), ExpLen = 0, CES = 0
Expanded(private IE's):

Raw(public IE's)




VDSU-2 - MON NOV 26 15:15:18 2007
UAPI message received(CpCallback), ExpandLen - 0, RawLen 0 ExL2 0, RaL2 0:
PD = 8, MsgType = DISCONNECT_CONF(131), PortID = a118840, CallID = 80be, TotLen = 0(Exp + Raw)), ExpLen = 0, CES = 0
Expanded(private IE's):

Raw(public IE's)




VDSU-2 - MON NOV 26 15:15:18 2007
UAPI message received(MtceCallback), ExpandLen - 4, RawLen 0 ExL2 4, RaL2 0:
PD = 8, MsgType = PRI_Card_Alert_List(31a), PortID = a1188e0 (link 0), CallID = 0, TotLen = 4(Exp + Raw), ExpLen = 4
, CES = 0 Expanded(private IE's):
1, 1, 1, 11,
Raw(public IE's)




VDSU-2 - MON NOV 26 15:19:46 2007
UAPI message received(MtceCallback), ExpandLen - 4, RawLen 0 ExL2 4, RaL2 0:
PD = 8, MsgType = PRI_Card_Alert_List(31a), PortID = a1188e0 (link 0), CallID = 0, TotLen = 4(Exp + Raw), ExpLen = 4
, CES = 0 Expanded(private IE's):
1, 1, 1, 53,
Raw(public IE's)



2007-11-26 15:19:51 Link 2 - To Mitel - HLen 3, ILen 0, IType 5
SAPI 63 TEI 126 C/R 1 P/F 0 NR 0 NS 0 UI

2007-11-26 15:19:51 Link 2 - To Mitel - HLen 3, ILen 0, IType 5
SAPI 0 TEI 0 C/R 1 P/F 1 NR 0 NS 0 SABME

2007-11-26 15:19:51 Link 2 - From Mitel - HLen 3, ILen 0, IType 4
SAPI 0 TEI 0 C/R 1 P/F 1 NR 0 NS 0 UA


VDSU-2 - MON NOV 26 15:19:51 2007
UAPI message received(MtceCallback), ExpandLen - 4, RawLen 0 ExL2 4, RaL2 0:
PD = 8, MsgType = PRI_Card_Alert_List(31a), PortID = a1188e0 (link 0), CallID = 0, TotLen = 4(Exp + Raw), ExpLen = 4
, CES = 0 Expanded(private IE's):
1, 1, 1, 12,
Raw(public IE's)

2007-11-26 15:19:52 Link 2 - To Mitel - HLen 4, ILen 13, IType 5
SAPI 0 TEI 0 C/R 1 P/F 0 NR 0 NS 0 INFO Orig PD=0x8 CR=0x00 RESTART(0x46)
Info - 0x18 0x3 0xa9 0x83 0x81 0x79 0x1 0x80

 

[casnev]

Edit my software version IS 7.1.3.4_2

casnev

 

[Lundah]

casnev,
This sounds VERY similar to the problem I'm seeing, and have Mitel product support looking at.

I suspect it's either a Mitel or an external/environmental issue, not a PRI problem. The reason I say this is that I have multiple sites fed from the same carrier, even the same CO switch in some cases, and some sites have this issue, while others run clean.

I've just upgraded all of my controllers to 8.0.7.16_2 w/ the SP6 patch last week, and so far I haven't had any new occurances, but it's only been a few days.

 

[casnev]

Lundah, I will be curious if the software upgrade does solve it. This has been a REAL pain. Thanks for the info.

casnev

 

[Lundah]

casnev,
Just for clarification, are you always seeing this problem on the 2nd link of the embedded T1/E1 card? Does this seem to start up after a programmed reboot? Those have been the two primary symptoms in my case.

 

[casnev]

We are seeing it on the 1st and now the 2nd link of the embedded T1. We started on the 1st link, and then tried changing to the second, and it didn't solve it. And yes, my vendor has numerous 3300's out on this same carrier, without any problems...just this particular system is having this. I'm wondering if a software upgrade would help.... I am close to changing the PRI also....

casnev

 

[casnev]

Sorry, didn't answer about the programmed re-boot....that hasnt seemed to make a difference. The system has been configure with our without the programmed reboot on, and the D channel still will randomly drop, and reset. The carrier *swears* the T1 is clean and there is no problem. If the D channel does drop, it seems to fix itself after a few minutes, and a few times it has hung, and only outbound calls would complete. When that happened, we physically unplugged the T1, then when it reset, calls in and out are fine.

casnev

 

[Lundah]

Okay, there's some differences in what you and I are seeing then. You might want to double-check the programming on your link, particularly the protocol settings, that would be a likely place for a mismatch to cause the D channel to drop intermittently.