T1 Problem

[dryaquaman]

I have a problem I've never stumbled into before.
I have a handful of sites doing this. All with the same carrier, same brand of PBX.

ESF/T1 B8ZS
24 channels of PLAR. E&M
All PLARS will break out to different locations on the far end via the carrier and connect via T1 to multiple different types of PBXs.
All signaling bits are normally all ones on idle.
Clocking source doesn't seem to make any difference, line build out, DB, etc.
I never see reports of frame loss or slips. Neither does the carrier.
If I move the T1 to a different port on the PBX the problem follows the T1.
The carrier delivers circ to prem via IP and hands off via a router.

Ok here's the problem:
Someone will make an outbound call and on one channel and this causes all channels to seize outbound from the PBX.
Doesn't happen every time/call. Doesn't happen with every channel and isn't always consistent with any single channel.
Doesn't happen with all T1s connected to the same PBX or line card.
The PBX never reports an outbound call. It will show an incoming call if the far end picks up the handset because their end rang and this causes my end to ring.
I put a piece of equipment between the carrier and my PBX that would generate syslogs to show the origination of the calls and it always shows my PBX as the originator.

I'm trying to figure out what may be causing this. The carrier has convinced me that our protocols are matching.
The only thing I can think of is that perhaps we are getting a "false frame" for the 193 frame but I have no way to prove it.

Does anyone have any ideas at all?

Dry Aquaman

http://www.mitelforums.com

 

[SYQUEST]

Hi Dry Aquaman,
I have some ideas, but I need some information about the network to better understand how it works:
* What kind of PBX is being used?
* Are the DS1 modules multi or single circuit in the PBX?
* What kind of router and VWIC is being used to provide the DS1 connection?
* With 24 channels of Private Line Automatic Rindown, where does the ringdown equipment or its functional equivalent reside?
* What kind of circuit and bandwidth is provisioned to the access router?
* Does the access router have any other traffic going through it?
* How are the channels accessed in the PBX?
* What test equipment is being used for testing or monitoring?
* Have any of the DS1 modules in the PBX been replaced?

I am guessing this network is being used by a stock exchange or trading business? I will await your response...

....JIM....

 

[dryaquaman]

Hi,
Thank you for the response.
You are correct about it being a trading business.
The PBX is a Mitel.
Anything related to the ringdown exist in my PBX if I understand your question.
Access is always programmed the same way - via ARS. Each trunk is in it's own trunk group.
Sometimes the phones(s) using the line is on the same PBX, other times not.
I don't know about the bandwidth. I'm led to believe it's adequate.
There is no other traffic on the routers other than the PLARs.
We have replaced DS1 modules, PBXs and gateways. Nothing has made a difference in that regard.

A note about some of the testing I did. I put an Adtran between the carrier and myself so that I could generate and capture detailed syslogs. The syslogs always show the PBX seizing all the channels when this happens. (The PBX never shows this even at low level debug) I put another Adtran (TSU120) between the original Adtran and the Carrier and then dropped out channel 24 so that it was isolated from the PBX/carrier. I then duplicated the problem and channel 24 still showed the issue happening.

Working on one of our sites with this issue we found that by changing the DS0 in the CISCO to "trunk" mode rather than PLAR mode we can stop the problem from happening. We don't have to do this to every DS0, we just have to find the one that causing the problem and change it. I've been going through the SMDR logs of my most recent problem system and identifying what looks to be the problem channel and having the carrier change the DS0 to trunk mode. I'm now monitoring to see if this makes a difference. It does not have to be the one that appears to be the cause in SMDR that is the actual cause.


There is nothing in the PBX that we have been able to change that has ever made a difference. It's always been changes that the carrier has made. There has been one case where a router has not been involved. There were two T1s directly to the carriers CO. For testing I had the carrier loop a trunk from 1 T1 to the other T1 so that I could check logs and see when the problem happened. Once that was done the problem never happened again.

Because of the following reasons I'm pretty sure the issue is sourced from the carrier:
[ul]
[li]The problem always follows the T1 to different ports on the Mitel, even without any programming changes of any kind.[/li]
[li]Every time we've cleared the issue it's always been changes that the carrier has made [/li]
[li]Even the lowest level debug on the PBX will never show an outbound seizure [/li]
[li]We do this same thing with other carriers and this is the only one that we have problems with [/li]
[/ul]

But my question remains what would cause this and how could I possibly prove it out?
I can only think that the router would be sending me a false frame or for some reason is shifting bits that would make it appear that the PBX is seizing all the channels when it is not. And since we've actually had the problem follow a specific channel to a different T1 that wasn't experiencing a problem before, how could a single channel create this problem?

Dry Aquaman




The Route is a Cisco 3900 Series (I think) but I have no idea about the VWIC.

http://www.mitelforums.com

 

[trvlr1]

It may not be the call that is causing this, it may be the number being called. If you call x1234 it goes out of circuit 1 to the next PBX. If in that PBX the number, x1234 is pointed back to the first PBX instead of say the 3rd PBX, it will route it down the second channel, the first PBX will route it back and so on until it quickly uses up all the channels. I do not know Mitel at all so cannot tell you where to look, but this is where I would start investigating.

 

[SYQUEST]

Hi Dry Aquaman,
This is a very interesting situation. I agree with your findings, in particular that everything points to the carrier as the source of the problem. Now, are the DS1s point-to-point connecting directly to another PBX? Are all the DS1s provisioned from a router, like the Cisco 3900 with WIC or VWIC modules? BTW, WIC = WAN Interface Card, VWIC = Voice WAN Interface Card. Above you mention two of the DS1s are direct to the CO, the location(s) that have these DS1s do they have this false seizure problem? The problem may have to do with the way the carrier provisioned the channels on the DS1 circuits, and how they are translated from protocol to protocol at the points where they connect the DS0 channels. The DS1 framing and transport can be affected by various changes caused by translating protocol to protocol and timing in the network, if the DS1 envelope is not contiguous from beginning to end. As is the case with HDSL provisioned DS1 circuits, the DS1 frame and signal can be affected with no alarms or errors reported, but the DS1 envelope is broken and the data or signal is affected. This will not happen on a DS1 riding a TDM DS3 on copper or fiber or digital microwave, or when the DS1 is a repeatered copper DS1 circuit(old style).

Do you know what the physical topology of the your client's network is? I know I stated a mouthful of information above, and you may or may not understand what I just stated, so I will stop here for your feedback and answers.

....JIM....

 

[dryaquaman]

trvlr1,
This are PLARs or Ring downs. No digits are being sent at all.

SYQUEST,
Thank you for your reply. You may easily be speaking over my head but I'll do my best here.
The DS1s do not go directly to another PBX. They feed back to the carrier which breaks out the DS0s and routes them to multiple destinations. The far end DS0 could be riding on a T1 or E1 depending on where it terminates.

The two DS1s that I said are direct to CO did have the problem until I had the carrier loop a channel back from one T1 to the second one so that I could see when the problem happened via my SMDR logs. If I saw an incoming call on T1 #2 that would mean my 1st T1 seized all channels outbound. Once the carrier created this looped trunk the problem ceased.

When you asked about the physical topology of my client's network I'm not quite sure if you're asking for the carriers topology or the my customers PBX layout. I can answer my customers PBX layout: The handoff to me from the carrier is via T1. From there, the IP phones may be connected directly to the same PBX or to another user gateway. That hasn't made a difference.

As for the carriers topology, The only thing I know is that we connect via a router and they hand off via SIP. Beyond that I'd don't know.

DryAquaman

http://www.mitelforums.com

 

[dryaquaman]

Update:
We found that by changing the T1 format from ESF/B8ZS to D4/AMI the problem goes away.
I'm still only guessing as to why. I can only surmise that we were getting some type of false frame or something.

Dry Aquaman.

http://www.mitelforums.com

 

[SYQUEST]

Hi DryAquaman,
So in changing the DS1 circuit format from ESF/B8ZS to D4/AMI, you would also change the Mitel's DS1 port parameters to match, correct? DS1 format errors can cause the types of problems described, but without a design layout record(DLR) or a circuit topology, it is hard to know how the various parts of the circuit are provisioned and what options are set. In using test equipment, like a T-Berd analyzer, it can identify some of the problems. So, if everything was correct on your end, whatever mismatch there was, was caused by the carrier, and they may or may not have generated DS1 alarms, but a major format mismatch like ESF, instead of D4, will cause DS1 alarms. There is no doubt about that... Referring back to some of the things I mentioned above, in the world of converting from one format to another, be it TCP/IP, or XDSL, or any other serial format, if the parameters don't match, you will have problems with it operating correctly. Did your client's carrier give any clues as to what was mismatched in the circuit?

....JIM....

 

[dryaquaman]

I used a Sage T1 tester rather than a T-berd. I can verify every thing matched at that level.
When we changed the format to D4/AMI the carrier and I both did it at the exact same time (or it wouldn't have worked at all).

Together we went over and over the conf on both our ends and we never found any type of mismatch.
The only thing we changed that resolved it was we both changed to D4/AMI.

DryAquaman

http://www.mitelforums.com

 

[lopes1211]

I have seen something like this before when there was a copper leg in the last mile of the circuit. This last mile was extended from the CO using HDSL cards which were optioned for D4 instead of ESF. CPE was ESF, and C.O. settings matched as ESF, but there was this HDSL copper extension manhole to manhole in the middle optioned for D4. Hard if not impossible for the LEC to "see" this since they are only looking at the C.O..

-CL