Error Code 803 Legend T1

[adc110]

I have a customer with a Merlin Legend CKE4 with a 100D module that is connected to their provider's Adtran. They are giving me a partial T (12 channels). Apparently the service provider decided to displace their switch at their CO with a new switch and moved the circuit over to their new switch about 3 weeks ago. We were unaware of this as it was done on a Sunday early morning. In any case, right after this we started getting a loud "BEEP BEEP BEEP........" after making a few successful calls. This happens to all users. I tested this and out of the 12 channels, I seem to always have the ability of making 5-6 successful calls, then (like clock work!!) get this long quiet pause when making the next call followed by the loud "BEEP BEEP BEEP BEEP" sound which continues until I hang up. This will happen for the next 2-3 attempts at dialing out then I will get the next 5-6 calls which will be successful. All the while this happens, the error log is plagued with Error CODE 803 which apparently (according to Lucent description) is "No External Release:Communication problems between module and CO. Far end has not disconnected within 4 minutes. If error occurs twice consecutively, trunk is busied out automatically whether or not automatic trunk maintenance busy is enabled. Check the nework interface. Check for faulty cable."

This was continually happening and can be re-created everytime. I have checked everything and then after contacting the provider it was mentioned that they had swapped T1 circuit to a new switch at about the same time!! The last email that I got from their switch technician was this:
"I have had a live signal state trace on that T1 intermittently, and I am seeing a lot of call setups from you with seize, wink, then nothing, and seize, wink, partial dials, with another attempt immediately following on the next sequential channel as if the system was partial dialing and then seizing another line to try again. Another strange issue I saw was you had a channel go into a constant seize, wink sequence (constant seizure) until I killed the channel and brought it back into service (this only happened once) and it was right after I made the line hunt change per our last conversation."

Our T1 is ESF/B8ZS/RB Signaling/E&M WINK and each of our trunks are configured as follows:

A TRUNK 810Slot/Port :8/ 2 Toll
A Direction: 2 Way E&M Signal: Type1SDialtone: Remote
A InType : WinkInMode: TouchTone AnsSupvr: 300ms
A OutType: WinkOutMode : TouchTone Disconnect: 300ms

Does anyone have any ideas as to what is going on? Any help would be much appreciated!

 

[phonesrus]

Have the CO confirm that they are sending you dialtone on all channels, you are set for remote dialtone. If you send a call out, you are sending a wink request and then waiting for dialtone/tone detector presentation before you outpulse digits, and may not be receiving it on all channels. Both sides are waiting and finally timing out due to a lack of response, and causing lockups.
They made the change, and now it's your problem, not likely! Make them check all provisioning and timing on the Adtran channel bank. Just running a trace is good, but it tells you what is happening, they need to find why.

 

[adc110]

I will certainly request them to review this. I will say, however that the error code 803 is happening on all channels as I can see it in the error log so would this make sense? Maybe it is a timing issue as you mention. Thank you for your help phonesrus

 

[SYQUEST]

You might want to have the Telco review and verify all 12 channels/trunks on a channel-by-channel or trunk-by-trunk basis to ensure that they have everything correct and they provisioned the CO switch and DS1 facility correctly for your client's DS1 service. Since it worked before the cut over, the burden is Telco's.

Just out of curiosity, what kind of CO switch (old/new) and what provider?

The error code 8403 is indicative of a Telco problem since that is where the false seizures are coming from.

....JIM....

 

[adc110]

Jim - thanks for the udpate. I will ask the provider (they are a localized in Vermont). I believe the last I spoke they had mentioned it was a Siemens switch but not sure.

 

[adc110]

Ok - so here is where we are at. We did a bunch of testing yesterday while the CO monitored/logged all activity. I negelected to ask the switch type but still believe it to be a Siemens. In any case, on this 12-channel T1 (not a PRI) the switch tech would see us make 5-6 successful calls in a row and then it would cause an error (essentially the infamous re-order tone beep beep beep .....) What he saw on his end was our Legend seizing a channel, and then he saw his switch send a Wink back and then the channel would go dead air (you can actually hear this on the legend as there seems to be a pause of dead air after dialing before the -re-order tone). They are now sending the detailed loggin of to the Switch manufacturer for diagnosis. The interesting thing is that we will get these errors 2-3 times in a row and then have the ability of making 5-6 good calls again. We (CO & I) switched from Wink to Delay which made no difference. They are thinking that it might be some sort of timing issue but if timing was an issue would this error out so consistently? And why not on more sporadic calls? I mean, we are able to always make 5-6 good calls, then 2-3 bad calls, then 5-6 good calls, then 2-3 bad calls, and it is cyclical from here. If timing was an issue it would seem it would work & not work sporadically but oh well!! Also - PHONESRUS - I had him check the settings for remote dialtone and he said that they were not configured to send remote dialtone (the field was set to false). I had him change this to send dialtone (as we are set to remote) which essentially made the following happen (we call out on a channel, wait through pause of silence, then hear another dialtone and then a disconnect) This did not resolve anything but seems a little wierd that we are set for remote (which is the way it was previously set on the other T1) with no issues and they are supposedly not set for providing dialtone. I will post when I find more!

 

[merlinman]

Do you know if the same channels are always failing?

If you have them "UNIQUELY" labeled, (805, 806, 807, ect.) then you can identify which ones are failing, and perhaps you could see a pattern.

Yes, the hunting within a pool is CIRCULAR, and that makes me believe you may only have a few lines that are your bad actors.

http://home.comcast.net/~merlinman

 

[merlinman]

Actually, I looked at the error a bit closer, and here's what I determined:

The Error code also reports the SLOT & PORT that had the problem.

Is the problem isolated to only certain ports on the T1 Module?

http://home.comcast.net/~merlinman

 

[SYQUEST]

Would you post a copy of the error codes as reported by the Legend? That way we can see the ports that are failing.

Also is incoming traffic affected?

....JIM....

 

[adc110]

Apparently, incoming traffic does not look like it is affected. The errors in the log indicate every channel and goes exactly in sequentional order (809 will have an error, then 810, 811, 812 all the way to 820 and then will cycle back to 809). The problem is across all channels.

 

[phonesrus]

OK, lets review a couple of things about E&M, and remember the switches are logical devices. I will refer to digital signaling, not analog, since this is a T-1 emulation.
A digital channel is 64kb, 56kb is for voice and 8kb is for signaling for E&M. 8kb is "robbed" from the channel for the signaling, hence the name.
Wink/Wink---The calling switch goes off-hook and winks (bit sequence)[ABCD bits], telling the receiving switch "I have a call for you"; the receiving switch goes off-hook and winks "I am ready to receive" The calling switch then sends a digit sequence [dialed digits] (bit sequence), the receiving switch accepts the digits and routes the call accordingly. This is the most reliable, since both sides have to agree.
Delay/Delay---Calling switch goes off-hook, receiving switch goes off-hook and after a timer expiry calling sends digits (bit sequence)This method assumes both switches are ready at the same time [timers are critical]. Remember E&M means Ear & Mouth-one side talks and the other listens!
Immediate/Immediate---Calling switch goes off-hook and sends digits, assumes receiving switch is immediately off-hook and ready to receive. [essentially no timers].

Dialtone may, or may not be present, this depends on the settings on both switches. If a switch is looking for dialtone and it is not presented, it may cause a problem and the converse is also true. Everything that is sent and received is done with a coded sequence of 1's and 0's, and if the code is not correct on either side then a corruption may occur, and it may be in one direction only.

 

[SYQUEST]

A correction to the WINK/WINK configuration. The calling switch goes off-hook by presenting a SEIZURE condition to the M lead or equivalent (toggles the signalling bits). This tells the receiving switch there is "a request for service", then the receiving switch acknowledges the request by sending a WINK signal back to the calling switch as a "green lite" to send the network address for the call to be processed. This sequence is the same for calling in the other direction for this type of arrangement.

Now there can be other combinations of signalling configurations for two-way trunking: WINK/DELAY or WINK/IMMEDIATE or DELAY/WINK or IMMEDIATE/WINK, in addition to the ones phonesrus mentioned above. Timing is everything for the WINK or DELAY types to function properly.

adc110,

It may be helpful if the Telco put up a T-Berd test set on the DS1 ckt (both sides TX & RX) and observed the channel signalling bits to get a full view of all 12 channels at once. Hopefully the problem can be identified and resolved then.

....JIM....

 

[adc110]

Thanks to the both of you Phonesrus & Syquest for the input. I all makes sense to me in conjunction with my own experience. I have already made a request to have them send a field tech for diagnostics. The switch vendor (as well as us) are trying to keep it a wink/wink configuration. The odd thing is that the vendor believes it to be some kind of timing issue but timing (at least from my perspective) would be alittle more sporadic as I mentioned before and not necessarily so predictive - but perhaps it's my inability to actually see what is going on. It's so unfortunate to not have the same logging/monitoring capabilities & features that are offered in the IP Office line. They have saved me numerous times already in our troubleshooting. I have about 14 customers with Legend/Magix as well as owned a Legend in my previous job and I do believe they are rock solid and extremely reliable but if you want to see "under the hood" it's left to the manufacturer for logging/interpretation. Too bad!

 

[merlinman]

Some of us can do it as well.

But, my question is:

Do you see a pattern in the error showing any particular LINE PORTS?

http://home.comcast.net/~merlinman

 

[phonesrus]

Sy
You are absolutely correct, I was trying to simplify the information just a bit, to not be too long winded. That is why I mentioned the ABCD bits, and did not mention the additional variables.

 

[adc110]

Merlin . . . When I look at the error log it shows each trunk which is tied to the T1 card for 12 channels (809 will have an error then the next error will be 810 all the way to 820 which is my 12 channels and then it will cycle back to 809 as the next error in the log)Unless I have my terminology wrong? I would assume that trunk 809 would grab channel 1 each time in the T1? Maybe I am wrong and trunk 809 would represent any available channel that it can seize at that moment?

 

[merlinman]

That's correct, so with a pattern like that, it really is no pattern, as far as a few failing.

Thus the correct next step is to get the provider involved.


Good luck!

http://home.comcast.net/~merlinman

 

[adc110]

Ok - so here is the update from the vendor as they sent a tech out on Friday to make a bunch of test calls all the while they had their switch manufacturer doing the traces:

"On Friday I sent a field tech to your customers site and had my vendor on the phone while making test calls.We were unable to replicate anything that was occurring using the test set, and there was no abnormal wink generation from the Metaswitch. At this time we continue to believe that there is a timer mismatch. What we believe is occurring when this happens is that your PBX is timing out not recognizing the wink back after your seizure. Since my tech had no issues, sending multiple back to back calls from his test set even fixed on a single channel, I would like to discuss you investigating anything you can about timer on your Merlin. Whether or not you can find a resource to know how to get to the wink related timers and either make those modifications or give me a dump of what you find so that I can try and correct the issue on this side. Please let me know what you find. The repair department has closed their ticket based on the findings by the field technician."

Anybody have some more insight based on "their" findings which of course didn't amount to much!!

 

[SYQUEST]

If it is a timing problem, it may be something like this:

There are several timing sequences when calling for service, (1) seizure to start-of-wink, (2) the wink itself and, (3) end-of-wink settling before digits are sent.

All of these time intervals are in milliseconds and have a minimum and maximum window in which to operate. I don't recall seeing any info on adjusting the E/M WINK timers on the Legend/Magix, but that may be available. Merlinman may be able to shed some light on the timer situation at the machine level. Now, I just reread the info adc110 posted, and since INCOMING calls are working, the timing issue would seem to be in one direction only. So that would lead me to believe that the METASWITCH is not providing the proper timing sequence that falls within the Merlin's window for wink-start when it calls for service to the CO. That being said, because of the time that the Legend/Magix product has been on the market, I would doubt very much if it is a Merlin issue. All the years I have participated on this forum I do not recall any issues like this before. The Metaswitch is a relatively new softswitch and I know nothing about its history or reputation. So I would throw the burden back to the provider to adjust or modify its timers to be compatible with the Merlin.

....JIM....

 

[adc110]

Here is another snipet from the provider that just came in:

"My vendor, and field technician both were unable to replicate the issue, or see the “Seize Wink, timeout” scenario in the tests we performed, however as soon as it was plugged back into the PBX our field tech could get this issue to occur in 4 or less repeated calls .

Just so you know exactly what we see on our end, when you seize the line, we wink back and then wait for digits. We wait 16 seconds (thus the long pause before your PBX is beeping). If we get nothing in 16 seconds we give up, and the seizure is released. When your calls are failing I see the seizure (PBX is actually grabbing that line, where originally you assume there was no lines available this is not so since this can occur when all lines are available.

Thank you Syquest. I will look at what timers are available under the E/M settings to see if there is anything to modify. What is most odd is why wouldn't the issue happen every time?