Erratic PRI behavior

[TRONIC1]

Hello all,
I have an issue here I just can't figure out. We're experiencing dropped calls on our 2 PRI circuits. Before it drops the caller hears a loud static. The called party doesn't hear it and the call drops. Here's one of the many reports I've looked at...
DS-1 LINK PERFORMANCE MEASUREMENTS DETAILED LOG REPORT

Counted Since: 9:45 am TUE FEB 2, 2010

Date Time ES BES SES UAS/FS CSS LOFC

02/08 11:43 10 0 7 0 3 0
02/08 11:28 1 0 1 0 1 0
02/08 11:13 7 0 5 0 4 0
02/08 10:58 3 0 2 0 2 0
02/08 10:43 14 0 9 0 6 0
02/08 10:28 8 0 5 0 1 0
02/08 10:13 2 0 1 0 2 0
02/08 09:58 0 0 0 0 0 0
02/08 09:43 0 0 0 0 0 0
02/08 09:28 5 0 4 0 3 0
02/08 09:13 0 0 0 0 0 0

The provider tells me that the tests they've run indicate something on our side. Any thoughts? If someone needs more info I'll be glad to provide it. I really need a hand here.
Thanks,
John

"Hope for the best, plan for the worst

 

[AvayaTier3]

who is your provider?
what does your ds1 programming look like?

what do you see in the hardware error logs?

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[TRONIC1]

Hi AvayaTier3,
Our provider is Telus, leasing lines from Bell (we're in Ontario). I've opened a ticket with them but all they've told me so far is that the errors they see seem to be coming from our switch (of course).
DS1 programming looks like...
DS1 CIRCUIT PACK

Location: 01A05 Name: 05DHAT982890BLC
Bit Rate: 1.544 Line Coding: b8zs
Line Compensation: 1 Framing Mode: esf
Signaling Mode: isdn-pri
Connect: network
TN-C7 Long Timers? n Country Protocol: 1
Interworking Message: PROGress Protocol Version: a
Interface Companding: mulaw CRC? n
Idle Code: 11111111
DCP/Analog Bearer Capability: 3.1kHz

T303 Timer(sec): 4


Slip Detection? y Near-end CSU Type: other
Page 2 of 2
DS1 CIRCUIT PACK

ESF DATA LINK OPTIONS

Network Management Protocol: tabs
Send ANSI-T1.403 One-Second Performance Reports? n
Far-end CSU Address: b

The errors displayed are...

HARDWARE ERROR REPORT - ACTIVE ALARMS

Port Mtce Alt Err Aux First Last Err Err Rt/ Al Ac
Name Name Type Data Occur Occur Cnt Rt Hr St

01A10 CO-BD 18 0 02/05/17:27 02/05/17:27 1 0 0 a y
01A1007 CO-TRK 04/01 18 0 02/05/17:27 02/05/17:27 1 0 0 a n
01B0117 ETH-PT 513 46226 02/05/16:24 02/05/16:24 1 0 0 a y
01B0817 ETH-PT 1538 0 02/05/17:24 02/08/13:22 255 3 4 a y
01B0117 ETH-PT 1538 0 02/05/17:24 02/08/13:22 255 3 4 a y
01A0621 PDMODULE 4020 769 40989 02/08/10:59 02/08/10:59 1 0 0 a n
01B0301 PDMODULE 4815 769 40989 02/08/11:01 02/08/11:01 1 0 0 a n
01B0309 PDMODULE 4816 769 40989 02/08/11:01 02/08/11:01 1 0 0 a n
01B0324 PDMODULE 4817 769 40989 02/08/11:02 02/08/11:02 1 0 0 a n
01A0621 PDMODULE 4020 513 0 02/08/11:11 02/08/11:12 3 180 0 a n
01B0309 PDMODULE 4816 513 0 02/08/11:11 02/08/11:12 3 180 0 a n
01B0301 PDMODULE 4815 513 0 02/08/11:11 02/08/11:12 3 180 0 a n
01B0324 PDMODULE 4817 513 0 02/08/11:11 02/08/11:12 3 180 0 a n

Thanks for the quick response. If there's any other info you need let me know.

"Hope for the best, plan for the worst

 

[AvayaTier3]

list config board 1a05 and show the type board and current FW

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[TRONIC1]

list configuration board 01a05

SYSTEM CONFIGURATION

Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa

01A05 DS1 INTERFACE TN464GP HW02 FW006 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
u u u u u u u u

"Hope for the best, plan for the worst

 

[AvayaTier3]

https://support.avaya.com/css/appma...x/C20090710161117669000461#wlp_Downloads_Page

Read the description for FW 22 - fixes board resets.

I would upgrade your board to FW 24

sounds like FDL messages being sent to your switch from your provider are causing the boards to reset.

This Firmware vintage 22 (FV22) is ONLY applicable to TN464HP HV13 and TN2464CP HV13
It is NOT applicable to any previous versions of TN464HP, TN464GP, TN2464BP, or TN2464CP. Any attempts to download this firmware to these boards will abort.

TN464HP / TN2464CP HV13+ FV22 eliminates a problem where if a maximum-size (268 byte) LAN bus packet is sent to the board, the board would go out of service. A workaround for this issue is to change the "Max Message Size to Send" to a value less than 257. FW22 fixes this issue and allows maximum message sizes to be used.

Previously, if a board received certain rare ESF Data Link messages from the far end, (or a corrupted signal that is interpreted as one of the problematic ESF DL messages) the board could log errors and possibly go out of service. This has been corrected.

Previously, when a TN464GP/TN2464BP HV4 and greater or TN464HP/TN2464CP received a red alarm from the far end and subsequently the red alarm was cleared, the circuit pack did not clear its yellow alarm signal in the proper length of time (15 +/- 5 seconds). This has been corrected.


A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[dufus2506]

I'd have your provider check the span. I'll bet you a dozen donuts that part of the span is optioned incorrectly.

I've run into quite a few of these problems, and they all turned out the same way. A span that was supposed to use ESF framing had a segment that was configured as D4. (Or, the other way around.)

Tell your provider to bring their T-Berd out, and run a full end-to-end test on the span. That will demonstrate that it's not your equipment. (Of course their testing shows problems with your equipment! They're only testing in your direction, not both!) After that, they'll have to move down the span's circuit to test at different fiber muxes, and other equipment. Then they'll find the incorrect leg of the span.

Carpe dialem! (Seize the line!)

 

[AvayaTier3]

Your provider may be able to turn off Facility Downlink messages. This is a layer 2 issue and you will not see this with a test set.

If the provider has this ability. You may have to get the circuit re-engineered. I have had a provider change from b8zs to d4 to turn off FDL's.

Trust me, upgrade your fw first. It's much easier.

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[TRONIC1]

Tech on site currently. I'll let you know how it goes. At least now I can speak to him in an intelligent manner about the possible issue. You guys are awesome. Thanks so much.

"Hope for the best, plan for the worst

 

[TRONIC1]

OK, Here's what I have from the Telus tech. He connected to their cards in the building and recorded 59 error second errors. I then got a call from tech at Telus saying he's looked at the screenshots the onsite tech sent him and the errors were only 2 error seconds. Negligable is how he referred to it. I'm lost here. We have a company we've done business with in the past with some Definity gurus on staff. Maybe they can help, as I've been dealing with this for a week now and can't track the problem down. The tech I talked to at Telus told me that nothing has changed on their network, so I don't know if doing a firmware upgrade on my ds1 cards would help. Arggggggh.

"Hope for the best, plan for the worst

 

[dufus2506]

It's the telus span.

Look at your DS-1 errors. It only takes a few span error seconds to knock out your span. If they think 59 error seconds aren't something to worry about, then you need to escalate this beyond the techs you're talking to.

Ask to speak to the tech's manager. Don't let them push you around.

Carpe dialem! (Seize the line!)

 

[TRONIC1]

Generally speaking, I'm a pretty easy going guy, but I think at this point I'm left with no choice but to do exactly as dufus2506 suggests and start being a hard ass. Telus' stance is that it's our equipment by the errors that they see, but when I quiz them about the errors specifically, they can't tell me what these errors are or mean. Of course, having Bell in the mix isn't helping, just one more layer of bs, I may have to try and contact them directly. I'll be posting to this tomorrow, when I'm a little more coherent and have something valueable to add. I'll keep this up to date, right up to resolution, in the hopes it helps somebody in the future. Thanks to everyone for all of your thoughts and support. Greatly appreciated.

"Hope for the best, plan for the worst

 

[1a22ip]

You got one good pi$$ing contest going. Loop back the DS1 card 01a05. If the Avaya is the issue the errors will not go away. At the same time loop the span (circuit) back to the telco and have the tech re-test. If errors persist the problem is the telco and tell the to fix them to FIX IT.

Hope this helps
ED

1a2 to ip I seen it all

 

[TRONIC1]

Hello all,
Sorry to have let this lapse so long.
Before I begin, some switch detail.
System: G3csiV12 Software Version: R012i.01.1.414.1

Terminal Type (513, 715, 4410, 4425, VT220): [513]

The problem of dropped calls that are preceeded by intense static on the callers end still exists. Here's the update. My boss decided to buy 2 DS1 cards and have me swap them out for the existing. This did not solve the problem. I tried to download the firmware update for the DS1 boards but don't have privileges, so I can't get them off of Avaya's support site. My boss has been in contact with Telus, and now they're saying to change out the cables from their demarc to the DS1 cards (of course). I'd rather not, as I feel this isn't the problem. I've scoured this site, and it's given me some, what I think, are very good ideas to prove to myself and the provider where the problem exists, so I'm here to ask what the gurus think.
1. It's been suggested in other threads that a CSU module could cause the problems I've described. I don't think I have this module. Would it show in the list when I do a list config all?
2. The cables running from the Telus demarc to our DS1 cards appears to be a DB15 to Amphenol. If I disconnect from the demarc and cross pin 1>3 and 9>11 on my side will this perform a loopback test on our system?
3. I can connect to the RS232 connector on Telus' demarc cards a run hyperterminal at the same time as my local system loopback. I presume if I see errors on my local loopback them it's the phone system, if not and there are errors are on the Telus cards then I can go back on Telus and show them proof of their system failure.

Does all of this sound logical? Any advice or suggestions are appreciated.

Thanks,
John

"Hope for the best, plan for the worst

 

[AvayaTier3]

http://www.bshtele.com/download/tn464hp-h0-f24-sig.bin

download your fw from link above

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[AvayaTier3]

http://www.bshtele.com/download/tn464hp-h0-f24-sig.bin

download your fw from link above

TN464GP HW02 FW006 <-- for your previous board

Not sure if compatible with your new DS1 board(s)

A great teacher, does not provide answers, but methods to teach others "How and where to find the answers"

bsh

36 years Bell, AT&T, Lucent, Avaya
Tier 3 for 26 years and counting

http://bshtele.com

 

[TRONIC1]

It looks like the boards are GP

01A05 DS1 INTERFACE TN464GP HW02 FW006 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
u u u u u u u u


01B09 DS1 INTERFACE TN464GP HW02 FW006 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
u u u u u u u u

So I guess the required firmware is...

tn464gp-f24-sig.bin

Any thoughts on whether my idea for the local loopback test would work?




"Hope for the best, plan for the worst

 

[mikeydidit]

tn464gp-f24-sig.bin is the latest for your version of boards. In the FAQ section is a section on how to do firm ware upgrades if you need it. Also the integrated CSU is an adapter that would fit on the backplane of the carrier for the slot you are using. If you have one your cable to your circuit will plug right into it. If not, you will have a little box this pugs into then to your switch is a amphinol cable.

Most all circuits I see have a smart jack on them that you can use for testing. I would look at your providers box that has their circuit cards and see if you can find one. If so we can help you build a loopback plug to test with your provider.

When is the last time you helped someone, just because you were able to?

For the best response to a question, read faq690-6594

 

[TRONIC1]

Thanks so much everyone. There is a smartjack plug on the carrier demarc. I found this thread for making the loopback plug.

http://www.tek-tips.com/viewthread.cfm?qid=1489905

I'm checking for the external CSU as I don't see any access on the backplane. I'll post shortly.

"Hope for the best, plan for the worst

 

[TRONIC1]

I see no test ports on the Definity side at all. Admittedly, there are a mess of cables but I've searched and see no way of doing a loopback on the phone system. Am I just chasing my tail here? When I do the loopback on the carrier demark, does this in effect separate the two units so I can determine which side is causing the problem. Any guidance is appreciated.

"Hope for the best, plan for the worst