Lots of CRC's and Input Errors

[mfoc]

So, I've gone round and round with Qwest now. They say the line looks great, but yet for the past few days, ny Internet T1 has been dropping. There are a ton of input errors and CRC's incrementing (this may have been happening before the line dropped for the first time), and when the line drops, obviously, I have an alarm on my WIC. I still have the CD as green, but the alarm comes on. I've swapped the WIC out and verified the settings with QWEST to make sure it's configured properly. Nothing has changed recently - just started happening Tuesday. When the line drops, it states "line protocol is down."

Here're a few items to look at:

sh int serial 0/0
Serial0/0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: connected to QwestVPN
Internet address is 65.117.41.222/30
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 11/255, rxload 44/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:14:05
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/12/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 269000 bits/sec, 43 packets/sec
5 minute output rate 69000 bits/sec, 38 packets/sec
39652 packets input, 29786817 bytes, 0 no buffer
Received 84 broadcasts, 0 runts, 0 giants, 0 throttles
22558 input errors, 1213 CRC, 21297 frame, 0 overrun, 0 ignored, 48 abort
35543 packets output, 9079608 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

sh service-module serial 0/0
Module type is T1/fractional
Hardware revision is 0.88, Software revision is 0.2,
Image checksum is 0x73D70058, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last user loopback performed:
dte loopback
duration 00:02:55
Last module self-test (done at startup): Passed
Last clearing of alarm counters 00:15:22
loss of signal : 0,
loss of frame : 0,
AIS alarm : 0,
Remote alarm : 0,
Module access errors : 0,
Total Data (last 67 15 minute intervals):
1 Line Code Violations, 13795 Path Code Violations
1948 Slip Secs, 4818 Fr Loss Secs, 1 Line Err Secs, 640 Degraded Mins
24792 Errored Secs, 24674 Bursty Err Secs, 4744 Severely Err Secs, 74 Unavai
l Secs
Data in current interval (772 seconds elapsed):
0 Line Code Violations, 255 Path Code Violations
8 Slip Secs, 7 Fr Loss Secs, 0 Line Err Secs, 5 Degraded Mins
110 Errored Secs, 110 Bursty Err Secs, 7 Severely Err Secs, 0 Unavail Secs

Qwest has pretty much told me that my router is bad and almost refuses to troubleshoot with me any more until I replace it. I don't buy it. Tha last three times I've had problems with this line, it's been because of their edge router. The last time it went down, they told me that I had too much internet traffic and it was causing their router to crash....????? They are supposed to be sending a local telco tech out this morning to see if they can run some head-to-head.

 

[robertjo24]

Issues with stubborn telco’s are the worst. I would insist on a head to head test from their smart jack/LIU at your location back to the provider. I would not rely upon tests performed remotely by them looping the smart jack/LIU, because there are some components in their equipment past the point of loop in the card. We have had issues with defective card slots in the NIU, where the telco has looped the smart jack and tested clean, yet we were still incrementing errors.
Just as a point of interest, have you checked the cabling between the router interface and their jack? It could cause some problems if the cable got pinched or attacked by rodents.
You never know…

 

[mfoc]

Just a little more info. This T1 comes off a DS3 in our server room. Fiber to the DS3 cabinet - coax from the FLM 150 to the mux - 6' patch from the mux breakout to the WIC. There's some confusion as to who owns the DS3 equipment. Our service manager tells us one thing, but the records indicate otherwise. The only equipment that's ours should be the 6' patch and in. I checked the cable and it was OK. It'll be interesting to see what they see off of that line.

 

[NTPeave]

Have you heard of the "Output Interpreter" from cisco.com? It's quite handy for troubleshooting some problems. It's reachable from the home page after you have logged in if you have an appropriate login. Ask your local Cisco folks for one - it shouldn't be particularly sensitive/hard to get access to. It works by just pasting in output from most of the 'show' commands, and then it gives recommendations of problems/fixes. As an example, here's what it says about the two show commands you included in your post:

SHOW INTERFACE SERIAL NOTIFICATIONS (if any)

Interface Serial0/0 (up/up)
INFO: Please click the link below to have the IP SUBNET CALCULATOR automatically
calculate the supported range of IP addresses for the configured network and subnet
mask.
'Ip Subnet Calculator for 65.117.41.222/30'

WARNING: This interface has received a high number of packets with incorrect
CRCs (corrupted data) (> 0.1%).
Problems that may cause this symptom include:
a. Noisy serial line
b. Serial cable is too long or cable from the CSU/DSU to the router is not
shielded
c. SCTE mode is not enabled on the DSU
d. The CSU line clock is incorrectly configured
e. A Ones density problem on the link (incorrect framing or coding
specification), exists
f. Verify the queuing strategies are the same on both ends of the link.
TRY THIS:
1. Ensure that the line is clean enough for transmission requirements. Shield
the cable if necessary.
2. Make sure the cable is within the recommended length (no more than 50 feet
[15.24 meters], or 25 feet [7.62 meters] for the link).
3. Ensure that all devices are properly configured for a common line clock.
Set serial clock transmit external (SCTE) on the local and remote DSU. If
you are attempting serial connections at speeds greater than 64 kbps with
a CSU/DSU that does not support (SCTE), you might have to invert the
transmit clock on the router. Inverting the transmit clock compensates
for phase-shifts between the data and clock signals.
4. Make certain that the local and remote CSU/DSU are configured for the
same framing and coding scheme as that used by the leased-line or other
carrier service (for example, ESF/B8ZS).
5. Contact your leased-line or other carrier service and have them perform
integrity tests on the line.

WARNING: This interface has received a high number of packets (more than 1% total
input interface traffic) with framing errors.
A framing error occurs when a packet does not end on an 8-bit byte boundary.
Problems that may cause this symptom include:
a. Noisy serial line
b. Improperly designed cable; serial cable is too long; the cable from the
CSU or DSU to the router is not shielded
c. SCTE (serial clock transmit external) mode is not enabled on the DSU; the
CSU line clock is incorrectly configured; one of the clocks is configured
for local clocking
d. There is a Ones density problem on the link (incorrect framing or
coding specification)
TRY THIS:
1. Ensure that the line is clean enough for transmission requirements. Shield
the cable if necessary.
2. Make sure the cable is within the recommended length (no more than 50 feet

[15.24 meters], or 25 feet [7.62 meters] for the link).
3. Ensure all devices are properly configured for a common line clock. Set
serial clock transmit external (SCTE) on the local and remote DSU. If you
are attempting serial connections at speeds greater than 64 kbps with a
CSU/DSU that does not support (SCTE), you might have to invert the
transmit clock on the router. Inverting the transmit clock compensates for
phase-shifts between the data and clock signals.
4. Make certain that the local and remote CSU/DSU are configured for the same
framing and coding scheme as that used by the leased-line or other carrier
service (for example, ESF/B8ZS).
5. Contact your leased-line or other carrier service and have them perform
integrity tests on the line.

REFERENCE: For more information on Serial Lines, see:
Troubleshooting Serial Line Problems
Configuring Serial Interfaces
Troubleshooting Serial Lines
Loopback Tests for T1/56K Lines

REFERENCE: Command Reference - 'show interface serial'

SHOW SERVICE-MODULE NOTIFICATIONS (if any)

For the 'T1/fractional' interface:
INFO: With Extended Super Frame, the remote alarm condition is signaled out
of band, in the Facility Data Link. Thus with ESF, it is recommended to enable
remote alarms with the 'service-module t1 framing esf' command.

NOTE: Although the output from the 'show service-module serial [number]' command
can be used to troubleshoot some problems, it cannot assist with all problems. It
should be used in conjunction with other commands such as 'show interface serial',
'show isdn status' and 'show isdn service' (if ISDN is applicable). Submit the
output from these commands to Output Interpreter for an expert analysis.

REFERENCE: For more information, see: Configuring Serial Interfaces for CSU/DSU
Service Modules.

 

[NTPeave]

Have you heard of the "Output Interpreter" from

http://www.cisco.com?

It's quite handy for troubleshooting some problems. It's reachable from the home page after you have logged in if you have an appropriate login. Ask your local Cisco folks for one - it shouldn't be particularly sensitive/hard to get access to. It works in one of several cool ways by just pasting in output from most of the 'show' commands, and then it gives recommendations of problems/fixes. As an example, here's what it says about the two show commands you included in your post:

SHOW INTERFACE SERIAL NOTIFICATIONS (if any)

Interface Serial0/0 (up/up)
INFO: Please click the link below to have the IP SUBNET CALCULATOR automatically
calculate the supported range of IP addresses for the configured network and subnet
mask.
'Ip Subnet Calculator for 65.117.41.222/30'

WARNING: This interface has received a high number of packets with incorrect
CRCs (corrupted data) (> 0.1%).
Problems that may cause this symptom include:
a. Noisy serial line
b. Serial cable is too long or cable from the CSU/DSU to the router is not
shielded
c. SCTE mode is not enabled on the DSU
d. The CSU line clock is incorrectly configured
e. A Ones density problem on the link (incorrect framing or coding
specification), exists
f. Verify the queuing strategies are the same on both ends of the link.
TRY THIS:
1. Ensure that the line is clean enough for transmission requirements. Shield
the cable if necessary.
2. Make sure the cable is within the recommended length (no more than 50 feet
[15.24 meters], or 25 feet [7.62 meters] for the link).
3. Ensure that all devices are properly configured for a common line clock.
Set serial clock transmit external (SCTE) on the local and remote DSU. If
you are attempting serial connections at speeds greater than 64 kbps with
a CSU/DSU that does not support (SCTE), you might have to invert the
transmit clock on the router. Inverting the transmit clock compensates
for phase-shifts between the data and clock signals.
4. Make certain that the local and remote CSU/DSU are configured for the
same framing and coding scheme as that used by the leased-line or other
carrier service (for example, ESF/B8ZS).
5. Contact your leased-line or other carrier service and have them perform
integrity tests on the line.

WARNING: This interface has received a high number of packets (more than 1% total
input interface traffic) with framing errors.
A framing error occurs when a packet does not end on an 8-bit byte boundary.
Problems that may cause this symptom include:
a. Noisy serial line
b. Improperly designed cable; serial cable is too long; the cable from the
CSU or DSU to the router is not shielded
c. SCTE (serial clock transmit external) mode is not enabled on the DSU; the
CSU line clock is incorrectly configured; one of the clocks is configured
for local clocking
d. There is a Ones density problem on the link (incorrect framing or
coding specification)
TRY THIS:
1. Ensure that the line is clean enough for transmission requirements. Shield
the cable if necessary.
2. Make sure the cable is within the recommended length (no more than 50 feet

[15.24 meters], or 25 feet [7.62 meters] for the link).
3. Ensure all devices are properly configured for a common line clock. Set
serial clock transmit external (SCTE) on the local and remote DSU. If you
are attempting serial connections at speeds greater than 64 kbps with a
CSU/DSU that does not support (SCTE), you might have to invert the
transmit clock on the router. Inverting the transmit clock compensates for
phase-shifts between the data and clock signals.
4. Make certain that the local and remote CSU/DSU are configured for the same
framing and coding scheme as that used by the leased-line or other carrier
service (for example, ESF/B8ZS).
5. Contact your leased-line or other carrier service and have them perform
integrity tests on the line.

REFERENCE: For more information on Serial Lines, see:
Troubleshooting Serial Line Problems
Configuring Serial Interfaces
Troubleshooting Serial Lines
Loopback Tests for T1/56K Lines

REFERENCE: Command Reference - 'show interface serial'

SHOW SERVICE-MODULE NOTIFICATIONS (if any)

For the 'T1/fractional' interface:
INFO: With Extended Super Frame, the remote alarm condition is signaled out
of band, in the Facility Data Link. Thus with ESF, it is recommended to enable
remote alarms with the 'service-module t1 framing esf' command.

NOTE: Although the output from the 'show service-module serial [number]' command
can be used to troubleshoot some problems, it cannot assist with all problems. It
should be used in conjunction with other commands such as 'show interface serial',
'show isdn status' and 'show isdn service' (if ISDN is applicable). Submit the
output from these commands to Output Interpreter for an expert analysis.

REFERENCE: For more information, see: Configuring Serial Interfaces for CSU/DSU
Service Modules.

 

[NTPeave]

Sorry! Trigger happy on the 'submit post' button...
As penance, I offer a better description of what cisco.com claims its output interpreter will do for you. You'll still need a login though...

"Did you know that Output Interpreter supports the following functionality?

Show command outputs from your Router, Switch or PIX Firewall. Reference the list of supported show commands for details.

Error Messages generated by your Router, Switch or PIX Firewall. Simply copy and paste your Error or Log Messages from your Router, Switch or PIX Firewall into the Output Interpreter.

Decodes and analyzes your Router or Switch's stack trace for any possible bugs. Copy and paste the "show version" command output followed by Traceback or Stack Trace and Alignment data.

Is able to convert the 'apply', 'conduit' and 'outbound' statements of your PIX Firewall to equivalent 'access-list' statements. Copy and paste "show tech-support" or "write terminal" command output of your PIX Firewall.

Decodes and analyzes Configuration Register. Copy and paste the "show version" or "show tech-support" command output into the Output Interpreter

 

[mfoc]

NTPeave - That's an interesting tool. I have a login at cisco.com, so I mght have to play with that a little more.

Well, I think we fixed the problem. The local tester came out and took a look at the line. Of course, the physical layer looked great (as usual), but I was still incrementing errors. We determined that the problem indicated some sort of clock source issue. My router is set to line, and their edge router was set to internal. This is the way they claim it should be in order to operate normally. Well, they switched their router line also and it cleared all the errors right up. I haven't had a single error since they changed it yesterday at about 4PM.

So, here's the funny thing... They are now confused as to how each router is getting it's clocking signal. Apparently, there's no equipment in between that could actually provide this. The tech that was on-site said my Widebank 28 mux had the capability, but it was set to internal, which he said actually meant line according to the manual on the unit....????? Are these guys professionals? I really wonder sometimes. Before the tech came out, it had reached the point of them not troubleshooting the problem any more until we REPLACED OUR ROUTER!!!

You know, the past three times now that I've had a problem on this line, I waste hours and hours of time having to prove to Qwest that my equipment is good. Each time, it turns out to be something on their side. It's getting old. Time to look for a redundant solution.

Thanks

 

[marsd]

It's not unusual for this sh!t to happen.
I worked for three years with an smds circuit that
was very 'weird' and learned to diagnose DCE issues.
I had one bad cable and two bad config errors. The bad
cable isue was mine and the bad config issues were theirs.
If you look at the US IT marketb you will see why this
sh!t happens. The senior guys are fired and new guys are
brought in at a certain salary level. Set your clockrate
by that.