Interface Flapping Issues 3

[DallasBPF]

Ok, so I am having issues with this router interface flapping and the circuit keeps bouncing. Below are the logs from that router for that specific interface... Let me know what you think;

WINSCOTT#sh logg
Syslog logging: enabled (1 messages dropped, 0 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)
Console logging: disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 9454 messages logged, xml disabled,
filtering disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
No active filter modules.
Trap logging: level informational, 9820 message lines logged
Log Buffer (4096 bytes):
.Sep 23 22:53:00.004: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
.Sep 23 22:53:09.988: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
.Sep 23 22:53:19.973: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
.Sep 23 22:53:29.957: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
.Sep 23 22:53:36.113: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
.Sep 23 22:53:38.109: %LINK-5-CHANGED: Interface Serial0/0:0, changed state to reset
.Sep 23 23:15:53.941: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
.Sep 23 23:15:54.045: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up
.Sep 23 23:15:54.561: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up
.Sep 23 23:16:22.678: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Up
Sep 23 23:28:52.042: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
Sep 23 23:31:31.790: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
Sep 23 23:34:03.601: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
Sep 23 23:34:05.597: %LINK-5-CHANGED: Interface Serial0/0:0, changed state to reset
Sep 23 23:34:05.617: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Down Interface flap
Sep 23 23:34:06.597: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to down
Sep 23 23:35:00.510: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
Sep 23 23:35:01.166: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up
Sep 23 23:35:02.170: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up
Sep 23 23:35:31.579: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Up
Sep 23 23:35:37.451: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
Sep 23 23:35:39.451: %LINK-5-CHANGED: Interface Serial0/0:0, changed state to reset
Sep 23 23:35:39.471: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Down Interface flap
Sep 23 23:35:40.451: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to down
Sep 23 23:35:53.423: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
Sep 23 23:35:55.419: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
Sep 23 23:36:43.344: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
Sep 23 23:36:44.652: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up
Sep 23 23:36:45.656: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up
Sep 23 23:37:16.417: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Up
Sep 23 23:46:02.383: %E1T1_MODULE-3-LINEFLAP: Interface Serial0/0:0 receiving excessive errored packets
Sep 23 23:46:05.403: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
Sep 23 23:46:07.399: %LINK-5-CHANGED: Interface Serial0/0:0, changed state to reset
Sep 23 23:46:07.419: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Down Interface flap
Sep 23 23:46:08.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to down
Sep 23 23:46:33.355: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
Sep 23 23:46:34.751: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up
Sep 23 23:46:35.755: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up
Sep 23 23:46:48.332: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to down (AIS detected)
Sep 23 23:46:50.332: %LINK-5-CHANGED: Interface Serial0/0:0, changed state to reset
Sep 23 23:46:51.332: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to down
Sep 23 23:47:46.237: %CONTROLLER-5-UPDOWN: Controller T1 0/0, changed state to up
Sep 23 23:47:47.209: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up
Sep 23 23:47:48.213: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up
Sep 23 23:48:09.033: %BGP-5-ADJCHANGE: neighbor 12.84.202.225 Up
WINSCOTT#



controller T1 0/0
framing esf
linecode b8zs
channel-group 0 timeslots 1-24




interface Serial0/0:0
description
bandwidth 15300
ip address #.#.#.# 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
encapsulation ppp
ip route-cache flow
no cdp enable
max-reserved-bandwidth 100



WINSCOTT#sh interfaces serial 0/0:0
Serial0/0:0 is up, line protocol is up
Hardware is DSX1
Description:
Internet address is #.#.#.#/30
MTU 1500 bytes, BW 15300 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation PPP, LCP Open
Open: IPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 8w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 710
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/27/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 15300 kilobits/sec
5 minute input rate 2000 bits/sec, 5 packets/sec
5 minute output rate 4000 bits/sec, 9 packets/sec
19911684 packets input, 1333062234 bytes, 0 no buffer
Received 0 broadcasts, 7 runts, 1683 giants, 0 throttles
6093022 input errors, 1491388 CRC, 2930389 frame, 0 overrun, 0 ignored, 167
1245 abort
28344028 packets output, 2101646143 bytes, 0 underruns
0 output errors, 0 collisions, 2145 interface resets
3 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
2803 carrier transitions
Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags
WINSCOTT#


------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[burtsbees]

7 runts, 1683 giants
2145 interface resets
2803 carrier transitions
6093022 input errors
0 output errors

No output errors...all input errors. What's on the other end? Looks like it's receiving frames that are too big, and a few that are too small, but I don't think MTU is the issue---is it part of an ISL VLAN on the other end?

Burt

 

[jneiberger]

What troubleshooting have you done so far? Have you had the provider test the circuit yet? If not, that's the first thing I'd do.

 

[DallasBPF]

Yes the circuit has been tested by the provider and they are coming clean the t1 shelf and dirty to the csu.


Below is the sh diag for that slot;


Slot 0:
C1760 1FE VE 4SLOT DV Mainboard Port adapter, 2 ports
Port adapter is analyzed
Port adapter insertion time unknown
EEPROM contents at hardware discovery:
Hardware Revision : 8.81
PCB Serial Number : FOC102854V8
Part Number : 73-7167-08
Board Revision : A0
Fab Version : 04
Product (FRU) Number : CISCO1760
EEPROM format version 4
EEPROM contents (hex):
0x00: 04 FF 40 03 16 41 08 51 C1 8B 46 4F 43 31 30 32
0x10: 38 35 34 56 38 82 49 1B FF 08 42 41 30 02 04 FF
0x20: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x30: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x40: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x50: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
WIC/VIC Slot 0:
T1 (1 port) Multi-Flex Trunk WAN daughter card
Hardware revision 1.0 Board revision B0
Serial number 0035206462 Part number 800-03568-04
FRU Part Number VWIC-1MFT-T1=
Test history 0x00 RMA number 00-00-00
Connector type PCI
EEPROM format version 1
EEPROM contents (hex):
0x20: 01 20 01 00 02 19 35 3E 50 0D F0 04 00 00 00 00
0x30: 58 00 00 00 06 09 23 00 FF FF FF FF FF FF FF FF


Other than that, I am going to take another router/card to the site and replace it temp. to see if it is the router/card and knock that off my list on Friday. But the last time I came across this type of scenario the providor loves to blame our equipment and it turns out to be their cabling to the local loop/termal point to the building/or a faulty niu/t1 shelf..

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[jneiberger]

Here's a trick I like to use with them. Go to the site and unplug your equipment from the NIU (or smartjack, whatever they want to call it). Then plug in a T1 loopback plug and have them test again. Quite often, the problem is in their NIU, but their tests only test to the front side of the unit. Put the loopback plug in there and tell them to test to your CSU again. Don't even mention the loopback plug again. If the test fails, tell them about the plug. This will prove that the problem is in the NIU. If the test passes, you know the problem is on your side.

 

[DallasBPF]

Thanks for that, I am planning on making a trip out there on Friday and will definetly test that!

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[jeter]

Depending on the NIU you can install a 9pin cable to your laptop run hyper terminal and review the errors on the NIU.

There will be a history of events. I like to pull the card for about 5sec and reinstall.. Wait for the circuit to go idle and reset the history and then monitor...

The carrier can test to the csu/dsu pass the NIU, make sure all your cables are in good condition.

Another thing you can do is install a loop plug on your equipment. Monitor the status then you can start pointing the trouble in the right direction....

You can also place another router for testing, I keep a Cisco 1720 with 2ea wics just for that reason alone...


SPC NVARNG
Tek-TIP Member 19,650

 

[DallasBPF]

Other than the monitoring of the niu, I was planning on doing the loopback on my equip first, if no errors then move it over to their equipment and have them test w/o telling them there is a loopback in it and see what happens from there, and I will just use that spare 1760 I have here if need be

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[burtsbees]

At this point, can you just copy the config to the other 1760 and bring the interfaces up to eliminate your end as the problem, or do you just have the one voice WIC?

Burt

 

[DallasBPF]

oh i have another... the problem with testing is that this office that is having issues is about a hour and half w/o traffic... I am uptown dallas and it is south of ft worth about 30 mins =)

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[burtsbees]

My company has an office in North Dallas (Addison, I believe). SMS---Systems Maintenance Services (

http://www dot

sysmaint dot com). We do hardware and configuration on servers (all makes), storage (all makes), and Cisco equipment. We offer SmartNet at discounted rates (usually). Check us out, and thanks for the stars!

Burt

 

[DallasBPF]

Sooo I went out there yesterday and tested the CSU and came back with no errors with the loopback adapter. Checked the cabling as well from the CSU to the NIU and it was fine...

So I called AT&T back to have them do their testing, and all I get back from them so far is they want to keep testing it... I don't get why they won't just send out the local lec to come test the smart jack...

Oh well... will keep you guys updated

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works