Tips for examining packet loss

[LingaringBell]

I have a PTP line between two buildings with a 1800 series Cisco on each end. I graph information from each router using snmp. Lately I am seeing packet discards on my interface graphs. I think this problem is due to overstuffing, but I'm not certain. What is a good method of figuring out why a router is dropping packets?

I'm sure that there must be a good article on cisco's website, but I just can't seem to search for the right words. Thank you.
-Bell

 

[VinceWhirlwind]

Start with a "show interface ..." from each end and see what they say.

What's a "PTP line"?

 

[ISPKing]

point to point, most likely a t1.

CCNP

 

[LingaringBell]

yes, it is a point to point t1 line. I have done a show interface on each end and I can see that errors are occurring, but I'm not sure where to go from there.


Router #1

Serial0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Description: PTP
Internet address is x.x.x.x/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 11/255, rxload 10/255
Encapsulation PPP, LCP Open
Open: IPCP, CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 198367
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 66000 bits/sec, 105 packets/sec
5 minute output rate 72000 bits/sec, 102 packets/sec
145467735 packets input, 3049646782 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 1 giants, 0 throttles
532 input errors, 33 CRC, 155 frame, 0 overrun, 0 ignored, 344 abort
149080008 packets output, 2749782233 bytes, 0 underruns
0 output errors, 0 collisions, 16 interface resets
0 output buffer failures, 0 output buffers swapped out
4 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up




Router #2

Serial0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Internet address is x.x.x.x/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 12/255, rxload 12/255
Encapsulation PPP, LCP Open
Open: IPCP, CDPCP, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1w2d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 17386
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 75000 bits/sec, 142 packets/sec
5 minute output rate 74000 bits/sec, 140 packets/sec
100114378 packets input, 2859045148 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 1 giants, 0 throttles
147 input errors, 23 CRC, 63 frame, 0 overrun, 0 ignored, 61 abort
97740382 packets output, 297262269 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

 

[ISPKing]

Those are Hard errors, Post a show controllers.

CCNP

 

[LingaringBell]

What do you mean by "hard errors"? Here is a show controller for each:

Router #1

Interface Serial0
Hardware is PowerQUICC MPC860 with Integrated FT1 CSU/DSU module
TX and RX clocks detected.
idb at 0x8391C95C, driver data structure at 0x83924078
SCC Registers:
General [GSMR]=0x2:0x00000030, Protocol-specific [PSMR]=0x8
Events [SCCE]=0x0000, Mask [SCCM]=0x001F, Status [SCCS]=0x06
Transmit on Demand [TODR]=0x0, Data Sync [DSR]=0x7E7E
Interrupt Registers:
Config [CICR]=0x00365F80, Pending [CIPR]=0x00008800
Mask [CIMR]=0x60204400, In-srv [CISR]=0x00000000
Command register [CR]=0x600
Port A [PADIR]=0x0000, [PAPAR]=0x0F0F
[PAODR]=0x0000, [PADAT]=0xE3FD
Port B [PBDIR]=0x0380F, [PBPAR]=0x0C00E
[PBODR]=0x00000, [PBDAT]=0x0055C
Port C [PCDIR]=0x008, [PCPAR]=0x000
[PCSO]=0x0A0, [PCDAT]=0xF04, [PCINT]=0x000
Receive Ring
rmd(FF003030): status 9000 length 42 address 5ABFF64
rmd(FF003038): status 9000 length 42 address 5ABE564
rmd(FF003040): status 9000 length 4C address 5AC60E4
rmd(FF003048): status 9000 length 42 address 5ABB164
rmd(FF003050): status 9000 length 42 address 5AC2CE4
rmd(FF003058): status 9000 length 42 address 5ABDEE4
rmd(FF003060): status 9000 length 2F address 5ABD1E4
rmd(FF003068): status 9000 length 42 address 5ABF264
rmd(FF003070): status 9000 length 4C address 5AC1FE4
rmd(FF003078): status 9000 length 42 address 5ABFF64
rmd(FF003080): status 9000 length 42 address 5AC0C64
rmd(FF003088): status 9000 length 2F address 5ABD864
rmd(FF003090): status 9000 length 4C address 5AC12E4
rmd(FF003098): status 9000 length 42 address 5ABAAE4
rmd(FF0030A0): status 9000 length 42 address 5AC39E4
rmd(FF0030A8): status B000 length 42 address 5AC3364
Transmit Ring
tmd(FF0030B0): status 5C00 length 40 address 5AB570A
tmd(FF0030B8): status 5C00 length 25C address 597EF54
tmd(FF0030C0): status 5C00 length 40 address 5A9F20A
tmd(FF0030C8): status 5C00 length 40 address 5A99F2A
tmd(FF0030D0): status 5C00 length 40 address 5AA11EA
tmd(FF0030D8): status 5C00 length 40 address 5AB7D4A
tmd(FF0030E0): status 5C00 length 40 address 5AB570A
tmd(FF0030E8): status 5C00 length 40 address 5A9E54A
tmd(FF0030F0): status 5C00 length 40 address 5A9926A
tmd(FF0030F8): status 5C00 length 40 address 5AB63CA
tmd(FF003100): status 5C00 length 40 address 5AA916A
tmd(FF003108): status 5C00 length 40 address 5AB10EA
tmd(FF003110): status 5C00 length 4A address 5A945EA
tmd(FF003118): status 5C00 length 38 address 5A9590A
tmd(FF003120): status 5C00 length 40 address 5AA97CA
tmd(FF003128): status 7C00 length 40 address 5AB042A

tx_limited=0(16)

SCC GENERAL PARAMETER RAM (at 0xFF003C00)
Rx BD Base [RBASE]=0x3030, Fn Code [RFCR]=0x18
Tx BD Base [TBASE]=0x30B0, Fn Code [TFCR]=0x18
Max Rx Buff Len [MRBLR]=1548
Rx State [RSTATE]=0x18008440, BD Ptr [RBPTR]=0x3080
Tx State [TSTATE]=0x18000548, BD Ptr [TBPTR]=0x3120

SCC HDLC PARAMETER RAM (at 0xFF003C38)
CRC Preset [C_PRES]=0xFFFF, Mask [C_MASK]=0xF0B8
Errors: CRC [CRCEC]=0, Aborts [ABTSC]=0, Discards [DISFC]=12
Nonmatch Addr Cntr [NMARC]=0
Retry Count [RETRC]=0
Max Frame Length [MFLR]=1610
Rx Int Threshold [RFTHR]=0, Frame Cnt [RFCNT]=7205
User-defined Address 0000/0000/0000/0000
User-defined Address Mask 0x0000


buffer size 1524

PowerQUICC SCC specific errors:
835 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts




Router #2

Interface Serial0
Hardware is PowerQUICC MPC860 with Integrated FT1 CSU/DSU module
TX and RX clocks detected.
idb at 0x83952690, driver data structure at 0x83959DAC
SCC Registers:
General [GSMR]=0x2:0x00000030, Protocol-specific [PSMR]=0x8
Events [SCCE]=0x0000, Mask [SCCM]=0x001F, Status [SCCS]=0x06
Transmit on Demand [TODR]=0x0, Data Sync [DSR]=0x7E7E
Interrupt Registers:
Config [CICR]=0x00365F80, Pending [CIPR]=0x00008800
Mask [CIMR]=0x40204000, In-srv [CISR]=0x00000000
Command register [CR]=0x600
Port A [PADIR]=0x0000, [PAPAR]=0x0F0F
[PAODR]=0x0000, [PADAT]=0xE6FB
Port B [PBDIR]=0x0380F, [PBPAR]=0x0C00E
[PBODR]=0x00000, [PBDAT]=0x0255C
Port C [PCDIR]=0x008, [PCPAR]=0x000
[PCSO]=0x0A0, [PCDAT]=0xFCC, [PCINT]=0x000
Receive Ring
rmd(FF003030): status 9000 length 42 address 5ABCB64
rmd(FF003038): status 9000 length 69 address 5AC46E4
rmd(FF003040): status 9000 length 42 address 5AC46E4
rmd(FF003048): status 9000 length 42 address 5AC7464
rmd(FF003050): status 9000 length 42 address 5AC3364
rmd(FF003058): status 9000 length 3F address 5AC60E4
rmd(FF003060): status 9000 length 42 address 5AC2664
rmd(FF003068): status 9000 length 3A address 5AC6764
rmd(FF003070): status 9000 length 42 address 5ABF8E4
rmd(FF003078): status 9000 length 42 address 5ABB164
rmd(FF003080): status 9000 length 42 address 5AC2CE4
rmd(FF003088): status 9000 length 42 address 5AC5A64
rmd(FF003090): status 9000 length 42 address 5AC12E4
rmd(FF003098): status 9000 length 70 address 5AC1FE4
rmd(FF0030A0): status 9000 length 42 address 5ABD1E4
rmd(FF0030A8): status B000 length 3A address 5ABAAE4
Transmit Ring
tmd(FF0030B0): status 5C00 length 40 address 5AAD12A
tmd(FF0030B8): status 5C00 length 40 address 5A9EBAA
tmd(FF0030C0): status 5C00 length 40 address 5AA31CA
tmd(FF0030C8): status 5C00 length 40 address 5AAFDCA
tmd(FF0030D0): status 5C00 length 3D address 5AB372A
tmd(FF0030D8): status 5C00 length 46 address 5A9D22A
tmd(FF0030E0): status 5C00 length 40 address 5A998CA
tmd(FF0030E8): status 5C00 length 40 address 5AB570A
tmd(FF0030F0): status 5C00 length 40 address 5A9728A
tmd(FF0030F8): status 5C00 length 40 address 5AAF76A
tmd(FF003100): status 5C00 length 3C address 5A94C4A
tmd(FF003108): status 5C00 length 40 address 5AB2A6A
tmd(FF003110): status 5C00 length 40 address 5AB30CA
tmd(FF003118): status 5C00 length 38 address 5AA6B2A
tmd(FF003120): status 5C00 length 46 address 5AAD12A
tmd(FF003128): status 7C00 length 40 address 5A9EBAA

tx_limited=0(16)

SCC GENERAL PARAMETER RAM (at 0xFF003C00)
Rx BD Base [RBASE]=0x3030, Fn Code [RFCR]=0x18
Tx BD Base [TBASE]=0x30B0, Fn Code [TFCR]=0x18
Max Rx Buff Len [MRBLR]=1548
Rx State [RSTATE]=0x18008440, BD Ptr [RBPTR]=0x3068
Tx State [TSTATE]=0x18000548, BD Ptr [TBPTR]=0x30C8

SCC HDLC PARAMETER RAM (at 0xFF003C38)
CRC Preset [C_PRES]=0xFFFF, Mask [C_MASK]=0xF0B8
Errors: CRC [CRCEC]=0, Aborts [ABTSC]=0, Discards [DISFC]=0
Nonmatch Addr Cntr [NMARC]=0
Retry Count [RETRC]=0
Max Frame Length [MFLR]=1610
Rx Int Threshold [RFTHR]=0, Frame Cnt [RFCNT]=42237
User-defined Address 0000/0000/0000/0000
User-defined Address Mask 0x0000


buffer size 1524

PowerQUICC SCC specific errors:
61 input aborts on receiving flag sequence
0 throttles, 0 enables
0 overruns
0 transmitter underruns
0 transmitter CTS losts

 

[ISPKing]

Physical Layer. Get the telco to Test.

CCNP

 

[LingaringBell]

ISPKing, can you please explain to me how you drew this conclusion? Thanks.

 

[ISPKing]

147 input errors, 23 CRC, 63 frame, 0 overrun, 0 ignored, 61 abort

so you had 23 crc and 63 fram + 61 aborts.

Input errors are synonymous with Line errors. my advice, just ignore it. There are only 147 in one weeks time. Hardly worth looking into. When you start seen Hundreds a second or a minute, then it becomes performance impacting.

CCNP

 

[LingaringBell]

Normally I would agree, but every time I get a few errors I'm getting complaints about voip drop outs. At first I thought it was a QoS problem because it only seems to happen during high load, but even after I increased the reserved bandwidth for voip I'm still having phone issues.

I've called the company that manages the physical layer and they claim there are no problems with the line and I am over loading it with too much traffic. However, I have rate limits set on the serial interfaces on each side and the line has been there for years and never had a problem. Basically, I'm trying to prove to the line techs that the problem is on their end.

The only other idea I've had is that a rate limit on only applies to tcp traffic, so I am overloading the line with udp traffic (which I doubt, but weirder things have happened). What kind of errors would I see if that was the case? Thank you for all your help.

 

[burtsbees]

Have the telco do an aggressive test to the LEC---guaranteed they are only pinging the smartjack.

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[ISPKing]

Intrusive testing will bring down the line. Make sure you schedule a maintenance window.

CCNP

 

[burtsbees]

You forgot to mention the 4 carrier transitions and 16 interface resets all point to either the WIC or the smartjack or th

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[ISPKing]

or the mux?

CCNP

 

[burtsbees]

Layer 1---lol

It still could be the WIC, and if the user has one laying around, it would be a lot quicker to at least eliminate that as a possibility...

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!