Unexplained Output Drops

[GM2005]

Hi

I have an issue with a server uplink on a Catalyst 6500. The Output Drops are incrementing yet I have no other interface errors. I have used SPAN and wireshark to look for excess broadcast etc but found nothing but general SMTP traffic etc.

Interface Output is below:

FastEthernet4/29 is up, line protocol is up (connected)
Hardware is C6k 100Mb 802.3, address is 0030.b6c8.bd68 (bia 0030.b6c8.bd68)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 8/255, rxload 7/255
Encapsulation ARPA, loopback not set
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:33:59
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4195
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2777000 bits/sec, 458 packets/sec
5 minute output rate 3324000 bits/sec, 611 packets/sec
890212 packets input, 682070067 bytes, 0 no buffer
Received 33 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
1200509 packets output, 796152958 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

[burtsbees]

Post a sh run int fa4/29, sh run vlan whatever, and any acls associated with this interface. Is the server directly connected to the interface?

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[GM2005]

Here is the show run for that interface:


interface FastEthernet4/29
description *********
no ip address
switchport
switchport access vlan 150
no cdp enable
end

The server is Exchange and it is patched directly in.

Thanks

 

[burtsbees]

No acl on vlan 150?

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[GM2005]

Hi

There is a single acl to prevent TCP or UDP 4444/4434 but it doesn't seem like there are enough hits to warrant the number of drops. Since I cleared the acl counters yesterday there are roughly 520 matches here but (see below) drops:

Extended IP access list 115
10 deny tcp any any eq 4444 (57 matches)
20 deny tcp any eq 4444 any (291 matches)
30 deny udp any any eq 4444 (3 matches)
40 deny udp any eq 4444 any (140 matches)
50 deny udp any any eq 1434 (22 matches)
60 permit ip any any (811026 matches)

Last clearing of "show interface" counters 21:29:17
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 88355

 

[burtsbees]

Is the server NIC set to 100/full?

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[GM2005]

Yes, i've had it checked and double checked. If it were that the NIC is being overwhelmed surely I would get some indication of the port counters?

 

[VinceWhirlwind]

do a
show int f4/29 switching

also, put
ip acc mac-address
on the interface and then do a
sh int mac-account

 

[GM2005]

Hi

The output from the switching is below, but it does not support ip accounting. The last line is a bit odd though so I have pasted the interface output again below



sh int fast 4/29 switching
FastEthernet4/29 DTCSMSXB01
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 0 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
No traffic sent or received on this interface.


FastEthernet4/29 is up, line protocol is up (connected)
Hardware is C6k 100Mb 802.3, address is 0030.b6c8.bd68 (bia 0030.b6c8.bd68)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 12/255, rxload 10/255
Encapsulation ARPA, loopback not set
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 20:22:59
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 80139
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4276000 bits/sec, 721 packets/sec
5 minute output rate 5091000 bits/sec, 956 packets/sec
17647977 packets input, 13194450509 bytes, 0 no buffer
Received 831 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
23589046 packets output, 15595741414 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

[GM2005]

I applied the accounting to the VLAN interface:


#sh int mac-accounting
Vlan150 Exchange Server LAN
Output (510 free)
0100.5e00.000d(82 ): 8 packets, 960 bytes, last: 26344ms ago
0100.5e00.0002(93 ): 82 packets, 9348 bytes, last: 924ms ago
Total: 90 packets, 10308 bytes

 

[VinceWhirlwind]

Do a show int vlan150 and a show run int vlan150 and try a show int vlan150 switching

 

[GM2005]

#sh int vlan 150
Vlan150 is up, line protocol is up
Hardware is EtherSVI, address is 0011.bc9c.2800 (bia 0011.bc9c.2800)
Description: Exchange Server LAN
Internet address is x.x.x.x/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d19h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 8598000 bits/sec, 1438 packets/sec
5 minute output rate 1854000 bits/sec, 407 packets/sec
L2 Switched: ucast: 50037216 pkt, 29074168318 bytes - mcast: 930841 pkt, 527741872 bytes
L3 in Switched: ucast: 57438171 pkt, 47025523041 bytes - mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 26525914 pkt, 14747693527 bytes mcast: 0 pkt, 0 bytes
115818875 packets input, 94579120974 bytes, 0 no buffer
Received 941473 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
26622574 packets output, 14648114092 bytes, 0 underruns
0 output errors, 0 interface resets
0 output buffer failures, 0 output buffers swapped out

interface Vlan150
description Exchange Server LAN
ip address x.x.x.x 255.255.255.0
ip access-group 115 in
ip access-group 115 out
ip helper-address x.x.x.x
no ip redirects
ip accounting mac-address output
ip pim sparse-mode
ip cgmp router-only
ip ospf message-digest-key 10 md5 7 1425171E1801383809091D
standby 150 ip x.x.x.x
standby 150 priority 10
standby 150 preempt
end


#sh int vlan 150 switching
Vlan150 Exchange Server LAN
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 67351 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 641951290 487861297056 2586648 289704576
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
IP Process 122899396 11528777436 59200307 6780909075
Cache misses 9139
Fast 867696 118461996 10783 914243
Auton/SSE 177969716521 115516718903398 33659941168 13296106214934
DEC MOP Process 773337 92405841 258044 33287676
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
ARP Process 27015486 1621013944 20233987 2266206544
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0

 

[burtsbees]

Post a sh access-list 115

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[GM2005]

Thanks. here it is.


sh access-lists 115
Extended IP access list 115
10 deny tcp any any eq 4444 (540 matches)
20 deny tcp any eq 4444 any (8069 matches)
30 deny udp any any eq 4444 (126 matches)
40 deny udp any eq 4444 any (739 matches)
50 deny udp any any eq 1434 (136 matches)
60 permit ip any any (3942722 matches)

These have been cleared in the last week.

 

[burtsbees]

Weird...Can you plug that server into a different blade?

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[GM2005]

Thats the hard part. It is already dual homed and in operation. The other switch port it is homed to looks exactly the same. I did a SPAN and captured 700Mb of traffic and its all SMTP and associated plus management traffic. There are a lot of out of sequence packets and packet fragments if that is any help.

I did intend to go down and investigate further but it will have to wait till Monday now.