Power Outage and Performance Problems 1

[lksixt]

Hi all,

We had a power outage over the weekend, when I came in Monday servers weren't talking. I shut everything down, powered up the 2621 router, then the switch, then the domain controllers... etc.

Now things are working but are very slow, a backup that usually took 3 hours took 10 last night.

I've compared configs from before (I try to document all configs and changes) nothing looks different...

Is there something that could have caused a performance hit? ROuter or Switch?

Still quite new to this part of the job, so all help would be appreciated...

laura

 

[dnels]

Laura,

I would

1. Run pings from one of the devices you are backing up to the backup server to see if there is any packet loss or funny response times.

2. Traceroute the same path to verify the traffic pattern are what you would expect.

3. Look for an interface that didn't come up properly, probably in an up/down state.

Dave

 

[thethmon]

I would look into the connections to your switch and insure that you have no duplex mis-matches. This is notorious for causing slowness. It's always been our policy to not use autonegotiation because this is not very reliable and can cause duplex mis-matches as well. Todd Hethmon
thethmon@hethmon.com

 

[lksixt]

Thanks guys...

I ran tracert (?) and pingpath (per w2k help) and they seem to be going to the right places. Not sure about the timing.

I ran a show config all on my switch (god there is a lot of stuff)

Am I looking for duplex on the set port stuff?

My router is duplex auto on both 0/0 and 0/1

dazed and confused

 

[dnels]

I agree with Todd. I code the duplex and speed setting on router to switch links. If you do a 'sho int e0/0' and 'sho int e0/1' you should see the duplex settings. Not sure what kind of switch you have but a 'sho port' or 'sho int' should show the setting on the switch. Also look for collisions and errors on both sides. That is an indication of a duplex mismatch. This kind of problem will have a impact on high throughput apps like backups.

Dave

 

[lksixt]

I hope this isn't inappropriate, but here is my router stuff, looks like both of them are full duplex. I don't see any collisions. Below that is the 2948G switch info. Everything there seems to mixed. I'm not sure what to look for in the show int on the switch.

FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1340 (bia 0005.32cc.1340)
Description: External interface to PIX
Internet address is 10.0.15.2/29
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:01:15, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 145000 bits/sec, 89 packets/sec
5 minute output rate 42000 bits/sec, 84 packets/sec
2626553 packets input, 850741134 bytes
Received 1 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
2501681 packets output, 252573449 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 1350 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet0/1 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Description: Internal trunked (multi-vlan) interface
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 76000 bits/sec, 90 packets/sec
5 minute output rate 183000 bits/sec, 95 packets/sec
42645083 packets input, 2809037734 bytes
Received 52359 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
42762401 packets output, 3402720242 bytes, 0 underruns
0 output errors, 0 collisions, 7 interface resets
0 babbles, 0 late collision, 3839012 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet0/1.1 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Description: Administration VLAN
Internet address is 10.0.15.129/25
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 10.
ARP type: ARPA, ARP Timeout 04:00:00
FastEthernet0/1.2 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Description: Academic VLAN
Internet address is 10.0.18.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 20.
ARP type: ARPA, ARP Timeout 04:00:00
FastEthernet0/1.3 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Description: VPN VLAN
Internet address is 10.0.20.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 30.
ARP type: ARPA, ARP Timeout 04:00:00
FastEthernet0/1.4 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Internet address is 10.0.15.33/27
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ISL Virtual LAN, Color 1.
ARP type: ARPA, ARP Timeout 04:00:00



2948G> (enable) show interface
sl0: flags=50<DOWN,POINTOPOINT,RUNNING>
slip 0.0.0.0 dest 0.0.0.0
sc0: flags=63<UP,BROADCAST,RUNNING>
vlan 10 inet 10.0.15.132 netmask 255.255.255.128 broadcast 10.0.15.255
me1: flags=62<DOWN,BROADCAST,RUNNING>
inet 0.0.0.0 netmask 0.0.0.0 broadcast 0.0.0.0

2948G> (enable) show port
Port Name Status Vlan Level Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
2/1 notconnect 10 normal auto auto 10/100BaseTX
2/2 connected 10 normal a-full a-100 10/100BaseTX
2/3 connected 10 normal a-half a-10 10/100BaseTX
2/4 connected 10 normal a-full a-100 10/100BaseTX
2/5 connected 10 normal a-half a-10 10/100BaseTX
2/6 connected 10 normal a-full a-100 10/100BaseTX
2/7 connected 10 normal a-full a-100 10/100BaseTX
2/8 notconnect 10 normal auto auto 10/100BaseTX
2/9 connected 10 normal a-full a-100 10/100BaseTX
2/10 connected 10 normal a-full a-100 10/100BaseTX
2/11 connected 10 normal a-full a-100 10/100BaseTX
2/12 connected 10 normal a-half a-10 10/100BaseTX
2/13 connected 10 normal a-full a-100 10/100BaseTX
2/14 connected 10 normal a-full a-100 10/100BaseTX
2/15 connected 10 normal a-full a-100 10/100BaseTX
2/16 notconnect 10 normal auto auto 10/100BaseTX
2/17 notconnect 10 normal auto auto 10/100BaseTX
2/18 notconnect 10 normal auto auto 10/100BaseTX
2/19 notconnect 10 normal auto auto 10/100BaseTX
2/20 notconnect 10 normal auto auto 10/100BaseTX
2/21 notconnect 10 normal auto auto 10/100BaseTX
2/22 notconnect 10 normal auto auto 10/100BaseTX
2/23 notconnect 10 normal auto auto 10/100BaseTX
2/24 notconnect 10 normal auto auto 10/100BaseTX
2/25 notconnect 10 normal auto auto 10/100BaseTX
2/26 notconnect 10 normal auto auto 10/100BaseTX
2/27 notconnect 10 normal auto auto 10/100BaseTX
2/28 notconnect 10 normal auto auto 10/100BaseTX
2/29 notconnect 10 normal auto auto 10/100BaseTX
2/30 notconnect 10 normal auto auto 10/100BaseTX
2/31 notconnect 10 normal auto auto 10/100BaseTX
2/32 notconnect 10 normal auto auto 10/100BaseTX
2/33 connected 10 normal a-full a-100 10/100BaseTX
2/34 notconnect 10 normal auto auto 10/100BaseTX
2/35 connected 10 normal a-full a-100 10/100BaseTX
2/36 connected 30 normal a-half a-100 10/100BaseTX
2/37 connected 20 normal a-full a-100 10/100BaseTX
2/38 connected 20 normal a-half a-100 10/100BaseTX
2/39 connected 20 normal a-full a-100 10/100BaseTX
2/40 connected 20 normal a-full a-100 10/100BaseTX
2/41 connected 20 normal a-full a-100 10/100BaseTX
2/42 connected 20 normal a-full a-100 10/100BaseTX
2/43 connected 20 normal a-full a-100 10/100BaseTX
2/44 connected 20 normal a-half a-100 10/100BaseTX
2/45 notconnect 20 normal auto auto 10/100BaseTX
2/46 notconnect 20 normal auto auto 10/100BaseTX
2/47 connected trunk normal a-full a-100 10/100BaseTX
2/48 connected trunk normal a-full a-100 10/100BaseTX
2/49 notconnect 1 normal full 1000 No GBIC
2/50 notconnect 1 normal full 1000 No GBIC

 

[dnels]

A 'sho port x/x' and a 'sho mac x/x' on the specific ports on the switch that connect to the router would be helpful.

dave

 

[dnels]

Sorry, just a sho int x/x on the switch interface where the router is connected. I also noticed on the int f0/1 that you have 3839012 deferred packets. These are packets that are ready to be sent but can't be because of the state of the link.

Dave

 

[lksixt]

Dave, Thank you VERY much for your help...

here is the show mac from the switch

Mon Dec 16 2002, 10:25:25
2948G> (enable) show mac 2/48

Port Rcv-Unicast Rcv-Multicast Rcv-Broadcast
-------- -------------------- -------------------- --------------------
2/48 42918638 3163 8649

Port Xmit-Unicast Xmit-Multicast Xmit-Broadcast
-------- -------------------- -------------------- --------------------
2/48 44395545 533241 49017

Port Rcv-Octet Xmit-Octet
-------- -------------------- --------------------
2/48 42292262039 43105193411

MAC Dely-Exced MTU-Exced In-Discard Lrn-Discrd In-Lost Out-Lost
-------- ---------- ---------- ---------- ---------- ---------- ----------
2/48 0 0 0 0 0 0

Last-Time-Cleared
--------------------------
Mon Dec 16 2002, 10:25:25
2948G> (enable) show mac 2/47

Port Rcv-Unicast Rcv-Multicast Rcv-Broadcast
-------- -------------------- -------------------- --------------------
2/47 326024 464408 5003

Port Xmit-Unicast Xmit-Multicast Xmit-Broadcast
-------- -------------------- -------------------- --------------------
2/47 345959 71982 52629

Port Rcv-Octet Xmit-Octet
-------- -------------------- --------------------
2/47 129128245 219833981

MAC Dely-Exced MTU-Exced In-Discard Lrn-Discrd In-Lost Out-Lost
-------- ---------- ---------- ---------- ---------- ---------- ----------
2/47 0 0 0 0 0 0

Last-Time-Cleared
--------------------------
Mon Dec 16 2002, 10:25:25
2948G> (enable)

and the show port

Mon Dec 16 2002, 10:25:25
2948G> (enable) show port 2/48
Port Name Status Vlan Level Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
2/48 connected trunk normal a-full a-100 10/100BaseTX

Port AuxiliaryVlan AuxVlan-Status
----- ------------- --------------
2/48 none none


Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex
----- -------- --------- ------------- -------- -------- -------- -------
2/48 disabled shutdown 0 0 1 disabled 56

Port Num-Addr Secure-Src-Addr Age-Left Last-Src-Addr Shutdown/Time-Left
----- -------- ----------------- -------- ----------------- ------------------
2/48 0 - - - - -

Port Status Channel Admin Ch
Mode Group Id
----- ---------- -------------------- ----- -----
2/48 connected auto silent 51 0

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
----- ---------- ---------- ---------- ---------- ---------
2/48 - 0 2 0 0

Port Single-Col Multi-Coll Late-Coll Excess-Col Carri-Sen Runts Giants
----- ---------- ---------- ---------- ---------- --------- --------- ---------
2/48 0 0 0 0 1 0 0

Last-Time-Cleared
--------------------------
Mon Dec 16 2002, 10:25:25
2948G> (enable) show port 2/47
Port Name Status Vlan Level Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
2/47 connected trunk normal a-full a-100 10/100BaseTX

Port AuxiliaryVlan AuxVlan-Status
----- ------------- --------------
2/47 none none


Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex
----- -------- --------- ------------- -------- -------- -------- -------
2/47 disabled shutdown 0 0 1 disabled 55

Port Num-Addr Secure-Src-Addr Age-Left Last-Src-Addr Shutdown/Time-Left
----- -------- ----------------- -------- ----------------- ------------------
2/47 0 - - - - -

Port Status Channel Admin Ch
Mode Group Id
----- ---------- -------------------- ----- -----
2/47 connected auto silent 51 0

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
----- ---------- ---------- ---------- ---------- ---------
2/47 - 0 2 0 0

Port Single-Col Multi-Coll Late-Coll Excess-Col Carri-Sen Runts Giants
----- ---------- ---------- ---------- ---------- --------- --------- ---------
2/47 0 0 0 0 1 0 0

Last-Time-Cleared
--------------------------
Mon Dec 16 2002, 10:25:25
2948G> (enable)

 

[dnels]

Are you still having response time issues? Everything seems to look fine. I would keep an eye on the defered packets on f0/1 on the router. Was it just the backups that were slow?

 

[lksixt]

Sorry about all the data, the router is attached to 2/48, another switch is on 2/47 so i guess we can ignore that.

I tried a show int 2/48 on the switch and got

2948G> (enable) show int 2/48
Usage: show interface
show interface trap

When you say the packets can't be sent because of the state of the link, what does that mean and where do you see that and lastly, how do i fix it

 

[lksixt]

We're crossing paths here, yes things still seem slow. The backups are fine locally, just very slow on remote servers which is why I suspected either the router or the switch.

the really bad news about all of this is that several months ago I removed all my access-lists from the router and moved them to the pix which really speeded things up and also fixed problems with my backup and things have been smooth ever since...

of course two days before I'm ready to leave on vacation (antigua!!!!) this happens...

 

[dnels]

The official description is: Deferred indicates that the chip had to defer while ready to transmit a frame because the carrier was asserted. This could be normal for your envionment on a FE with subinterfaces. You'd have to have historical data to track it. The counter is the 3rd line from the bottom on the sho F0/1 interface below. There was a bug on the 3600 router platform where packets were deferred on an auto-negotiated FE interface. The recommendation was to code speed and duplex, which is what I tend to do.


FastEthernet0/1 is up, line protocol is up
Hardware is AmdFE, address is 0005.32cc.1341 (bia 0005.32cc.1341)
Description: Internal trunked (multi-vlan) interface
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of &quot;show interface&quot; counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 76000 bits/sec, 90 packets/sec
5 minute output rate 183000 bits/sec, 95 packets/sec
42645083 packets input, 2809037734 bytes
Received 52359 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
42762401 packets output, 3402720242 bytes, 0 underruns
0 output errors, 0 collisions, 7 interface resets
0 babbles, 0 late collision, 3839012 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

[lksixt]

Thanks David, I guess I'll look into how to do that next.

I appreciate all your help...

Cheers!

 

[dnels]

the bug on the 3600 is CSCds19771. To config speed and dupplex on the router:

config t
int f0/1
speed 100
duplex full
exit
exit

On the switch:

set port speed 2/48 100
set port duplex 2/48 full

And repeat for all interfaces/ports you need to change.

Have a good vacation.
Dave

 

[lksixt]

Dave, dave, dave

That did it my friend...

Remote backup back to 252mb/min (it was down around the 30mb/min range) and everything else is noticeably faster

It may seem strange to use a small test backup to monitor performance, but it works

Thank you, thank you...

I'll have a big tropical drink for you!