Cisco 1841 bandwidth 100% on IPMAN/IPWAN Interface 1

[MelSysAdmin]

Hello All,

Can someone help me? I am administrating a Cisco 1841 Router. Just after Lunch time, we had major network issues in one of our offices. Our office is connected vai this router to a IPMAN/IPWAN to a co-location site to get to the www. The interface connected to the IPMAN/IPWAY had 100% bandwidth usage. I had to shut down the router and start again to gain network speed back.

Can someone advise me how i can track what caused this 100% usage?

Regards,
Melissa

 

[burtsbees]

Post a sh run and sh int for that interface for starters...
I would also look into NBAR...

Burt

 

[vipergg]

If the 1841 supports netflow turn that on , that will show you all the ip flows and who is going where . Someone was probably doing some download .

 

[MelSysAdmin]

Hi, Here is the sh int report
routeflem#sh int
FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is 001c.5843.22d0 (bia 001c.5843.22d0)
Description: RISA LAN$ETH-LAN$
Internet address is 10.1.0.100/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 461000 bits/sec, 157 packets/sec
5 minute output rate 861000 bits/sec, 180 packets/sec
5327658 packets input, 2161954139 bytes
Received 21226 broadcasts, 0 runts, 0 giants, 0 throttles
2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
5209346 packets output, 2606007849 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

What do you think? Im guess it had something to do with the 2 input errors and the broadcasts?

 

[burtsbees]

Now post a sh run and sh ver. Also, please indicate whether or not the problem is still happening. Whether it is or not, do this...

router>en
router#clear counters

That way, we can see real time errors if it still happening. The sh ver output will indicate the IOS version, which will tell us what features you can configure (like NBAR, class maps for QoS or CAR for rate limiting, etc.). It will also indicate router uptime, so these stats in the "sh int" output (broadcasts, 2 errors---NOT the suspect), etc. have all occured since the router was last rebooted...in the sh int output, you see
Last clearing of "show interface" counters never
The sh run output will indicate any possible misconfigurations.
This line...
reliability 255/255, txload 2/255, rxload 1/255
means that it is 100% reliable, and 2/255 transmit is less than 1% of what the interface is capable of, and receive is less than 1/2%, and also these lines
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 461000 bits/sec, 157 packets/sec
5 minute output rate 861000 bits/sec, 180 packets/sec
no dropped packets going in OR out, which is inconsistent with someone downloading something on a live network and congesting it---not on this interface, anyway---and 461Kbps in and 861Kbps out...definitely no downloads, or this is not the outgoing interface...is this the incoming/outgoing for the internet (what would normally be the WAN, but you may have metro e or something)??? With that slow BW, it looks like this is NOT the WAN interface, or if it is, then it is NOT busy...the network is as fast as the slowest link.
Does the site connect to this CoLo router via VPN, or how does it connect (WAN circuit type and on what interface)?

Burt

 

[burtsbees]

Also, what indication did you have that there was 100% throttling on this interface? Whatever the reason, if you don't have syslogging setup or a packet sniffer sniffing the server all the time, then there is not much to guess at---could be anything, but the fact whatever it was did NOT resume after rebooting then that could eliminate many things, like a regular download. Bit-torrent could have timed out while the router was coming back up, though...
One of the interface resets (or both, who knows) is caused from the interface queue dropping packets from a timeout exceeded, like when the router was rebooting. That reminds me---if the "sh int" output is from AFTER the router was rebooted, then a sh ver is only going to show a short amount of time. Perhaps you can tell us how long (about) the router had been running before you rebooted?
Future suggestion---look at "sh ver" to get the router uptime, write it down, then look at the interface counters (sh int), copy and paste, THEN reboot...

Burt

 

[tjbradford]

at the very least polling the interface to monitor its busy and idle points during the day , it could be something like clients updating at the same time AV or patching etc.

mrtg will give you a basic view of day week and month graphs but you could just use solarwinds trial for this i would think as a short term fix anyway.

 

[burtsbees]

NBAR and netflow are free, depending on her version of IOS, also.
You can do some port mirroring (RSPAN) in some switchports to do what TJ has suggested. I use Wireshark (the old Ethereal, though I still have Ethereal) myself.

Burt