Router Lockup

[lancepr]

Hi everyone,

We have a 3640 that has locked up twice this week.
The syslogs aren't very helpful.
The thing I have noticed is right before it locks up all of my server starts showing a huge increase in Network traffic (I monitor the switch ports using mrtg) usually my network traffic on the ports is between 400k and 1MB, just before the router locked up the ports were up to 80MB, I have Norton AV on all of my machines and ran a manual scan with no results. This is a strange problem indeed. Can you guys recommend any other software to try and monitor the network so I can see what is causing this tremendous network load?
All ideas appreciated.

Lance

 

[tschouten]

Someone could be hitting you with a DDoS, the router just locks up? What IOS do you have on it, and are you using CAR to limit a DoS attack?

 

[lancepr]

IOS Version 12.2(5a)

I am not sure what CAR is, how do I check.

 

[tschouten]

Commited Access Rates

Here is an example for protecting against nimda, you can protect against DDoS by setting it up to perform the same way for ICMP packets. If you need more information just ask, I'm willing to help.

http://www.cisco.com/en/US/products/sw/iosswrel/ps1835/products_tech_note09186a0080110d17.shtml

 

[tschouten]

basically it would look something like this, note you need to have IP CEF enabled to perform comitted access rates

Anyhow, I'm not saying this is the exact answer either, was just saying you could be getting attacked. It could be a faulty port/interface or a chattering nic somewhere, you just never know. You might try using a debug ip packets and have your syslog pick these up, would give you a lot of information but at least this way you can see what packets are coming accross. (I know debug degrades performance. But hey you already have crapy performance and know it will lock up so what is it going to hurt?) Another question to look at is does it happen at the same time everytime it happens or is it random?

class-map match-any icmp
match protocol icmp
!
!
policy-map ratelimiticmp
class arp
police 8000 1500 1500 conform-action transmit exceed-action drop violate-action drop

 

[queues]

I agree with tschouten that it is always a good idea to rate limit if you know what normal traffic looks like. I would recommend getting a sniffer on a port to see what is actually occuring. Also, if you have an idea what port you are dealing with, you can run the debug against an ACL to limit the number of hits to syslog. Another issue we have seen lately is the SQL Slammer worm. It causes the same type of symtoms. The recommended workaround prior to patching the servers is to block port 1434 on the router. Here's a good overview which also covers CAR (you can't beat CCO for information!).

http://www.cisco.com/warp/customer/cc/so/neso/sqso/worm_wp.htm

-Ken

 

[tschouten]

I totally forgot about that stinking slammer worm!

Ken is right, a sniffer would help out. And an acl would be great in limiting the amount of info you get off of a ip packet debug. My only concern would be to be very careful with what you block with the acl you may block something that could give you some information you really needed in order to find the problem.

Number one reason to always have a few months worth of LAN and WAN captures to use as a baseline on your network. Can't stress knowing you network enough, if you keep a baseline you can normally pinpoint problems in minutes.