More 2600 problems - 99% CPU usage - IP Input

[Guest_imported]

Hello-

I wrote recently because my cisco 2600 has been crashing frequently and i don't know what to do. well, today at three different times the router seemed to die. I can get into it through a terminal, but it responds EXTREMELY slowly to whatever i type.

"show proc CPU" shows 99% usage, almost all of it being taken up by the "IP Input" process. I also have several errors in my syslog similar to:
%SYS-3-CPUHOG: Task ran for 2148 msec (20/13), Process = IP Input, PC = 3199482 -Traceback= 314B5E6 319948A

I read that that error is the cisco watchdog timer basically telling me that the "IP Input" process has been hanging and using too much CPU.

What could cause this? How can I fix it? Am I being DOS'ed?

Any help would be appreciated. Thank you.

 

[wybnormal]

This can be an ARP storm, broadcast storm, DOS attack, screwed up route tables with EIGRP or OSPF and the list goes on.

Do a show ip traffic and see which protocol is the heavy user.. a sniffer would be be very useful for this..

MikeS
Find me at

http://www.packetattack.com

"Diplomacy; the art of saying 'nice doggie' till you can find a rock" Wynn Catlin

 

[Guest_imported]

Well, I discovered the problem and I want to share my experience.

first, sho proc cpu showed 99% usage. i reloaded the router and within an hour it was back to 99% (almost all being taken up by "IP Input".)

i then disabled all logging destinations except logging buffer and cleared logging buffer (great instructions here:

http://www.cisco.com/warp/public/63/highcpu.html#sampleip

)

then by running:
#debug ip packet detail
#undebug all
#show logging

i was able to see the traffic flowing through the router. i had a ton of the following, even though debugging was only on for a few seconds:
01:03:39: TCP src=4815, dst=80, seq=2285476180, ack=0, win=16384 SYN
01:03:39: IP: s=<ip address hidden> (FastEthernet0/0), d=10.1.38.244 (Serial0/0), g=10.1.38.244, len 48, forward

all with same source address, but different destination- a lot of which were unroutable ip addresses. this told me that the traffic was coming from inside our network (in FastEthernet, out Serial).

here i wasted some time trying to set up an ACL to block the source address while i tried to figure out what was going on, but the source address shown on the router was the public NAT address, and not the private internal address of the machine. i needed the internal address of the machine.

time for the sniffer. i set up our switch to mirror all of our router's traffic to my machine, then i started up the sniffer. within 3 seconds it said &quot;300 packets captured, 500, 1000...). almost every packet was from one internal ip address.

so, i took that machine off of the network and the router cpu went down to 4% instantly. now i need to figure out what's wrong with that box. seems like a trojan of some sort- or perhaps code red since all traffic has destination port 80 (doesn't code red search for that?), but the box was never directly accessible by the outside world, so i'm not sure how it could have been infected. it was never patched because no one was aware it was running IIS. it is not used for web serving or anything. it may not be code red either, but that's where i am now.

this was a great learning experience for me. i hope it helps others also.

 

[wybnormal]

Way cool that you found it

port is code red's favorite port to use.. my firewall has been dumping alot of queries on that port. Cable company's filtering is for naught.

Someone may have uploaded files with the virus? Stuff like this while frustrating is alot of fun to chase down ...

We have found a few &quot;webservers&quot; .. nonsense like installing frontpage which in turn installs MS personal webserver.. nasty trick of MS

MikeS Find me at

http://www.packetattack.com

&quot;Diplomacy; the art of saying 'nice doggie' till you can find a rock&quot; Wynn Catlin