cisco 2621 router mem full need help 3

[backupbob]

Hello,
I have a cisco 2621 router, and the other day I was makeing changes to the config. But the config did not work. So I copied the old config back to the router. Every thing worked fine till about 6:30am then the internet started slowing down. Once every 24 hours I have to reboot the router I never had to reboot the router before with this same config. These are my findings.

------------------ show process memory ------------------

Total: 15365292, Used: 14383276, Free: 982016
PID Allocated Freed Holding Getbufs Process
24 16007688 45948 11982856 30504 IP Input

------------------ show process cpu ------------------


CPU utilization for five seconds: 100%/3%; one minute: 98%; five minutes: 90%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min Process
24 48535948 619885 78299 67.46% 89.16% 85.81% IP Input

Thank you

 

[FredUG]

If you need a memory upgrade try

http://www.upgradegroup.com

they do a lot of cisco upgrade stuff at a good price with good service.

Hope that helps,

FredUG

 

[vipergg]

You need to find out what is driving the cpu to near 100 % , what process shows high when you do a show proc cpu . Make sure you have fast switching turned for the interfaces . Any of the interfaces running high ? I'm not sure it's a memory issue .

 

[baddos]

What routing protocols are you using?

 

[backupbob]

Thanks for the reply.
IP Input is running the cpu at around 97%
Ip Input is holding 11,982,856 of my 15,365,292 of memory

Fast switching is running it seems like it is my inside interface that is getting nailed.



internet#show ip interface
FastEthernet0/0 is up, line protocol is up
Internet address is 10.100.0.254/16
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is 1
Inbound access list is 1
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP Feature Fast switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is enabled, interface in domain inside
WCCP Redirect outbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled
Serial0/0 is up, line protocol is up
Internet address is 64.XX.XX.38/30
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is 1
Inbound access list is 1
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is disabled
IP Feature Fast switching turbo vector
IP multicast fast switching is disabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is enabled, interface in domain outside
WCCP Redirect outbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled
FastEthernet0/1 is up, line protocol is up
Internet address is 64.xx.xx.29/29
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is 1
Inbound access list is 101
Proxy ARP is enabled
Security level is default
Split horizon is enabled
ICMP redirects are always sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is disabled
IP Feature Fast switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
WCCP Redirect outbound is disabled
WCCP Redirect exclude is disabled
BGP Policy Mapping is disabled

 

[baddos]

This looks interesting....

FastEthernet0/0 is up, line protocol is up
Outgoing access list is 1
Inbound access list is 1
ICMP unreachables are always sent

It might be that you are getting a lot of denies on your ACL, and your router is sending a lot of unreachable messages. Try turning this off and seeing if you see improvemnet. The command is "no ip unreachables" under the interface.

Sample:
configure terminal
interface fastethernet0/0
no ip unreachables
end
copy running-config startup-config

 

[backupbob]

Thanks, baddos I will give that a shot.

 

[backupbob]

I put in the "no ip unreachables" it did seem to get me 2 more hours before I had to reboot. I did find some machines with the love.san virus on the network. I am in the process of checking them. So far I think that virus scan has got most of them. Here is my current config. Thank you for the help.

Building configuration...

Current configuration:
!
version 12.0
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname internet
!
enable password #############
!
!
!
!
!
ip subnet-zero
no ip domain-lookup
!
!
!
!
interface FastEthernet0/0
ip address 10.100.0.254 255.255.0.0
ip access-group 1 in
ip access-group 1 out
no ip unreachables
no ip directed-broadcast
ip nat inside
speed auto
full-duplex
!
interface Serial0/0
ip address 64.xxx.xxx.38 255.255.255.xxx
ip access-group 1 in
ip access-group 1 out
no ip directed-broadcast
ip nat outside
no ip mroute-cache
no fair-queue
!
interface FastEthernet0/1
ip address 64.xxx.xxx.29 255.255.255.xxx
ip access-group 101 in
ip access-group 1 out
no ip directed-broadcast
speed auto
full-duplex
!
ip default-gateway 64.xxx.xxx.37
ip nat inside source list 7 interface FastEthernet0/1 overload
ip classless
ip route 0.0.0.0 0.0.0.0 64.xxx.xxx.37
ip route 8x.0.0.0 255.0.0.0 10.100.0.111
ip route 172.xxx.1.0 255.255.255.0 10.100.23.150
ip route 192.168.1.0 255.255.255.0 10.100.0.111
ip route 192.168.10.0 255.255.255.0 10.100.0.111
ip route 192.168.20.0 255.255.255.0 10.100.0.111
ip route 192.168.30.0 255.255.255.0 10.100.0.111
ip route 192.168.40.0 255.255.255.0 10.100.10.2
ip route 2xx.xxx.xxx.0 255.255.255.0 10.100.0.250
ip route 2xx.xxx.xxx.100 255.255.255.255 10.100.0.250
no ip http server
!
access-list 1 permit any
access-list 2 permit 64.xxx.xxx.28
access-list 2 permit 10.100.0.0 0.0.255.255
access-list 7 permit 10.100.0.0 0.0.255.255
access-list 7 permit 192.168.40.0 0.0.0.255
access-list 101 permit tcp any any eq www
access-list 101 permit tcp any any eq ftp
access-list 101 permit tcp any any eq ftp-data
access-list 101 permit tcp any any eq domain
access-list 101 permit tcp any any eq smtp
access-list 101 permit udp any any eq domain
access-list 101 permit udp any any eq tftp
access-list 101 permit tcp 10.100.0.0 0.0.255.255 any eq telnet
access-list 101 permit tcp host 64.xxx.xxx.28 host 64.xxx.xxx.29 eq telnet
access-list 101 permit udp host 64.xxx.xxx.28 host 64.xxx.xxx.29 eq tftp
access-list 101 permit icmp any any
access-list 101 permit tcp any any eq pop3
access-list 101 permit udp any any eq isakmp
access-list 101 permit esp any any
access-list 101 permit ahp any any
snmp-server engineID local 0000000902000003E32749A0
snmp-server community mypublic RO
!
line con 0
transport input none
line aux 0
line vty 0 4
access-class 2 in

login
!
end

 

[baddos]

Try this and see if it helps you out...

configure terminal
ip cef
no access-list 1
interface Serial0/0
no ip unreachables
interface FastEthernet0/1
no ip unreachables
end
copy running-config startup-config

 

[backupbob]

Thanks for your help baddos, I tried those changes but I am still getting IP Input useing all the cpu. It has stopped useing all of the memory.

 

[CiscoGod]

Hello,

There are two issues that I notice here:

1) You have "ip access-group 1 in and out" on almost all of your interfaces. This filter permits all data packets. Is there a reason why you put this on? This is causing your router to evaluate ALL data packets that traverse this router against this access-list. This may or may not be the root cause of your problem, but it will increase the CPU usage on a router significantly as more and more data is pumped through the router. An interface by default allows everything through it so you do not need to specifically apply an access-list to do this.

2) You have mentioned that several of the PCs have been infected by a virus outbreak. I would first verify that all PCs and servers on your network are checked with the latest virus definition pattern from your favorite AV system. After you have done this and made certain that all machines have been checked, I would put a sniffer on the line where the router is connected to and look at the data that router is seeing and try to pick out the culprit that is chattering.

Also if you can can you post the results of a show interface
of all your routers interfaces.

Hope this helps.

CiscoGod

 

[backupbob]

Thanks for the reply CiscoGod. I inherited this router just 3 months ago. I am not sure why they put that access-group on there but I will take it out. I do not have a sniffer is there any way I can see the traffic on the router by ip address?

internet#show interface
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0005.5e63.33a0 (bia 0005.5e63.33a0)
Internet address is 10.100.0.254/16
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 30/75, 2496 drops
5 minute input rate 48000 bits/sec, 74 packets/sec
5 minute output rate 86000 bits/sec, 17 packets/sec
1737897 packets input, 137587278 bytes
Received 232758 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
364763 packets output, 317495171 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is up, line protocol is up
Hardware is PQUICC with Fractional T1 CSU/DSU
Internet address is 64.xxx.xxx.38/30
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 11/255, rxload 18/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 3/75, 0 drops
5 minute input rate 114000 bits/sec, 56 packets/sec
5 minute output rate 67000 bits/sec, 83 packets/sec
1323361 packets input, 415927714 bytes, 0 no buffer
Received 4129 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1967209 packets output, 156529981 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

FastEthernet0/1 is up, line protocol is up
Hardware is AmdFE, address is 0005.5e63.33a1 (bia 0005.5e63.33a1)
Internet address is 64.xxx.xxx.29/29
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:05, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 40000 bits/sec, 18 packets/sec
5 minute output rate 48000 bits/sec, 41 packets/sec
482325 packets input, 65182329 bytes
Received 1015 broadcasts, 0 runts, 0 giants, 0 throttles
1356 input errors, 1356 CRC, 682 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
983232 packets output, 117139927 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Thank you.

 

[baddos]

Humm... I see another problem with your router that probably isn't causing your high CPU, but isn't helping it either.

Look at the configuration settings for interface FastEthernet0/1. Is it plugged into a switch, hub, or crossover? It's setup to run at 10Mb Half Duplex right now.

 

[backupbob]

Yeah, believe it or not it is plugged into a hub I will be changing that, hopefully after I get this issue taken care of.

 

[baddos]

At the time that you got these interface statistics, was the CPU high then?

 

[backupbob]

Yes, I think it was running about 97% for Ip Input.

 

[serviceguy]

I have seen this issue quite often lately tied to all the viruses circulating. They drive the router nuts. Try adding this access-list inbound on your Ethernet and see if you're taking lots of hits on it. This has helped me narrow down the problem.

!
access-list 175 deny icmp any any echo
access-list 175 deny icmp any any echo-reply
access-list 175 permit ip any any
!
L3#sh access-lists
Extended IP access list 175
deny icmp any any echo (92255 matches)
deny icmp any any echo-reply
permit ip any any (7738 matches)

If you are, there may still be a virus lurking on your network...

 

[CiscoGod]

I think the root cause of this issue is due to a viruses running rampant within your network. I found this clever method to avoid NACHI type viruses and to diagnose if this is in fact what is going on.

Credit:

http://www.farrokhi.net/blog/archives/2003_08.html

Paste from above archive:

Say goodbye to Nachi
Posted at 07:21 PM | Permalink | Comments (2) | TrackBack (0)
Worms are hot topic of day. Blaster and Nachi are making trouble for ISPs as well as end users worldwide.
There are different ways to combat these worms. If you are a network administrator utilizing Cisco gear in your network, you can reduce the effect of these worms using some simple tricks. I have already posted an article on blocking Blaster worm in a Cisco router.
But blocking Nachi is a little bit tricky since it uses ICMP echo/reply to map your network and propagate its code. This will cause a heavy ICMP storm in your network (that you may have already noticed). The most simple way is blocking all ICMP traffic which is not a good solution and harms your customers (They won't be able to do PING measurement in this case).

Here is what I did to protect against Nachi (in a Cisco router):

Setup your NULL0 interface like this:
!
interface Null0
no ip unreachables
!Then make an access-list that matches ICMP echo/reply packets:
!
ip access-list extended nachi-list
permit icmp any any echo
permit icmp any any echo-reply
!Now the trick:
!
route-map nachi permit 10
match ip address nachi-list
match length 92 92
set interface Null0

!Fortunately, Nachi uses fixed size ICMP packets (92 bytes, including IP header) as reachability probe. Above route-map will forward all ICMP packets with size of 92 bytes to Null0 interface. Null0 will not return any unreachable code and just drops the packet.

You should put this route-map on your network interface, like this (necessary parts listed only):
!
interface FastEthernet0/0
description Connected to Local Network
ip route-cache policy
ip policy route-map nachi
!That "ip route-cache policy" is very important because it asks the router to cache all policy-route information in order to reduce processor load. (CEF won't be useful here).

This is the result after 5 minutes:

router#sh route-map nachi
route-map nachi, permit, sequence 10
Match clauses:
ip address (access-lists): nachi-list
length 92 92
Set clauses:
interface Null0
Policy routing matches: 190909 packets, 20236354 bytes

The key is to do a SH ROUTE-MAP nachi after 5 minutes or so to verify if the majority of your traffic is ICMP ECHOS which WILL cause network performance degredation rather quickly. Give it shot and let me know if this helps.

CiscoGod

 

[backupbob]

Thank you to everyone that helped me. I ended up enableing netflow on my router. I then used the command (sh ip cache flow | include 0087) this showed me the ip address of the 8 pc's that were infected with blaster on the network. As soon as I cleaned the virus off of them the router went down to 4% cpu usage. I can now go home and not have to worry about wakeing up at 2:00am to reboot the router, so people can have the internet.

Thanks.