CPU inconsistancy

[rn4it]

We have a router 2621 running IOS version 12.0(7)XK1. The last couple of day days we have noticed some latency on our 10Mb circuit. When we look at bandwidth utilization on the circuit it is the same as any other business day. However CPU utilization is at 99%, when I do a show process cpu on the router, it show 99% in the 3 time intervals, but when I add up the process it only adds up to approx. 55-60% CPU utilization.

Has anyone had any similiar cases and if some what was done to resolve it or find out what was using up the missing 30 odd %?

thanks

 

[wybnormal]

if you do a show proc cpu, it will tell you which process is sucking up the CPU power by showing which process is using how much and for how long.

CPU utilization for five seconds: 7%/2%; one minute: 8%; five minutes: 6%
PID Runtime (ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 2456 228 10771 4% 1% 0% 17 Virtual Exec
2 52658920 539402 97626 0% 4% 3% 0 Check heaps
3 0 2 0 0% 0% 0% 0 Timers
4 428 209 2047 0% 0% 0% 0 ARP Input

:::end of sample out:::

MikeS
Find me at

http://www.packetattack.com

"Take advantage of the enemy's unreadiness, make your way by unexpected routes, and attack unguarded spots."
Sun Tzu

 

[rn4it]

I realize that, the problem is that, when I do a sh proc cpu, it states 99% cpu utilization, however the remainder of the processes only add up to 60%, what is using up the other 30%.

On the 2 interfaces we have ip route-chache same interface, when then changed it to ip route-chach, and the CPU untilization dropped. here is a the output from the cpu process output interripter

You have chosen to display errors, warnings, general information, and helpful references.
Headings are displayed for all supported commands that you submitted.

SHOW PROCESS CPU Analysis

Back to top
==========================================================================
SHOW PROCESS CPU NOTIFICATIONS (if any)
==========================================================================

INFO: Total CPU Utilization is comprised of process and interrupt percentages.
Total CPU Utilization: 28%
Process Utilization: 0%
Interrupt Utilization: 28%
These values are found on the first line of the output:
CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%
Total CPU Utilization: x%
Process Utilization: (x - y)%
Interrupt Utilization: y%
Process Utilization is the difference between the Total and Interrupt (x and y).
The one and five minute utilizations are exponentially decayed averages (rather
than an arithmetic average), therefore recent values have more influence on the
calculated average.

INFO: Total CPU Utilization is 28% for the past 5 seconds.
Compare this CPU value to the baseline or average utilization. If a baseline is
not available, create one by collecting 'show process cpu' outputs during the
course of several days. If available, use the 'show process cpu history' command
(available in IOS 12.2 and higher).

any ideas?

 

[rsivanandan]

You were previously using "ip route-cache same interface"? Do you have secondary addresses configured on the same interface?

By just adding ip route-cache, you have enabled fast switching which reduces the cpu overhead caused by the process IP INPUT. can you put a sample "show proc cpu" and "show proc mem" along with it?

Cheers,
Rajesh

 

[rn4it]

Worked with Cisco TAC yesterday, they felt that it was traffic based. Routing change on ISP network, DOS attack, large file transfer.etc.

By adding ip route-cache and ip route mcache to the interfaces and IP cef to the global helped the switching process on the routers. It's still kind of odd as to what changed.

 

[rsivanandan]

I thought so! Most of the customers I have worked with have the high cpu utilization because of high traffic or because of high NAT translations. Cisco suggests like 3000 entries but in real customer situatons, they have like 10000-20000 NAT entries which have a high tcp clear out timer set!

Even though the problem is solved now, most likely you will experience the same soon. Watch out for it!, then the only solution is to split the traffic.

Cheers,
Rajesh