Cause for high CPU usage? 2

[hinesjrh]

I have a 2620 router at a remote site running IOS 12.2(5d) and all of a sudden everyday this week our monitoring tools are showing the CPU very high (often at or near 100%). I found the following in the log and wonder if anyone can help be decrypt this or suggest a root cause.

1d16h: %SYS-3-CPUHOG: Task ran for 2368 msec (7046/653), process = Virtual Exec,
PC = 803A6C08.
-Traceback= 803A6C0C 803075F4 8030809C 80307940 80306DA0 80307550 8030809C 80307
940 8028B5D8 8028BABC 8028BEE4 8028BFD4 8028C104 8032594C 8033427C 803BA750

 

[burtsbees]

Post a sh proc cpu

/

 

[hinesjrh]

SOUTHSIDE#sh proc cpu
CPU utilization for five seconds: 95%/91%; one minute: 94%; five minutes: 94%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 54332 7068 7687 0.00% 0.01% 0.00% 0 Load Meter
2 4 11 363 0.00% 0.00% 0.00% 0 PPP auth
3 628476 24347 25813 0.00% 0.07% 0.09% 0 Check heaps
4 4 1 4000 0.00% 0.00% 0.00% 0 Chunk Manager
5 408 10 40800 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 94 0 0.00% 0.00% 0.00% 0 Serial Backgroun
8 19240 6924 2778 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC
9 3420 1187 2881 0.00% 0.00% 0.00% 0 Environmental mo
10 437616 15837 27632 0.40% 0.15% 0.09% 0 ARP Input
11 0 5 0 0.00% 0.00% 0.00% 0 DDR Timers
12 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
13 4 3 1333 0.00% 0.00% 0.00% 0 Entity MIB API
14 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
15 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
16 79244 5848 13550 0.08% 0.01% 0.00% 0 Net Background
17 28 42 666 0.00% 0.00% 0.00% 0 Logger
18 270304 32838 8231 0.16% 0.07% 0.06% 0 TTY Background
19 563984 33088 17044 0.00% 0.12% 0.11% 0 Per-Second Jobs
20 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou
21 2516 102 24666 0.57% 0.31% 0.07% 66 Virtual Exec
22 86480 20662 4185 0.08% 0.00% 0.00% 0 Net Input
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
23 90488 7068 12802 0.00% 0.00% 0.00% 0 Compute load avg
24 299820 726 412975 0.00% 0.03% 0.05% 0 Per-minute Jobs
25 4 7 571 0.00% 0.00% 0.00% 0 Service-module a
26 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
27 1889568 90064 20980 0.65% 0.49% 0.45% 0 IP Input
28 32792 3592 9129 0.00% 0.00% 0.00% 0 CDP Protocol
29 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
30 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
31 4 9 444 0.00% 0.00% 0.00% 0 PPP IP Add Route
32 50864 795 63979 0.00% 0.00% 0.00% 0 IP Background
33 52528 631 83245 0.00% 0.00% 0.00% 0 Adj Manager
34 188 35 5371 0.00% 0.03% 0.00% 0 TCP Timer
35 8 5 1600 0.00% 0.00% 0.00% 0 TCP Protocols
36 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
37 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
38 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
39 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
40 1851416 56311 32878 0.21% 0.40% 0.43% 0 DHCPD Receive
41 1428 590 2420 0.00% 0.00% 0.00% 0 IP Cache Ager
42 0 1 0 0.00% 0.00% 0.00% 0 COPS
43 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
44 4 2 2000 0.00% 0.00% 0.00% 0 X.25 Background
46 0 1 0 0.00% 0.00% 0.00% 0 Router Autoconf
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
47 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
48 3081864 44914 68617 0.60% 0.55% 0.67% 0 CEF process
49 0 2 0 0.00% 0.00% 0.00% 0 CCP manager
50 32 217 147 0.00% 0.00% 0.00% 0 PPP manager
51 152020 33835 4492 0.00% 0.02% 0.00% 0 Multilink PPP
52 0 2 0 0.00% 0.00% 0.00% 0 Multilink PPP ou
53 44 12 3666 0.00% 0.00% 0.00% 0 Multilink event
54 4696 1231 3814 0.00% 0.00% 0.00% 0 CEF Scanner
55 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers
56 720484 33300 21636 0.00% 0.10% 0.11% 0 IP SNMP
57 873612 18334 47649 0.00% 0.10% 0.14% 0 PDU DISPATCHER
58 3979380 19637 202647 0.83% 0.56% 0.62% 0 SNMP ENGINE
59 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
60 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps
61 1940 298 6510 0.00% 0.00% 0.00% 0 DHCPD Timer
62 22256 10032 2218 0.00% 0.00% 0.00% 0 DHCPD Database

 

[burtsbees]

Also please post sh logg and sh run...there is a very high percentage of interrupts (91%), which can be caused by less-than-optimum switching methods, which can possibly be fixed by ip cef and/or ip route-cache/ip route-cache flow (if ip accounting has been turned on). Also, it can be caused by voice ports configured on it. Let's see the log and look for a few things, as well as the config. Is this interrupting traffic flow, i.e. slow network traffic going through the router? Is there a lot of traffic? You can run Wireshark by SPANning the switchport attached to the router as the input and running Wireshark on a node attached to the output port. If you do this, run Wireshark with at least a 100MB buffer (if the pc has enough memory), and uncheck "Update list of packets in real time", specify an output file (with the .cap extension), check "use multiple files" and "stop capture" after 24 hours. Before doing this, clear all counters and reboot the switch to flush all buffers out...

router#clear counters

answer "yes" to "clear counters on all interfaces?"

If the file is larger than 1.5GB in the end, and you don't have the boot.ini file modified to allow 3GB of physical RAM to be used for processes (which is the max for anything less than Windoze Server 2003 Enterprise), then loading the .cap file after the capture will cause your machine to run out of memory. This is with even 4GB of memory. I have Server 2008 Enterprise 64 bit installed on a Proliant DL585 G2 with two dual-core 64 bit procs and 24GB of RAM, and a 16GB .cap crashed it after about 2 minutes!

/

 

[burtsbees]

Oh yeah...SPAN...you need a switch attached to the router...HP ProCurve? Cisco?

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml

/

 

[Jsteve]

Burt is right 91% of the CPU usage is traffic being punted to the CPU

Normally I have seen all of my traceback errors related to memory leaks triggered by an IOS bug.

If it's a memory leak,a switch reload will fix the issues.

If a reload does not fix the issue, and if this is a 6500, you can span the traffic that is being punted to the CPU to see what it is. Here is a good doc that can help.

http://tecun.cimex.com.cu/tecun/sof...o/Catalyst_SWs/6500/generales/6k_high_cpu.pdf

 

[hinesjrh]

We seem to have solved the issue by backing off how often one of our network monitoring tools was capturing data (was every 15 sec.). CPU on router and WAN response time seems mormal now (since Friday). Thanks for your suggestions.