High CPU Usage

[chriswww]

I don't know much about cisco routers, but perhaps someone will be able to assist here anyway. Recently we started seeing some high CPU usage on two of our main cisco routers:

"show proc cpu sorted" reports this:

Router2>show proc cpu sorted
CPU utilization for five seconds: 94%/0%; one minute: 90%; five minutes: 89%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 157    49362457 216080233        228  0.63%  0.35%  0.32%   0 IP Input         
 144        1159       216       5365  0.31%  0.38%  0.26%   1 SSH Process      
 152     3014241  11105817        271  0.15%  0.04%  0.00%   0 CDP Protocol     
 224    30035520  26494076       1133  0.15%  0.22%  0.09%   0 SNMP ENGINE      
  71      640825 324334775          1  0.15%  0.03%  0.00%   0 hpm main process 
 175      563285  23831489         23  0.15%  0.01%  0.00%   0 TCP Timer        
 160    18560059 189411715         97  0.15%  0.09%  0.08%   0 Spanning Tree    
 115     3026173   7815728        387  0.15%  0.10%  0.12%   0 HQM Stack Proces 
 182      761726   5333523        142  0.15%  0.01%  0.00%   0 IGMPSN           
 108       82835   8936220          9  0.15%  0.01%  0.00%   0 HIPV6 bkgrd proc 
  10           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager   
   9           0         2          0  0.00%  0.00%  0.00%   0 AAA high-capacit 
  11          17        11       1545  0.00%  0.00%  0.00%   0 Entity MIB API   
  12           0         1          0  0.00%  0.00%  0.00%   0 IFS Agent Manage 
  15       76101  38708978          1  0.00%  0.00%  0.00%   0 IPC Periodic Tim 
   8           0         1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT 
  17        6980   2610261          2  0.00%  0.00%  0.00%   0 IPC Seat Manager 
  13        5009    653036          7  0.00%  0.00%  0.00%   0 IPC Dynamic Cach 
  14           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager 
  20       78897  38708981          2  0.00%  0.00%  0.00%   0 Dynamic ARP Insp 
  16       57736  38708983          1  0.00%  0.00%  0.00%   0 IPC Deferred Por 
  22           0         2          0  0.00%  0.00%  0.00%   0 XML Proxy Client 
  23           0         1          0  0.00%  0.00%  0.00%   0 Critical Bkgnd   
  24      407285  18024234         22  0.00%  0.01%  0.00%   0 Net Background  
...

A bit of digging discovered that possibly these two routers are trying to communicate:

742644: Jun 10 13:33:06.360 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active
742645: Jun 10 13:33:37.356 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active
742646: Jun 10 13:34:07.957 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active
742647: Jun 10 13:34:38.449 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.195.2, group 5, remote state Active
742648: Jun 10 13:35:08.556 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.195.2, group 5, remote state Active
742649: Jun 10 13:35:38.612 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.195.2, group 5, remote state Active
742650: Jun 10 13:36:10.354 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active
742651: Jun 10 13:36:40.955 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active
742652: Jun 10 13:37:11.380 UTC: %HSRP-4-BADAUTH: Bad authentication from 192.168.91.2, group 5, remote state Active

(where 192.168.91.2 is the ip of the other router). Is it possible that these failed logins could be causing the very high CPU usage? If so, why isn't that CPU usage reported when I do a "show proc cpu sorted"?

Version information:

Cisco IOS Software, C3560 Software (C3560-ADVIPSERVICESK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

ROM: Bootstrap program is C3560 boot loader
BOOTLDR: C3560 Boot Loader (C3560-HBOOT-M) Version 12.2(25r)SEE4, RELEASE SOFTWARE (fc1)


cisco WS-C3560G-24TS (PowerPC405) processor (revision D0) with 118784K/12280K bytes of memory.

Last reset from power-on
4 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces

 

[chriswww]

It doesn't look like the HSRP errors had anything to do with it, actually. Some google searching revealed the cause, and those have since been eliminated (changing standby authentication).

...So the question remains - why is there such high CPU usage? How do I go about finding out what's causing this high CPU usage?

 

[burtsbees]

Post the complete "sh proc cpu"

I would guess from interrupts, but who knows...do you have a CCO login?

/

 

[vipergg]

Might want to read thru this . As Burt said seeing that there isn't a specific process driving it , its probably interrupt driven. This will give you some guidance.

http://www.cisco.com/en/US/products...h_note09186a00801c2af0.shtml#traffic_overload

 

[Jsteve]

I have seen the same issues, but on 3750's. I had to change the SDM profile to routing. Once I did and rebooted the CPU dropped to 4%

the command I believe is

sdm prefer routing

http://www.cisco.com/en/US/docs/swi...ase/12.2_46_se/configuration/guide/swsdm.html

 

[chriswww]

Jsteve - after some more investigating into our situation, it seems very likely that switching the sdm preference to routing will resolve the problem. Thank you for your assistance!!

 

[burtsbees]

Also, you may want to check region memory allocation...

/