High peaks in cpu usage -- AIX server

[mohmin]

From past 2 days, the AIX server we have here is runnning high cpu usage peaks, sometimes 100% too, but just in few moments the cpu usage is going down to 2% or so.

can u explain me whats happening and how to handle this.


Thanks,
Mohmin.

 

[hirschaj]

First question is, what changed 2 days ago? Second, can you provide some output ... vmstat 2 10, iostat 2 3, ps aux | head -10 and whatever else you feel relevant?


Jim Hirschauer

http://www.aixexpert.com

 

[kHz]

sometimes 100% too, but just in few moments the cpu usage is going down to 2% or so.

A 20 second vmstat or 6 second iostat isn't going to show anything. Unless you run it when the cpu is high.

The question is: what does the user use as a 'high' base number?

And as asked by Jim, did something change two days ago, or has this always been happening and was just noticed?

 

[mohmin]

Here are the ouputs asked by Jim and a little recent history about the server is that, last week (friday) there was very high paging going on and no unusual processes could be identified, so we had to reboot the server.
after we rebooted the paging/swapping went donwn considerably but since then we see many occassional peaks in cpu usage.

# vmstat 2 10
System Configuration: lcpu=4 mem=4096MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
6 1 1151962 2130 0 8 9 33 245 0 534 8023 726 7 8 83 2
0 0 1151966 2124 0 0 0 0 0 0 495 2108 541 1 1 98 0
0 0 1151966 2122 0 0 0 0 0 0 488 2165 586 1 2 97 0
0 0 1151966 2122 0 0 0 0 0 0 474 1679 457 0 2 98 0
0 0 1151966 2122 0 0 0 0 0 0 480 1926 459 0 2 98 0
0 0 1151966 2122 0 0 0 0 0 0 490 2103 553 0 2 98 0
2 0 1151967 2121 0 0 0 0 0 0 483 1953 439 1 2 98 0
0 0 1151967 2121 0 0 0 0 0 0 481 1986 546 0 2 98 0
0 1 1151971 2042 0 37 0 0 0 0 514 1833 537 1 2 93 4
0 0 1151971 2042 0 0 0 0 0 0 498 2172 518 0 2 98 0
# iostat 2 3

System configuration: lcpu=4 disk=10

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.9 82.1 6.5 8.2 83.3 2.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 6.7 89.0 12.7 11632043 11385760
hdisk1 4.0 41.9 7.2 3859059 6980016
hdisk4 0.4 34.3 1.1 6966067 1901760
hdisk5 0.0 0.0 0.0 779 0
hdisk2 0.1 7.8 0.2 717611 1288916
hdisk3 2.9 33.9 5.9 4280043 4478748
hdisk7 0.1 27.7 0.5 7056766 116728
hdisk6 0.0 1.5 0.0 133563 264824
hdisk9 0.0 6.3 0.0 1637426 0
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 446.3 0.2 0.4 99.1 0.2

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 0.5 2.0 0.5 4 0
hdisk1 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk7 0.0 0.0 0.0 0 0
hdisk6 0.0 0.0 0.0 0 0
hdisk9 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 428.5 0.1 1.5 97.5 0.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 10.5 72.0 18.0 0 144
hdisk1 0.0 0.0 0.0 0 0
hdisk4 2.0 44.0 3.0 0 88
hdisk5 0.0 0.0 0.0 0 0
hdisk2 1.0 8.0 2.0 0 16
hdisk3 0.5 2.0 0.5 0 4
hdisk7 0.0 0.0 0.0 0 0
hdisk6 0.0 0.0 0.0 0 0
hdisk9 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 0 0
# ps aux |head 10
10: A file or directory in the path name does not exist.
#
# ps -aux |head -10
UID PID TTY TIME CMD
210 143392 pts/2 0:00 tail
216 307274 pts/4 0:00 tail
0 372882 pts/3 0:00 head
210 581790 pts/2 0:00 ksh
0 823382 pts/3 0:00 ps
# ps aux |head -10
USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 8196 21.5 0.0 48 40 - A Oct 21 3701:43 wait
root 12294 21.4 0.0 48 44 - A Oct 21 3686:13 wait
root 16392 21.3 0.0 48 44 - A Oct 21 3668:24 wait
root 20490 21.3 0.0 48 40 - A Oct 21 3664:53 wait
root 876666 2.2 5.0 280492 196152 - A 19:34:49 115:28 /opt/WebSphere5.
root 864426 1.3 8.0 370368 334076 - A 15:19:51 7:14 /opt/WebSphere5.
root 114772 0.3 0.0 82280 4584 - A Oct 21 52:40 /usr/lib/errdemo
root 569416 0.2 6.0 292040 239124 - A Oct 21 25:57 /usr/WebSphere/D
seladm 577562 0.1 20.0 897292 819232 - A 19:39:01 7:22 /apps/Selectica/
#

 

[hirschaj]

mohmin, thanks for the output but I meant for you to gather it during one of your CPU spikes (sorry for not clarifying that earlier). Again, what actually changed on the server (new app, patching, etc...)?


Jim Hirschauer

http://www.aixexpert.com

 

[mohmin]

Actually i could not get a spike, even if it comes it goes back to low value very quickly.
last week what was happening is that the system was paging/swapping very much and we could not identify any major memory consuming processes so we went with a reboot of the server.

After the reboot i notice these frequent ups in cpu usage.

Moh.