AIX high sendmail cpu useage

[5irc]

We have a box running AIX 5.3 and the sendmail cpu useage is always high

have no clue the reason


Fri Apr 8 11:26:54 2011 Interval: 2 Cswitch 12461 Readch 2761.1K
Syscall 90803 Writech 98966
CPU User% Kern% Wait% Idle% Physc Entc Reads 1821 Rawin 0
ALL 48.4 49.7 0.1 1.7 0.80 802.6 Writes 423 Ttyout 234
Forks 39 Igets 0
Network KBPS I-Pack O-Pack KB-In KB-Out Execs 38 Namei 43161
lo0 314.9 2710.8 2710.8 157.4 157.4 Runqueue 2.0 Dirblk 0
en0 7.8 31.1 38.1 2.9 4.9 Waitqueue 0.5

Disk Busy% KBPS TPS KB-Read KB-Writ PAGING MEMORY
hdisk0 25.0 1.2K 267.0 91.0 798.0 Faults 14928 Real,MB 4096
hdisk1 0.0 0.0 0.0 0.0 0.0 Steals 128 % Comp 90
PgspIn 34 % Noncomp 9
Name PID CPU% PgSp Owner PgspOut 1 % Client 9
sendmail 348394 26.6 5.4 root PageIn 67
tecad_lo 884950 25.5 2.1 root PageOut 1 PAGING SPACE
portmap 209020 12.8 1.0 root Sios 69 Size,MB 5056
gil 61470 3.8 0.9 root % Used 50
java 602352 1.8 259.4 webas NFS (calls/sec) % Free 50
java 663648 0.8 214.2 webas ServerV2 0
httpd 508132 0.5 4.4 ibmhttp ClientV2 0 Press:
syslogd 151636 0.3 0.3 root ServerV3 0 "h" for help
java 618722 0.2 151.9 root ClientV3 0 "q" to quit



vmstat 2 5

System configuration: lcpu=2 mem=4096MB ent=0.10

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
8 1 1239457 2874 0 15 6 70 124 0 256 86997 12343 48 50 2 0 0.73 725.5
3 0 1239555 2697 0 25 0 0 0 0 316 92968 12806 48 50 2 0 0.81 811.8
4 0 1239469 2721 0 12 0 0 0 0 246 86016 11884 48 50 2 0 0.75 750.9
6 0 1239562 2715 0 14 0 64 85 0 329 103077 14148 48 50 2 0 0.85 850.8
2 1 1239458 2788 0 34 0 64 76 0 311 96256 13328 48 50 2 0 0.81 814.2



root:/var/spool/mqueue]iostat 5 5

System configuration: lcpu=2 drives=2 ent=0.10 paths=8 vdisks=0

tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 13.4 48.3 49.8 1.8 0.2 0.8 777.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 31.0 1218.4 292.2 1480 4612

tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 71.6 47.8 50.3 1.8 0.1 0.8 792.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 27.0 1168.0 282.4 1112 4728

tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 71.6 48.0 50.2 1.7 0.1 0.8 820.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 22.8 1279.2 280.8 1572 4824

tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 71.6 48.6 49.4 1.8 0.1 0.8 817.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 27.6 1306.4 292.8 1456 5076

tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 71.6 48.5 49.5 1.7 0.3 0.7 720.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 44.6 1447.2 288.8 2004 5232

 

[blarneyme]

Problems:
1) You are out of CPU resources (48.4. + 49.7)
2) You have 4G of memory but still have some pages available (2788)
3) You are using 50% of your paging space
4) Your wait times are over 80% and you have processes in the run queue (but at least they aren't 4x#CPU's)
5) You have 90% computational memory usage
6) hdisk0 is 25% busy
7) hdisk1 is 100% idle

Things to look at:
1) Run filemon
2) Look at lru_file_repage, minperm%, maxperm%, minfree, maxfree, maxclient%
3) Find out if you're running sequential or random I/O
4) Look at minpgahead, maxpgahead, j2_minpgahead, j2_maxpgahead, numclust, maxrandwrt, j2_maxRandomWrite, j2_nRandomCluster, sync_release_ilock