Performance Problem - High CPU utilization

[gutolm]

Hello everybody.

I have a problem with my AIX 5.3. Recently my unix shows a high cpu utilization with sar or topas.

I need to find what I have to do to solve this problem, in fact, I don't know what is my problem.

I had the same problem with another AIX 5.3 running the same application. The %sys and %user normally shows 45 (user) and 21 (sys) average. I don't know what happened, but the performance decreased and my %usr shows 21 and %sys 50. With a reboot the server came back with normal load and good performance. I don't want to reboot my other server without knowing my problem.

Please, help me to find what process is consuming too much %SYS.

If it will be necessary, I can atach the ps, and sar.

 

[cts123]

Can you findot the processes which are eating up your CPU?

Try this ( find top 10 CPU-consuming processes) :-

# ps aux | head -1; ps aux | sort -rn +2 | head -10

If you see any application is eating up your CPU, you can take action accordingly.

Regards,

-Sam

 

[gutolm]

The ps aux command returns:

USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 17214 1.8 0.0 40 40 - A Mar 25 24903:26 wait
root 16678 1.8 0.0 40 40 - A Mar 25 23886:51 wait
root 17750 1.8 0.0 40 40 - A Mar 25 23820:12 wait
root 17482 1.7 0.0 40 40 - A Mar 25 23790:38 wait
root 16946 1.7 0.0 40 40 - A Mar 25 23718:42 wait
root 1090 1.7 0.0 40 40 - A Mar 25 23284:53 wait
root 822 1.7 0.0 40 40 - A Mar 25 23255:50 wait
root 1358 1.7 0.0 40 40 - A Mar 25 23200:53 wait
root 554 1.7 0.0 40 40 - A Mar 25 22974:08 wait
root 286 1.7 0.0 40 40 - A Mar 25 22921:20 wait
root 8196 1.7 0.0 40 40 - A Mar 25 22914:21 wait
root 53274 1.7 0.0 40 40 - A Mar 25 22904:41 wait
root 21580 1.6 0.0 40 40 - A Mar 25 21561:14 wait
root 21044 1.6 0.0 40 40 - A Mar 25 21555:04 wait
root 21848 1.6 0.0 40 40 - A Mar 25 21489:16 wait
root 13116 1.6 0.0 40 40 - A Mar 25 21284:39 wait
root 12848 1.6 0.0 40 40 - A Mar 25 21281:00 wait
root 57372 1.6 0.0 40 40 - A Mar 25 21228:35 wait
root 13384 1.6 0.0 40 40 - A Mar 25 21137:35 wait
root 13652 1.5 0.0 40 40 - A Mar 25 21072:47 wait
root 12580 1.5 0.0 40 40 - A Mar 25 20993:58 wait
root 20776 1.5 0.0 40 40 - A Mar 25 20962:46 wait
root 49176 1.5 0.0 40 40 - A Mar 25 20770:34 wait
root 21312 1.5 0.0 40 40 - A Mar 25 20577:27 wait
informix 221444 0.4 0.0 6140 3872 - A Apr 29 644:57 oninit
informix 258532 0.4 0.0 6100 3792 - A Apr 29 570:42 oninit
informix 73828 0.4 0.0 6088 3792 - A Apr 29 546:55 oninit
informix 246212 0.4 0.0 6092 3792 - A Apr 29 544:14 oninit
informix 90824 0.4 0.0 6068 3792 - A Apr 29 543:48 oninit
informix 111094 0.4 0.0 6064 3792 - A Apr 29 539:06 oninit
informix 106702 0.4 0.0 6076 3792 - A Apr 29 536:48 oninit
informix 205806 0.4 0.0 6072 3792 - A Apr 29 531:44 oninit
informix 172280 0.4 0.0 6084 3792 - A Apr 29 525:50 oninit
informix 254524 0.4 0.0 6268 4012 - A Apr 29 525:20 oninit
informix 152292 0.4 0.0 6320 4008 - A Apr 29 524:36 oninit
informix 180398 0.3 0.0 6080 3792 - A Apr 29 516:48 oninit
informix 151672 0.3 0.0 6104 3792 - A Apr 29 514:07 oninit
informix 262358 0.3 0.0 6096 3792 - A Apr 29 505:23 oninit
informix 127462 0.3 0.0 6256 3996 - A Apr 29 476:59 oninit
informix 106946 0.3 0.0 6192 3932 - A Apr 29 423:15 oninit
informix 230008 0.3 0.0 6252 3992 - A Apr 29 405:58 oninit
informix 115442 0.3 0.0 6176 3924 - A Apr 29 404:20 oninit
informix 98560 0.2 0.0 6256 3996 - A Apr 29 354:03 oninit
informix 102718 0.2 0.0 6248 3988 - A Apr 29 321:30 oninit
informix 81946 0.1 0.0 10496 8232 - A Apr 29 178:57 oninit
informix 205358 0.1 0.0 6048 3788 - A Apr 29 161:12 oninit
informix 83226 0.1 0.0 6040 3784 - A Apr 29 118:43 oninit
root 25180 0.0 0.0 116 116 - A Mar 25 274:08 gil
root 16392 0.0 0.0 124 124 - A Mar 25 230:55 lrud
root 24588 0.0 0.0 60 60 - A Mar 25 149:19 xmfreed
P0605437 29340 0.0 0.0 764 408 pts/0 A Apr 27 18:04 -sh
P0605437 246308 0.0 0.0 764 408 pts/1 A Apr 27 17:56 -sh
informix 140272 0.0 0.0 9560 7308 - A Apr 29 10:01 oninit
oracle 54460 0.0 0.0 39336 31172 - A Apr 04 59:54 /oracle9i/app/or
informix 196874 0.0 0.0 704 740 - A 00:01:00 1:02 sh -c /usr/shell
root 46022 0.0 0.0 520 528 - A Mar 25 71:01 /usr/sbin/syncd
informix 230294 0.0 0.0 700 736 - A 07:00:00 0:21 sh -c /usr/shell
informix 188576 0.0 0.0 6048 3784 - A Apr 29 6:04 oninit
root 45078 0.0 0.0 52 52 - A Mar 25 35:34 xmgc
root 1 0.0 0.0 684 672 - A Mar 25 29:59 /etc/init
informix 157064 0.0 0.0 772 404 pts/8 A Apr 16 7:31 -ksh
informix 147764 0.0 0.0 764 408 pts/4 A Mar 25 17:04 -ksh
informix 172492 0.0 0.0 752 408 pts/9 A Apr 16 7:25 -ksh
informix 127506 0.0 0.0 756 404 pts/3 A Mar 25 16:56 -ksh
informix 161084 0.0 0.0 760 416 pts/7 A May 02 0:28 -ksh
informix 172572 0.0 0.0 768 412 pts/6 A May 02 0:28 -ksh
informix 160134 0.0 0.0 6044 3780 - A Apr 29 0:58 oninit
informix 250346 0.0 0.0 6032 3780 - A Apr 29 0:53 oninit
root 4384 0.0 0.0 48 48 - A Mar 25 7:20 sched
root 49672 0.0 0.0 8848 8612 - A Mar 25 6:26 /usr/sbin/snmpmi
root 0 0.0 0.0 52 52 - A Mar 25 5:59 swapper
root 33906 0.0 0.0 304 304 - A Mar 25 5:10 nfsd
root 58836 0.0 0.0 1124 1052 - A Mar 25 5:07 /usr/sbin/snmpd
root 4652 0.0 0.0 48 48 - A Mar 25 4:27 sched
root 38100 0.0 0.0 408 400 - A Mar 25 4:15 /usr/sbin/inetd
root 5188 0.0 0.0 48 48 - A Mar 25 4:11 sched
root 148082 0.0 0.0 60 60 - A Mar 26 4:00 aioserver
root 156080 0.0 0.0 60 60 - A Mar 26 3:59 aioserver
root 4920 0.0 0.0 48 48 - A Mar 25 4:01 sched
root 41952 0.0 0.0 1512 1512 - A Mar 25 3:51 /usr/sbin/cron
root 65630 0.0 0.0 60 60 - A Mar 26 3:43 aioserver
root 37680 0.0 0.0 60 60 - A Mar 25 3:37 aioserver
root 5456 0.0 0.0 48 48 - A Mar 25 3:23 sched
informix 116214 0.0 0.0 6012 3756 - A Apr 29 0:21 oninit
root 132226 0.0 0.0 60 60 - A Mar 26 2:56 aioserver
root 98990 0.0 0.0 1716 1424 - A Apr 13 1:31 /usr/DynamicLink
root 242270 0.0 0.0 1060 1040 - A Apr 22 0:49 /opt/flex/fval
informix 164186 0.0 0.0 6036 3776 - A Apr 29 0:18 oninit
informix 200892 0.0 0.0 6040 3776 - A Apr 29 0:17 oninit
root 185336 0.0 0.0 60 60 - A Mar 26 2:26 aioserver
root 94686 0.0 0.0 192 192 - A Mar 25 2:28 rpc.lockd
informix 75124 0.0 0.0 6008 3756 - A Apr 29 0:15 oninit
root 37276 0.0 0.0 3444 3340 - A Mar 25 1:05 sendmail: accept
informix 131446 0.0 0.0 6024 3772 - A Apr 29 0:03 oninit
informix 238240 0.0 0.0 6024 3772 - A Apr 29 0:03 oninit
root 40980 0.0 0.0 400 400 - A Mar 25 0:21 pilegc
root 45460 0.0 0.0 1604 1604 - A Mar 25 0:14 j2pg
root 74198 0.0 0.0 988 688 - A Mar 25 0:13 /usr/sbin/aixmib
root 61620 0.0 0.0 52 52 - A Mar 25 0:06 rgsr
root 61700 0.0 0.0 48 48 - A Mar 25 0:04 n4bg
root 8750 0.0 0.0 52 52 - A Mar 25 0:04 sched
root 9018 0.0 0.0 52 52 - A Mar 25 0:04 sched
root 12294 0.0 0.0 52 52 - A Mar 25 0:03 sched
root 9554 0.0 0.0 52 52 - A Mar 25 0:03 sched
root 9286 0.0 0.0 52 52 - A Mar 25 0:03 sched
root 8482 0.0 0.0 52 52 - A Mar 25 0:03 sched
root 69684 0.0 0.0 52 52 - A Mar 25 0:02 rdpgc
root 37552 0.0 0.0 324 328 - A Mar 25 0:01 /usr/sbin/syslog
P0604992 79034 0.0 0.0 3168 3164 - A 11:34:05 0:00 ftpd
root 78840 0.0 0.0 684 696 - A 11:28:13 0:00 telnetd -a
oracle 90898 0.0 0.0 696 732 - A Apr 04 0:00 /bin/sh /oracle9
root 111478 0.0 0.0 608 620 - A 06:38:45 0:00 rshd
informix 136008 0.0 0.0 144 148 pts/4 A 12:10:43 0:00 sleep 10
P0605437 152794 0.0 0.0 144 148 pts/1 A 12:10:48 0:00 sleep 1
P0605437 91166 0.0 0.0 156 160 pts/0 A 12:10:48 0:00 [cut]
P0605437 198064 0.0 0.0 3480 3072 pts/0 A 12:10:48 0:00 [onstat]
root 83138 0.0 0.0 572 588 - A Mar 25 0:00 rlogind
informix 181696 0.0 0.0 144 148 - A 12:10:21 0:00 sleep 900
root 115542 0.0 0.0 48 48 - A Mar 26 0:00 aioserver
root 30098 0.0 0.0 60 60 - A Mar 25 0:00 nfsSM
root 26060 0.0 0.0 52 52 - A Mar 25 0:00 usbCfgDev
informix 91578 0.0 0.0 716 704 pts/9 T Apr 16 0:00 vi zeze
root 38274 0.0 0.0 684 696 - A Apr 27 0:00 telnetd -a
root 42162 0.0 0.0 3352 3372 - A Mar 25 0:00 /usr/sbin/rpc.mo
root 50368 0.0 0.0 48 48 - A Mar 25 0:00 random
root 54694 0.0 0.0 684 696 - A Apr 27 0:00 telnetd -a
P0605437 107868 0.0 0.0 636 280 pts/0 A Apr 27 0:00 -sh
root 177038 0.0 0.0 700 528 - A 12:00:00 0:00 /bin/bsh /usr/li
root 164626 0.0 0.0 608 620 - A 19:09:49 0:00 rshd
root 70830 0.0 0.0 572 588 - A Mar 25 0:00 rlogind
informix 152842 0.0 0.0 644 300 pts/7 A May 02 0:00 -ksh
informix 222014 0.0 0.0 144 148 pts/3 A 12:10:46 0:00 sleep 10
P0605437 111964 0.0 0.0 636 280 pts/1 A Apr 27 0:00 -sh
root 58518 0.0 0.0 700 736 pts/12 A 11:30:41 0:00 sh
root 49586 0.0 0.0 52 48 - A Mar 25 0:00 hdlm_log_push_process
root 57768 0.0 0.0 1012 1012 - A Mar 25 0:00 dog
root 78144 0.0 0.0 708 732 pts/12 A 12:10:48 0:00 ps aux
root 24922 0.0 0.0 52 52 - A Mar 25 0:00 netm
root 29022 0.0 0.0 64 64 - A Mar 25 0:00 wlmsched
root 33132 0.0 0.0 48 48 - A Mar 25 0:00 lvmbb
root 82412 0.0 0.0 276 276 - A Mar 25 0:00 /usr/sbin/rpc.lo
root 180516 0.0 0.0 596 612 - A May 02 0:00 rlogind
informix 225582 0.0 0.0 716 752 - A 06:38:45 0:00 -ksh -c /home/ad
informix 233830 0.0 0.0 720 756 - A 19:09:49 0:00 -ksh -c /home/ad
root 143776 0.0 0.0 724 760 pts/4 A Mar 25 0:00 -ksh
informix 151948 0.0 0.0 636 280 pts/4 A Mar 25 0:00 -ksh
P0604992 176496 0.0 0.0 716 752 pts/12 A 11:28:13 0:00 -sh
root 36882 0.0 0.0 48 48 - A Mar 25 0:00 psgc
informix 77900 0.0 0.0 648 280 pts/8 A Mar 27 0:00 -ksh
root 90340 0.0 0.0 844 880 vty1 A Mar 25 0:00 /bin/ksh /usr/li
root 20490 0.0 0.0 48 48 - A Mar 25 0:00 vmptacrt
root 28686 0.0 0.0 48 48 - A Mar 25 0:00 memp_rbd
root 32784 0.0 0.0 52 52 - A Mar 25 0:00 memgrdd
informix 127162 0.0 0.0 648 304 pts/9 A Mar 26 0:00 -ksh
informix 204804 0.0 0.0 144 148 pts/9 A 12:10:43 0:00 sleep 10
root 217204 0.0 0.0 672 660 pts/12 A 11:30:33 0:00 vi /tmp/x
informix 221186 0.0 0.0 144 148 pts/7 A 12:10:46 0:00 sleep 10
F0106042 135284 0.0 0.0 716 744 pts/9 A Mar 26 0:00 -sh
root 143612 0.0 0.0 596 612 - A May 02 0:00 rlogind
informix 147610 0.0 0.0 144 148 - A 12:10:46 0:00 sleep 2
P0605437 160490 0.0 0.0 148 156 pts/0 A 12:10:48 0:00 [wc]
root 184928 0.0 0.0 684 696 - A Mar 27 0:00 telnetd -a
root 209466 0.0 0.0 684 696 - A Mar 26 0:00 telnetd -a
informix 123472 0.0 0.0 144 148 - A 12:10:47 0:00 sleep 2
informix 131686 0.0 0.0 144 148 pts/6 A 12:10:45 0:00 sleep 10
informix 135712 0.0 0.0 632 280 pts/3 A Mar 25 0:00 -ksh
F0106042 225804 0.0 0.0 716 748 pts/8 A Mar 27 0:00 -sh
root 58354 0.0 0.0 96 96 vty1 A Mar 25 0:00 /usr/sbin/instal
root 62458 0.0 0.0 148 148 - A Mar 25 0:00 /usr/sbin/biod 6
root 66538 0.0 0.0 196 188 - A Mar 25 0:00 /usr/sbin/nfsd 3
root 25416 0.0 0.0 48 48 - A Mar 25 0:00 hdlm_link_proc
root 49944 0.0 0.0 60 60 - A Mar 25 0:00 kbiod
root 54030 0.0 0.0 1004 1000 - A Mar 25 0:00 /usr/sbin/portma
root 274934 0.0 0.0 208 216 - A 12:00:00 0:00 /usr/lib/sa/sadc
daemon 33532 0.0 0.0 2880 2856 - A Mar 25 0:00 /usr/sbin/rpc.st
root 41680 0.0 0.0 768 764 - A Mar 25 0:00 /usr/sbin/srcmst
informix 238064 0.0 0.0 144 148 pts/8 A 12:10:42 0:00 sleep 10
P0605437 242058 0.0 0.0 212 224 pts/0 A 12:10:48 0:00 [grep]
informix 270836 0.0 0.0 636 280 pts/6 A May 02 0:00 -ksh
root 53948 0.0 0.0 736 680 - A Mar 25 0:00 /usr/lib/errdemo
root 74490 0.0 0.0 2564 1484 - A Mar 25 0:00 /usr/local/sbin/
informix 82624 0.0 0.0 144 148 - A 11:42:33 0:00 sleep 1800
root 119362 0.0 0.0 724 760 pts/3 A Mar 25 0:00 -ksh
root 58092 0.0 0.0 48 48 - A Mar 25 0:00 rtcmd
root 62164 0.0 0.0 160 140 - A Mar 25 0:00 /usr/ccs/bin/shl
informix 66056 0.0 0.0 716 704 pts/8 T Apr 16 0:00 vi zeze

 

[khalidaaa]

I'm not familiar with informix but it seems that the problem with oninit process of the informix! I believe this process is used to initialize informix so why do you have that much instances of the same process?

How many CPUs do you have on that system? It seems you have 24 CPUs on that LPAR!

Regards,
Khalid

 

[gutolm]

The oninit process is normal on this server. The informix is my aplication, and the big number of process is normal, because each connection opens a new process named oniit.

But my problem is the average %sys utilization.

I have 12 dual-core processors.

See the return of sar command:

08:00:01 %usr %sys %wio %idle physc
08:20:01 7 7 0 85 12.00
08:40:01 11 17 0 72 12.00
09:00:00 16 25 0 59 12.00
09:20:00 15 31 0 54 12.00
09:40:00 17 43 0 41 12.00
10:00:00 20 47 0 33 12.00
10:20:00 19 49 0 32 12.00
10:40:00 18 48 0 34 12.00
11:00:00 24 54 0 22 12.00
11:20:00 23 56 0 21 12.00
11:40:00 23 55 0 22 12.00
12:00:00 23 48 0 30 12.00

The average %sys utilization, it's above the normal.

 

[khalidaaa]

It depends on what the informix application's users do!

Is it normal to have that much iowait?

Check iostat

Regards,
Khalid

 

[RMGBELGIUM]

badly written queries can cause such behaviour,seen it many times before ...

rgds,

R.

 

[zaxxon]

Can you post a "vmstat 1 20" and an "iostat 1 5" over the time the bad performance is occuring?
Additionally a vmstat -v might come handy.
Is Async. I/O activated? "lsdev -Cl aio0"
If it is available, get a "lsattr -El aio0" please too.

Did you do any tuning so far or is the box running like it was installed?

laters
zaxxon