Aix LPAR Performance question?

[sreedca]

Hi,
We have a Aix5.2 LPAR on P690,running Production Websphere/MQ/Maestro.topas,nmon64 ahows a Cpu wait of 90% or so.Can someone let me know,what should i be looking for?.also ia there any tool,i can use to consistantly get top applications cpu and memory usage?

Thanks

Sreedca

 

[RiazAhamed]

Hi Sreedca,

To start with, check nmon utility from the IBM website:

http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/index.html

You can start with nmon download and install the same onto your p690 LPAR.
Good luck.

 

[sreedca]

Hi

I am using nmon utility.It shows CPU utilization wau=it state at 90% range.But disk usage does not sem too much.

Thx

Sreedca

 

[sreedca]

Hi,
Here's the output of vmstat,showing pending io waits.Also my nmon output.No clue what's causing cpu wait?

Thanks for any help.

Sreedca

vmstat -s
1536645018 total address trans. faults
364414958 page ins
203448927 page outs
252736 paging space page ins
586728 paging space page outs
0 total reclaims
429601399 zero filled pages faults
3665418 executable filled pages faults
34377547 pages examined by clock
12 revolutions of the clock hand
6275626 pages freed by the clock
17643063 backtracks
0 lock misses
9 free frame waits
0 extend XPT waits
313647459 pending I/O waits
486820606 start I/Os
486798848 iodones
8301403453 cpu context switches
4196410567 device interrupts
0 software interrupts
0 traps
32277813056 syscalls

My nmon output

nmon64 v9a Hostname=ctap7apps1 Refresh=2.0secs 12:28.31
CPU Utilisation +-------------------------------------------------+
CPU User% Sys% Wait% Idle|0 |25 |50 |75 100|
0 0.0 0.5 0.0 99.5| > |
1 0.0 1.0 94.5 4.5|

http://WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW

>
2 0.0 1.0 0.0 99.0| > |
3 0.5 0.0 4.5 95.0|WW > |
+-------------------------------------------------+
0.1 0.6 24.8 74.5|

http://WWWWWWWWWWWW

> |
+-------------------------------------------------+
Memory Use Physical Virtual Paging pages/sec In Out VM parameters
% Used 78.7% 2.4% to Paging Space 0.0 0.0 numperm 63.3%
% Free 21.3% 97.6% to File System 138.1 0.0 minperm 19.2%
MB Used 8053.9MB 595.2MB Page Scans 0.0 maxperm 76.7%
MB Free 2186.0MB 23980.8MB Page Cycles 0.0 minfree 3072
Total(MB) 10239.8MB 24576.0MB Page Reclaim 0.0 maxfree 10240
Top Processes Procs=341 mode=3 (1=Basic, 2=CPU 3=Perf 4=Size 5=I/O w=wait-procs)
PID %CPU Size Res Res Res Char RAM Paging Command
Used KB Set Text Data I/O Use io other repage
106548 1.0 56 32 0 32 0 0% 0 0 0 wlmsched
2076896 1.0 472 484 36 448 0 0% 137 103 0 find
1966112 0.5 2848 2884 124 2760 4083 0% 0 0 0 nmon64
1884236 0.5 2672 1600 400 1200 0 0% 0 0 0 snatnsrvr_mt
73764 0.0 56 52 0 52 0 0% 0 0 0 reaper
77862 0.0 60 36 0 36 0 0% 0 0 0 lrud
81960 0.0 56 36 0 36 0 0% 0 0 0 xmdetd
86058 0.0 56 36 0 36 0 0% 0 0 0 vmptacrt
90156 0.0 80 64 0 64 0 0% 0 0 0 pilegc
94254 0.0 56 44 0 44 0 0% 0 0 0 xmgc
98352 0.0 56 44 0 44 0 0% 0 0 0 netm
102450 0.0 152 128 0 128 0 0% 0 0 0 gil = TCP/IP
1 0.0 880 340 36 304 0 0% 0 0 0 init
114842 0.0 56 32 0 32 0 0% 0 0 0 aioserver
118856 0.0 56 32 0 32 0 0% 0 0 0 aioserver
122966 0.0 56 32 0 32 0 0% 0 0 0 aioserver
127048 0.0 104 32 0 32 0 0% 0 0 0 dog
131176 0.0 56 32 0 32 0 0% 0 0 0 aioserver
135244 0.0 56 32 0 32 0 0% 0 0 0 aioserver
139336 0.0 56 32 0 32 0 0% 0 0 0 aioserver
143434 0.0 56 32 0 32 0 0% 0 0 0 aioserver
147602 0.0 56 32 0 32 0 0% 0 0 0 aioserver
155762 0.0 56 32 0 32 0 0% 0 0 0 aioserver
159842 0.0 56 32 0 32 0 0% 0 0 0 aioserver
163982 0.0 668 304 8 296 616 0% 0 0 0 syncd

 

[dlpar]

whay about this commands:
ps auwwx
vmstat 1

 

[sreedca]

Here's the outpu for ps and vmstat

/var/adm ==> ps -auwwx
UID PID TTY TIME CMD
0 1282106 pts/4 0:00 ksh
0 1356010 pts/3 0:00 ksh
337 1523766 pts/6 0:00 vi
0 1552558 pts/0 0:00 ksh
0 1798168 pts/0 0:00 ps
0 1888276 pts/1 0:00 ksh
0 2113554 pts/1 0:02 vi
0 2191508 0:00 <defunct>

/var/adm ==> vmstat 1
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 530260 208679 0 0 0 7 33 0 2426 19604 4583 7 4 61 27
0 1 530265 208676 0 0 0 0 0 0 826 15617 841 0 2 74 24
0 1 530265 208705 0 0 0 0 0 0 825 11710 800 0 2 74 24
0 1 530265 208760 0 0 0 0 0 0 828 10592 734 0 0 75 25
1 0 530265 208757 0 0 0 0 0 0 800 10380 714 0 1 73 25
0 1 530295 209285 0 0 0 0 0 0 922 16810 1024 1 2 72 24
0 1 530020 209565 0 0 0 0 0 0 827 14552 821 0 2 74 24
0 1 530020 209571 0 0 0 0 0 0 842 12785 841 0 2 72 26
0 1 530020 209671 0 0 0 0 0 0 860 11182 795 0 2 74 23
0 1 530020 209911 0 0 0 0 0 0 872 12324 863 0 2 74 24
0 1 530020 209913 0 0 0 0 0 0 847 13197 945 1 1 74 24
0 1 530020 210175 0 0 0 0 0 0 844 13352 859 0 2 74 24
0 1 530020 210181 0 0 0 0 0 0 749 10245 662 0 2 74 24
0 1 530020 210177 0 0 0 0 0 0 739 8825 570 0 1 74 24
0 1 530020 210176 0 0 0 0 0 0 749 9527 610 0 1 74 24
0 1 530050 209876 0 0 0 0 0 0 925 15979 964 1 3 71 25


Thanks


sreedca

 

[dlpar]

ps auwwx (whithout -)

 

[sreedca]

Hi,

Here's the out put of ps auwwx
thx

Sri
USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 20490 22.7 0.0 48 44 - A Oct 10 30416:22 wait
root 16392 22.3 0.0 48 44 - A Oct 10 29869:36 wait
root 8196 22.2 0.0 48 40 - A Oct 10 29851:52 wait
root 12294 22.0 0.0 48 44 - A Oct 10 29595:38 wait
root 1626266 1.2 0.0 472 484 - A 08:00:01 4:38 find / -name core -exec rm {} ;
root 163982 0.5 0.0 668 300 - A Oct 10 629:00 /usr/sbin/syncd 60
root 757902 0.4 2.0 144604 119636 - A Oct 10 491:30 java -Xms32M -Xmx256M -DATEVAL_CHAN=jni -showversion -DZIP=1 -DARCH=aix4-r1 -DLCF_DATDIR=/opt/Tivoli/
lcf/dat/1 -DLCF_EPLABEL=ctap7apps1 -DTRACE_LEVEL=0 -DTRACE_SIZE=5000000 -DREBOOT=true -DIP_ADDRESS=172.31.99.97 -DFQHOSTNAME=ctap7apps1.ctal.ctc -DREHOSTNAME=ctap7apps1 -
Ddb2j.stream.error.file=/opt/Tivoli/lcf/dat/1/LCFNEW/AMW/logs/db2j.log -Ddb2j.stream.error.logSeverityLevel=50000 -Ddb2j.infolog.append=false -Ddb2j.language.logStatement
Text= -Ddb2j.locks.waitTimeout=360 -Ddb2j.drda.host=172.31.99.97 com.tivoli.dmunix.ep.agent.Main
root 1568794 0.3 1.0 62616 45992 - A Oct 10 427:06 /usr/tivoli/TSRM/java/bin/java -DTivoliTSRM.servicePipe=/etc/Tivoli/TSRM/lock/TSRMagt1 -Xmx128m com.tiv
oli.itsrm.agent.svp.Agent
root 1589296 0.1 0.0 4572 2680 - A Oct 10 107:40 /usr/sbin/rsct/bin/IBM.CSMAgentRMd
root 106548 0.1 0.0 56 32 - A Oct 10 103:28 wlmsched
root 487674 0.0 0.0 428 372 - A Oct 10 65:47 nfsd
root 1630238 0.0 0.0 1772 480 - A Oct 10 65:35 dtgreet
root 102450 0.0 0.0 152 128 - A Oct 10 52:23 gil
root 0 0.0 0.0 64 60 - A Oct 10 47:19 swapper
root 1835148 0.0 0.0 2432 2300 - A Oct 10 25:09 bin/tecad_logfile -n -c /opt/Tivoli/lcf/dat/1/../../bin/aix4-r1/TME/TEC/adapters/etc/tecad_logfile.conf
root 1458378 0.0 0.0 2288 1792 - A Oct 10 16:34 /usr/sbin/sddsrv
root 491770 0.0 0.0 776 464 - A Oct 10 16:22 /bin/ksh /direct/scripts/ftmaster
was 1667074 0.0 0.0 764 804 pts/5 A 16:34:17 0:26 /bin/ksh ./countConnections.sh
maestro 1646616 0.0 0.0 2700 2844 - A 11:59:55 0:30 /usr/local/Tivoli/maestro/bin/batchman -parm 32000
root 1884236 0.0 0.0 2672 1492 - A Oct 15 9:41 snatnsrvr_mt -p8 -s
root 1831038 0.0 0.0 812 448 - A Oct 10 10:21 /usr/opt/db2_08_01/bin/db2fmcd
nobody 1679480 0.0 0.0 4748 1980 - A 07:45:12 0:02 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
mqm 1618040 0.0 0.0 40244 23620 - A Oct 10 9:19 amqzlaa0 -mCTALQM1 -fip100
mqm 1048598 0.0 0.0 1680 896 - A Oct 10 9:17 amqcrsta
nobody 1593496 0.0 0.0 5200 2036 - A 08:36:00 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
maestro 1282116 0.0 0.0 2936 3308 - A 11:59:50 0:20 /usr/local/Tivoli/maestro/bin/mailman -parm 32000 -- 2002 CTAP7APPS1 CONMAN UNIX 8.2 MESSAGE
mqm 753786 0.0 0.0 19780 7416 - A Oct 10 8:22 DataFlowEngine CTALQM1 89957c9d-fd00-0000-0080-e660bbe89f56 CCSM 0
root 561176 0.0 0.0 972 648 - A Oct 10 8:09 /opt/Tivoli/lcf/bin/aix4-r1/mrt/lcfd
nobody 2064482 0.0 0.0 5188 2488 - A 07:19:13 0:02 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 2080922 0.0 0.0 5012 1868 - A 08:29:20 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 2052322 0.0 0.0 5112 1976 - A 08:29:20 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1192062 0.0 0.0 4984 2304 - A 07:18:08 0:02 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1978514 0.0 0.0 5084 2004 - A 08:22:34 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 2056376 0.0 0.0 5012 1940 - A 08:22:34 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1175714 0.0 0.0 5000 1936 - A 08:22:33 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1806472 0.0 0.0 5160 2104 - A 08:22:33 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 675928 0.0 0.0 5276 2236 - A 08:21:59 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1425450 0.0 0.0 4904 1876 - A 08:21:53 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
mqm 782466 0.0 0.0 21300 7088 - A Oct 10 6:55 DataFlowEngine CTALQM1 1d6a7c9d-fd00-0000-0080-e660bbe89f56 CCSI 0
cdm 1998972 0.0 0.0 3284 2020 - A Oct 15 5:02 /opt/direct/ndm/bin/cdpmgr -i /opt/direct/ndm/cfg/ctalprod/initparm.cfg
nobody 1278188 0.0 0.0 5028 2088 - A 08:11:12 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1404998 0.0 0.0 5232 2268 - A 08:11:12 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1798330 0.0 0.0 5048 2076 - A 08:11:12 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 2121964 0.0 0.0 5104 2156 - A 08:11:12 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer
nobody 1360038 0.0 0.0 5096 2172 - A 08:11:11 0:01 /usr/IBMHttpServer/bin/httpd -d /usr/IBMHttpServer

 

[dlpar]

sredca,

If i look at the vmstat:
You have only 23% of wait
You have 75% of idle
To, the partition looks OK, far away of 90% of wait.

To analyse the CPU,I/Os,... during a long periode i think that you need to use the rotating sar:
For that you can add this in the crontab(adm):
0,5,10,15,20,25,30,35,40,45,50,55 * * * 1-5 /usr/lib/sa/sa1 &
It will store the datas in /var/adm/sa for every day.
To consulte the results:
sar
sar -f sa03 (-d.....)
This way, you will see if the CPU is always at 90% of wait during all the time or no.

In your example, il doesn't look at 90% of wait.