topas gives wrong stats

[RMGBELGIUM]

Hi all,

I've got the following phenomenon on some of our servers :

when I look at the output of the topas command, the first output I get, looks normal

Kernel 29.9 |#########
User 26.3 |########
Wait 20.4 |######
Idle 23.4 |#######

when I get the first refresh, I get this :


Kernel 54.0 |################
User 46.0 |#############
Wait 0.0 |
Idle 0.0 |


The system is getting a 0% idle time.The same goes with vmstat :

take a look at the output of vmstat 5 2


kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 934011 6691 0 11 11 183 500 0 566 29270 1209 26 30 23 20
9 3 932412 7538 0 4 3 0 0 0 1152 41167 2626 56 44 0 0

We don't have that problem on our other machines, and the only difference is that the other systems are dual processor, and this one is single proc.
I'm running AIX5.2 on these machines, on 1 of the problem machines ML 4 has been installed, but the problem has't gone away.
Has anyone encountered this kind of behaviour before?

thx in advance,

greetingz and a happy newyear

RMGBelgium

 

[banders]

Looks to me that you are running into some CPU and Disk bound issues. The first output from topas and vmstat are statistics since the boot time of the machine. So until you see the second statistic and so on, you won't see the correct information. Your run queue on the vmstat output is very high for a single cpu machine, but you'll need to post some more info. vmstat will need to be run for a longer period of time. Also, post the output of the vmtune -a command. Have you looked at iostat?

 

[RMGBELGIUM]

Hello banders,

First of all, thx for the respons!

Here's the output from vmstat 2 15 :
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 828200 1642 0 11 11 185 504 0 568 29800 1315 27 30 23 20
5 3 825239 4277 0 2 0 0 0 0 584 74190 3915 52 48 0 0
5 2 825237 3930 0 1 0 0 0 0 596 37349 4548 54 46 0 0
5 2 825267 3543 0 2 0 0 0 0 724 38578 5123 51 49 0 0
5 1 826334 1789 0 26 0 0 0 0 599 36119 4303 50 50 0 0
3 3 825615 2151 0 0 0 0 0 0 631 40967 4022 48 52 0 0
4 1 823523 3697 0 1 0 0 0 0 457 45881 5683 63 37 0 0
4 0 823477 3181 0 7 0 0 0 0 411 44466 4855 61 39 0 0
4 1 823762 2391 0 3 0 0 0 0 394 36765 4116 47 53 0 0
6 1 823501 2254 0 2 0 0 0 0 518 38860 6020 57 42 0 0
4 1 823486 1105 0 3 0 0 0 0 532 50194 6392 58 42 0 0
6 1 828260 594 0 6 4 2148 3548 0 705 36694 5571 52 48 0 0
5 1 828303 514 0 5 0 0 0 0 669 35228 5616 50 50 0 0
8 1 828263 446 0 3 0 134 241 0 661 40196 4788 48 52 0 0
7 0 828306 393 0 8 1 467 759 0 641 42150 5098 50 50 0 0


iostat :

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.2 31.4 26.7 30.4 22.9 20.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 2.0 19.6 4.3 25049226 52720344
hdisk0 2.7 36.7 5.3 84544130 60584251
hdskC213 2.7 23.6 3.1 16786313 76744360
hdskA216 2.5 23.9 3.5 33585137 61083688
hdskA209 0.9 10.8 1.0 21770057 20929076
hdskA214 0.2 12.7 0.2 16674641 33662816
hdskC214 0.2 12.8 0.2 17147537 33662816
hdskC216 0.8 8.7 0.8 13533273 20929076
hdskA210 6.5 48.9 6.7 63490762 130014300
hdskC209 2.7 28.5 3.3 65598020 47104316
hdskC210 6.6 44.4 6.0 37849557 137978380
hdskF105 0.5 28.3 1.0 76422651 35717303
hdskF101 0.3 14.0 0.8 51688622 3751405
hdskF107 2.9 34.8 4.1 47692275 90122405
hdskE105 0.4 24.7 0.8 62251955 35717303
hdskE101 0.2 4.1 0.4 12554783 3751401
hdskF103 3.8 55.7 5.7 127259223 93262583
hdskF104 0.0 0.0 0.0 15792 0
hdskA212 3.0 26.6 3.6 28601309 76823872
hdskE103 2.5 43.6 4.1 79480211 93229815
hdskF106 0.0 0.0 0.0 2115 0
hdskE107 2.7 30.1 3.5 28210659 91058085
hdskA213 2.9 17.9 3.7 15423398 55477504
hdskE104 0.0 0.0 0.0 2115 0
hdskE106 0.0 0.0 0.0 2115 0
hdskC211 0.2 3.9 0.3 9768413 5657204
hdskA211 0.2 4.3 0.4 10346593 6737032
cd0 0.0 0.0 0.0 0 0
hdskC212 2.4 19.2 2.7 14827785 61041216
hdskC408 0.0 0.0 0.0 0 0
hdskC405 0.0 0.0 0.0 0 0
hdskB308 0.0 0.0 0.0 0 0
hdskB305 0.0 0.0 0.0 0 0


the output from vmtune -a :

memory_frames = 1048576
pinnable_frames = 835977
maxfree = 128
minfree = 120
minperm% = 20
minperm = 193356
maxperm% = 80
maxperm = 773424
strict_maxperm = 0
maxpin% = 80
maxpin = 838861
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 2228224
npskill = 17408
npswarn = 69632
v_pinshm = 0
pta_balance_threshold = 0
pagecoloring = 0
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = -1
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
minpgahead = 2
maxpgahead = 8
pd_npages = 65536
maxrandwrt = 0
numclust = 1
numfsbufs = 196
sync_release_ilock = 0
lvm_bufcnt = 9
j2_minPageReadAhead = 2
j2_maxPageReadAhead = 8
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_maxRandomWrite = 0
j2_nRandomCluster = 0
jfs_clread_enabled = 0
jfs_use_read_lock = 1
hd_pvs_opn = 24
hd_pbuf_cnt = 3584
hd_pendqblked = 5
psbufwaitcnt = 5567604
fsbufwaitcnt = 2554969
rfsbufwaitcnt = 0
xpagerbufwaitcnt = 4000


greetingz,have a Merry Newyear and Happy Xmazz!

R.

 

[zaxxon]

Hi,
for having different numbers on the first record of vmstat, this is explained by the manpage of ps, ie. it is the average since startup:

"The Interval parameter specifies the amount of time in seconds between each
report. The first report contains statistics for the time since system startup.
Subsequent reports contain statistics collected during the interval since the
previous report. If the Interval parameter is not specified, the vmstat command
generates a single report and then exits. The Count parameter can only be..."

I would just do a
"ps aux

http://wwwww|

head -10"
to get a top 10 of your processes, sorted by CPU activity to see what is really going on.


laters
zaxxon

 

[banders]

I had similiar output from vmstat from one of our AIX machines. In the fre column you can see that your memory fre pages are constantly reducing. Based upon your vmtune settings minfree and maxfree, as memory is used on your machine, pages in memory won't be freed until it hits the maxfree mark. Thus you are seeing numbers under the po column and a high number in your sr and fr columns. The po is your paging out activity and the sr and fr columns are how many memory pages are scanned to be freed and the fr column is the actual pages that were freed. By reviewing some performance tuning documents, I noticed that the maxperm% from vmtune was set too high. What this does is set the limit as to where file pages will be kept in memory versus computational pages. I took and reduced the maxperm% from 80% to 50% and then I kept a close eye on it to see what it would do. You can make this change on the fly with the vmo command. Remember the maxclient% has to be changed at the same time. The maxclient% has to be equal or less than maxperm%. The other thing I noticed was that your fsbufwaitcnt from vmtune was high. This is the number of times that a filesystem buffer in memory is waited for. By increasing the number of buffers, it can reduce this quantity and help performance. Please post the output of svmon -G. I have some other recommendations as well, but I want to see your svmon output first. Also, these parameters are set until boot time. AIX 5.2 has made it easier to maintain the settings across boot, so I can show you how to do that as well.

 

[zaxxon]

Oops,
just saw it, I meant "manpage of vmstat". Sorry for the confusion.

laters
zaxxon

 

[RMGBELGIUM]

Hi all,

first of all zaxxon : I've read the man pages for vmstat, but the thing I don't get : whenever I do a vmstat, the idle time of my machine is 0.How does the average value get 22.9, if in 99 % of the time I'm getting an idle time 0?( and the machine has an uptime of several weeks)?

Hi Banders,

Thx so far for the feedback
here's the output of svmon -G:

size inuse free pin virtual
memory 1048576 1048454 122 212940 871233
pg space 2228224 506884

work pers clnt lpage
pin 212696 0 244 0
in use 556721 486798 4935 0


thx in advance,

greetingz,

RMGBelgium

 

[banders]

Based upon your svmon output, you are using about all of your memory on the machine, which in turn is causing paging space usage. It looks like you have 4 gb of memory and about 9 gb of paging space. Your are using all of the 4 gb and about 2 gb of your paging space. I would make the changes that I recommended in my previous post. The main ones to look at now are the maxperm% and maxclient%. I'd make that change and then monitor your vmstat output and vmtune info and repost that output after a day or so.

 

[zaxxon]

You wrote "How does the average value get 22.9, if in 99 % of the time I'm getting an idle time 0?"

Did you really monitor it's CPU usage since startup? Not that I get it wrong: Is 99% that you get the message same as 99% of the uptime of the box or just 99% of the time you check it?
Couldn't it be that it was running well in the first time and got worse later which might explain the average 22,9% idle time?

Can you please post a full "ps aux

http://wwww"?

I am just curious, thank you in forward.

laters
zaxxon

 

[RMGBELGIUM]

Hi zaxxon,

I've been monitoring the box on a daily basis since startup.At any time any day, the stats are the same.

here's the output from ps aux

http://wwww:

USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 401564 57.7 1.0 68048 46476 - A Jan 06 3986:00 /softp/app/oracle/producti/ias902/jdk/bin/java -Xmx256m -classpath :/softp/app/oracle/producti/ias902/ifs/cmsdk/custom_classes:/softp/app/oracle/producti/ias902/ifs/cmsdk/samplecode:/softp/app/oracle/producti/ias902/ord/Annotator/lib/Annotator.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/settings:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk_client.jar:/softp/app/oracle/producti/ias902/lib/servlet.jar:/softp/app/oracle/producti/ias902/lib/xmlparserv2.jar:/softp/app/oracle/producti/ias902/jlib/javax-ssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/jssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/swingall-1_1_1.jar:/softp/app/oracle/producti/ias902/jlib/jewt4.jar:/softp/app/oracle/producti/ias902/jlib/share.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/nis.jar:/softp/app/oracle/producti/ias902/jlib/providerutil.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/aurora.zip:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/jdbc/lib/classes12.zip:/softp/app/oracle/producti/ias902/jdbc/lib/nls_charset12.zip:/softp/app/oracle/producti/ias902/jlib/regexp.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/inspect-4_1_6.jar:/softp/app/oracle/producti/ias902/classes:/softp/app/oracle/producti/ias902/lib/vbjorb.jar:/softp/app/oracle/producti/ias902/jlib/help3.jar:/softp/app/oracle/producti/ias902/jlib/help3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oracle_ice5.jar:/softp/app/oracle/producti/ias902/jlib/ewt3.jar:/softp/app/oracle/producti/ias902/jlib/ewt3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oemlt-9_0_2.jar:/softp/app/oracle/producti/ias902/jlib/emConfigInstall.jar:/softp/app/oracle/producti/ias902/jlib/repository.jar:/softp/app/oracle/producti/ias902/jlib/ldap.jar:/softp/app/oracle/producti/ias902/dcm/lib/oc4j_deploy_tools.jar:/softp/app/oracle/producti/ias902/dcm
/li
root 8196 34.9 0.0 44 44 - A Nov 15 28516:03 wait
root 86136 2.1 0.0 684 288 - A Nov 15 1733:17 /usr/sbin/syncd 20
oracle 1896604 1.0 1.0 61912 34640 - A 08:03:49 0:07 oracleDBP1 (LOCAL=NO)
root 225412 0.7 0.0 1772 240 - A Nov 15 549:47 dtgreet
oracle 1290272 0.7 1.0 61428 34156 - A 08:12:30 0:01 oracleDBP1 (LOCAL=NO)
oracle 2080972 0.4 0.0 15896 1840 - A Dec 19 138:08 /softp/app/oracle/producti/920/bin/tnslsnr LISTENER -inherit
oracle 1273862 0.4 1.0 55344 28064 - A 07:57:35 0:04 oracleDBP1 (LOCAL=NO)
oracle 1511516 0.3 1.0 61620 34348 - A 08:02:43 0:02 oracleDBP1 (LOCAL=NO)
oracle 1806460 0.3 1.0 52520 21796 - A Dec 21 74:01 ora_lgwr_DBP1
oracle 1757388 0.2 1.0 61868 34584 - A 07:54:20 0:03 oracleDBP1 (LOCAL=NO)
prepress 2060392 0.2 0.0 520 296 - A Jan 09 5:06 /softp/oraprog/adman/proc/scan/scan_adman -d/Baskets/Out/AdStatus -m/softp/inout/adman/AdStatus_verwerkt -e/softp/inout/adman/AdStatus_error -b* -xstatus -p/softp/oraprog/adman/proc/readadp/readadp.sh -s/softp/oraprog/adman/proc/readadp/nareadadp.sh -i10
oracle 1261632 0.2 1.0 54476 23452 - A Dec 21 63:26 ora_dbw0_DBP1
root 16392 0.2 0.0 44 40 - A Nov 15 170:26 lrud
oracle 1298612 0.2 1.0 61308 26572 - A 06:46:48 0:08 oracleDBP1 (LOCAL=NO)
root 778388 0.1 0.0 232 180 - A Nov 15 91:25 nfsd
oracle 1359948 0.1 1.0 52904 25616 - A 07:57:51 0:01 oracleDBP1 (LOCAL=NO)
root 200818 0.1 0.0 3140 644 - A Nov 15 78:53 /usr/sbin/rsct/bin/IBM.CSMAgentRMd
root 36882 0.1 0.0 116 96 - A Nov 15 78:29 gil
root 1544258 0.1 0.0 3500 1392 - A 16:03:56 0:51 /usr/local/sbin/sshd2
root 2125846 0.1 0.0 1180 916 - A Dec 19 25:16 /usr/sbin/rsct/bin/hats_ssa_nim
root 0 0.1 0.0 68 60 - A Nov 15 62:34 swapper
svcagent 1568894 0.1 0.0 19480 13240 - A 00:21:52 0:21 java -Xbootclasspath:/usr/svcagent/lib/sa.jar:/usr/svcagent/lib/collections.jar:/usr/svcagent/lib/swingall.jar:/usr/svcagent/lib/xercesImpl.jar:/usr/svcagent/lib/xmlParserAPIs.jar:/usr/svcagent/lib/ibmjsse.jar:/usr/svcagent/lib/Jt.jar:/usr/svcagent/lib/sahelp.jar:/usr/svcagent/lib/ibmpkcs.jar:/usr/java131/jre/lib/rt.jar:/usr/java131/jre/lib/i18n.jar:/usr/java131/jre/classes -DJSpace=on -mx64M com.ibm.ecf.apps.ODS
oracle 1597676 0.1 1.0 61356 22448 - A 06:02:47 0:05 oracleDBP1 (LOCAL=NO)
root 1069090 0.1 0.0 7228 7176 - A Dec 19 18:08 /usr/sbin/rsct/bin/hatsd -n 1 -o deadManSwitch
root 69772 0.0 0.0 88 72 - A Nov 15 38:57 rtcmd
root 1753312 0.0 0.0 28196 15792 - A Jan 06 3:06 /softp/app/oracle/producti/ias902/jdk/bin/java -classpath :/softp/app/oracle/producti/ias902/ifs/cmsdk/custom_classes:/softp/app/oracle/producti/ias902/ifs/cmsdk/samplecode:/softp/app/oracle/producti/ias902/ord/Annotator/lib/Annotator.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/settings:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk_client.jar:/softp/app/oracle/producti/ias902/lib/servlet.jar:/softp/app/oracle/producti/ias902/lib/xmlparserv2.jar:/softp/app/oracle/producti/ias902/jlib/javax-ssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/jssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/swingall-1_1_1.jar:/softp/app/oracle/producti/ias902/jlib/jewt4.jar:/softp/app/oracle/producti/ias902/jlib/share.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/nis.jar:/softp/app/oracle/producti/ias902/jlib/providerutil.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/aurora.zip:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/jdbc/lib/classes12.zip:/softp/app/oracle/producti/ias902/jdbc/lib/nls_charset12.zip:/softp/app/oracle/producti/ias902/jlib/regexp.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/inspect-4_1_6.jar:/softp/app/oracle/producti/ias902/classes:/softp/app/oracle/producti/ias902/lib/vbjorb.jar:/softp/app/oracle/producti/ias902/jlib/help3.jar:/softp/app/oracle/producti/ias902/jlib/help3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oracle_ice5.jar:/softp/app/oracle/producti/ias902/jlib/ewt3.jar:/softp/app/oracle/producti/ias902/jlib/ewt3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oemlt-9_0_2.jar:/softp/app/oracle/producti/ias902/jlib/emConfigInstall.jar:/softp/app/oracle/producti/ias902/jlib/repository.jar:/softp/app/oracle/producti/ias902/jlib/ldap.jar:/softp/app/oracle/producti/ias902/dcm/lib/oc4j_deploy_tools.jar:/softp/app/oracle/producti/ias902/dcm/lib/dcm.ja
r
root 413720 0.0 1.0 34392 31576 - A Dec 19 11:27 /softp/vtom/abm/bin/bdaemon
root 122974 0.0 0.0 1896 1832 - A Nov 15 26:44 kbiod
root 1650852 0.0 0.0 1248 984 - A Dec 19 10:42 /usr/sbin/rsct/bin/hats_nim
root 1691724 0.0 0.0 1248 984 - A Dec 19 9:53 /usr/sbin/rsct/bin/hats_nim
oracle 647332 0.0 1.0 52604 21840 - A Dec 21 8:01 ora_ckpt_DBP1
oracle 1093880 0.0 1.0 52596 21740 - A Dec 19 8:39 ora_ckpt_DBPS
root 1147042 0.0 0.0 9004 4056 - A Jan 06 1:30 /softp/app/oracle/producti/ias902/jdk/bin/java -classpath :/softp/app/oracle/producti/ias902/ifs/cmsdk/custom_classes:/softp/app/oracle/producti/ias902/ifs/cmsdk/samplecode:/softp/app/oracle/producti/ias902/ord/Annotator/lib/Annotator.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/settings:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk_client.jar:/softp/app/oracle/producti/ias902/lib/servlet.jar:/softp/app/oracle/producti/ias902/lib/xmlparserv2.jar:/softp/app/oracle/producti/ias902/jlib/javax-ssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/jssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/swingall-1_1_1.jar:/softp/app/oracle/producti/ias902/jlib/jewt4.jar:/softp/app/oracle/producti/ias902/jlib/share.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/nis.jar:/softp/app/oracle/producti/ias902/jlib/providerutil.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/aurora.zip:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/jdbc/lib/classes12.zip:/softp/app/oracle/producti/ias902/jdbc/lib/nls_charset12.zip:/softp/app/oracle/producti/ias902/jlib/regexp.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/inspect-4_1_6.jar:/softp/app/oracle/producti/ias902/classes:/softp/app/oracle/producti/ias902/lib/vbjorb.jar:/softp/app/oracle/producti/ias902/jlib/help3.jar:/softp/app/oracle/producti/ias902/jlib/help3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oracle_ice5.jar:/softp/app/oracle/producti/ias902/jlib/ewt3.jar:/softp/app/oracle/producti/ias902/jlib/ewt3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oemlt-9_0_2.jar:/softp/app/oracle/producti/ias902/jlib/emConfigInstall.jar:/softp/app/oracle/producti/ias902/jlib/repository.jar:/softp/app/oracle/producti/ias902/jlib/ldap.jar:/softp/app/oracle/producti/ias902/dcm/lib/oc4j_deploy_tools.jar:/softp/app/oracle/producti/ias902/dcm/lib/dcm.jar
oracle 606246 0.0 1.0 52496 21732 - A Dec 19 6:31 ora_ckpt_DBFS1
root 12294 0.0 0.0 52 48 - A Nov 15 15:11 reaper
oracle 1339430 0.0 1.0 56152 23116 - A 06:25:36 0:01 oracleDBP1 (LOCAL=NO)
root 262274 0.0 0.0 2608 1320 - A Nov 15 11:55 /usr/local/sbin/sshd2
root 1728616 0.0 0.0 3660 552 - A Dec 19 4:47 /usr/es/sbin/cluster/clstrmgr
root 770236 0.0 0.0 188 160 - A Nov 15 9:24 rpc.lockd
root 274570 0.0 0.0 56 44 - A Nov 15 8:34 aioserver
root 290972 0.0 0.0 56 44 - A Nov 15 8:27 aioserver
oracle 2117844 0.0 1.0 56116 24640 - A 05:31:10 0:01 oracleDBP1 (LOCAL=NO)
root 254088 0.0 0.0 56 44 - A Nov 15 8:14 aioserver
root 286866 0.0 0.0 56 44 - A Nov 15 8:14 aioserver
root 270472 0.0 0.0 56 44 - A Nov 15 8:13 aioserver
root 258176 0.0 0.0 56 44 - A Nov 15 8:12 aioserver
root 282766 0.0 0.0 56 44 - A Nov 15 8:06 aioserver
oracle 1470612 0.0 1.0 52728 22324 - A Jan 06 0:39 oracleDBFS1 (LOCAL=NO)
oracle 1286220 0.0 1.0 52704 22104 - A Jan 06 0:39 oracleDBFS1 (LOCAL=NO)
root 278668 0.0 0.0 56 44 - A Nov 15 7:38 aioserver
root 73790 0.0 0.0 56 44 - A Nov 15 7:38 aioserver
oracle 2048206 0.0 1.0 56572 22384 - A Dec 21 2:40 ora_arc0_DBP1
root 266374 0.0 0.0 56 40 - A Nov 15 7:20 aioserver
oracle 1941644 0.0 1.0 56572 25856 - A Dec 21 2:34 ora_arc1_DBP1
oracle 2134146 0.0 1.0 56208 22300 - A 05:00:08 0:01 oracleDBP1 (LOCAL=NO)
root 1585390 0.0 0.0 3816 2980 - A Dec 19 2:42 harmad -t HACMP -n Productie
oracle 1990844 0.0 1.0 52560 22012 - A Dec 21 2:18 ora_cjq0_DBP1
oracle 1466612 0.0 1.0 102104 22008 - A Dec 21 2:17 ora_pmon_DBP1
oracle 1708138 0.0 1.0 56332 21424 - A Jan 06 0:28 oracleDBP1 (LOCAL=NO)
root 192612 0.0 0.0 10496 476 - A Nov 15 5:16 /usr/lpp/X11/bin/X -x abx -x dbe -x GLX -D /usr/lib/X11//rgb -T -force :0 -auth /var/dt/A:0-g2MfEa
oracle 2105570 0.0 1.0 53004 22140 - A Dec 21 1:47 ora_smon_DBP1
oracle 925742 0.0 1.0 52512 21828 - A Dec 19 1:58 ora_cjq0_DBPS
oracle 1134812 0.0 1.0 107748 21992 - A Dec 19 1:57 ora_pmon_DBPS
oracle 1179666 0.0 1.0 54476 23284 - A 08:37:20 0:05 oracleDBP1 (LOCAL=NO)
oracle 475278 0.0 1.0 52528 21828 - A Dec 19 1:56 ora_cjq0_DBFS1
prepress 2011184 0.0 0.0 6744 604 - A 11:54:11 0:04 /softp/oraprog/adman/proc/ovb/ovb_answers -d/datacom2/oracleou -i30
oracle 712738 0.0 1.0 107744 21984 - A Dec 19 1:36 ora_pmon_DBFS1
root 1 0.0 0.0 880 164 - A Nov 15 3:51 /etc/init
oracle 512114 0.0 1.0 53040 22032 - A Dec 19 1:28 ora_smon_DBFS1
oracle 1036334 0.0 1.0 52648 22020 - A Dec 19 1:26 ora_smon_DBPS
root 163924 0.0 0.0 1784 164 - A Nov 15 3:26 /usr/es/sbin/cluster/clcomd -d
oracle 1175706 0.0 1.0 52960 22452 - A 11:54:11 0:03 oracleDBP1 (LOCAL=NO)
oracle 2064606 0.0 1.0 54344 21640 - A Dec 19 1:17 ora_dbw0_DBFS1
root 110770 0.0 0.0 408 164 - A Nov 15 2:57 /usr/sbin/syslogd
oracle 1761302 0.0 1.0 57424 21424 - A 08:42:29 0:03 oracleDBP1 (LOCAL=NO)
oracle 708628 0.0 1.0 52452 21672 - A Dec 19 0:55 ora_lgwr_DBFS1
oracle 1114192 0.0 1.0 54340 21636 - A Dec 19 0:49 ora_dbw0_DBPS
oracle 999660 0.0 1.0 57916 23604 - A Jan 06 0:10 oracleDBFS1 (LOCAL=NO)
oracle 1220610 0.0 1.0 57912 23792 - A Jan 06 0:10 oracleDBFS1 (LOCAL=NO)
oracle 1446046 0.0 1.0 57788 22680 - A Jan 06 0:09 oracleDBFS1 (LOCAL=NO)
oracle 1630222 0.0 1.0 57916 23704 - A Jan 06 0:09 oracleDBFS1 (LOCAL=NO)
oracle 844014 0.0 1.0 59508 21424 - A 18:04:32 0:01 oracleDBP1 (LOCAL=NO)
oracle 1105952 0.0 1.0 52460 21676 - A Dec 19 0:34 ora_lgwr_DBPS
prepress 1994780 0.0 0.0 516 288 - A Jan 09 0:02 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/vicking -m/softp/inout/adman/vicking_verwerkt -e/softp/inout/adman/vicking_error -b* -x* -p/softp/oraprog/adman/proc/penverw1/penverw1.sh -s/softp/oraprog/adman/proc/penverw1/napenverw1.sh -i10
root 1593482 0.0 0.0 8056 476 - A Dec 19 0:28 hagsd grpsvcs
oracle 1331434 0.0 1.0 52656 21908 - A 11:54:06 0:01 oracleDBP1 (LOCAL=NO)
root 1294374 0.0 0.0 620 244 - A Dec 19 0:27 /usr/es/sbin/cluster/clinfo
root 1388762 0.0 0.0 1232 212 - A Dec 19 0:27 /usr/sbin/snmpd
oracle 1785992 0.0 1.0 52868 21424 - A 09:52:06 0:01 oracleDBP1 (LOCAL=NO)
root 32784 0.0 0.0 52 44 - A Nov 15 0:58 netm
oracle 532642 0.0 1.0 54844 22472 - A Jan 06 0:04 oracleDBFS1 (LOCAL=NO)
root 143446 0.0 0.0 132 24 - A Nov 15 0:47 /usr/sbin/uprintfd
root 1167366 0.0 0.0 952 108 - A Dec 19 0:15 /usr/sbin/aixmibd
root 28686 0.0 0.0 52 52 - A Nov 15 0:36 xmgc
root 426198 0.0 0.0 792 464 - A Nov 15 0:30 qdaemon
oracle 1016030 0.0 1.0 53696 22356 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
oracle 745660 0.0 1.0 53604 22512 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
oracle 921740 0.0 1.0 53968 22436 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
oracle 483352 0.0 1.0 53664 22432 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
oracle 1527812 0.0 1.0 56772 22484 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
oracle 667778 0.0 1.0 54128 22424 - A Jan 06 0:02 oracleDBFS1 (LOCAL=NO)
root 118970 0.0 0.0 696 328 - A Nov 15 0:22 /usr/sbin/portmap
oracle 438312 0.0 1.0 52464 21704 - A Dec 19 0:06 ora_arc0_DBFS1
oracle 893146 0.0 1.0 52480 21704 - A Dec 19 0:06 ora_arc0_DBPS
root 102616 0.0 0.0 1984 268 - A Nov 15 0:14 /usr/sbin/rsct/bin/rmcd -r
root 94334 0.0 0.0 492 252 - A Nov 15 0:13 /usr/sbin/cron
root 65616 0.0 0.0 52 44 - A Nov 15 0:12 random
root 422138 0.0 0.0 428 96 - A Nov 15 0:10 /usr/sbin/timed -S
root 2101284 0.0 0.0 1856 1056 - A Dec 19 0:04 sendmail: accepting connections
oracle 1515520 0.0 1.0 52548 21944 - A Dec 21 0:02 ora_reco_DBP1
root 209050 0.0 0.0 1228 48 - A Nov 15 0:05 /usr/sbin/rsct/bin/IBM.ServiceRMd
root 176220 0.0 0.0 1336 48 - A Nov 15 0:05 /usr/sbin/rsct/bin/IBM.AuditRMd
root 1654784 0.0 0.0 1076 172 - A Dec 19 0:02 /usr/sbin/snmpmibd
root 217196 0.0 0.0 1840 44 - A Nov 15 0:04 /usr/sbin/rsct/bin/IBM.ERrmd
root 98356 0.0 0.0 132 108 - A Nov 15 0:04 j2pg
root 2093246 0.0 0.0 896 84 - A Dec 19 0:01 /usr/sbin/hostmibd
root 1581240 0.0 0.0 4392 228 - A Dec 19 0:01 haemd HACMP 1 Productie SECNOSUPPORT
root 626866 0.0 0.0 1780 596 - A Nov 15 0:02 /usr/sbin/rpc.mountd
root 188534 0.0 0.0 10648 272 - A Nov 15 0:01 /usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1
root 61530 0.0 0.0 548 320 - A Nov 15 0:01 /usr/sbin/srcmstr
root 53340 0.0 0.0 868 96 - A Nov 15 0:01 /usr/lib/errdemon
root 2023656 0.0 0.0 528 544 pts/1 A 08:15:02 0:00 ps auxwwww
oracle 2044150 Z 0:00 <defunct>
prepress 1638556 0.0 0.0 6716 508 - A 11:54:06 0:00 /softp/oraprog/adman/proc/daem/lees_pipe -bpipe_adman
root 1998936 0.0 0.0 340 24 - A Dec 19 0:00 /usr/sbin/writesrv
oracle 1986714 0.0 1.0 52748 25472 - A 07:49:24 0:00 oracleDBP1 (LOCAL=NO)
linotron 2121940 0.0 0.0 816 520 pts/0 A 05:24:11 0:00 -ksh
oracle 2007242 0.0 1.0 58236 28884 - A 08:06:29 0:00 oracleDBP1 (LOCAL=NO)
oracle 2052278 0.0 1.0 56280 28952 - A 07:45:37 0:00 oracleDBP1 (LOCAL=NO)
oracle 1601722 0.0 1.0 56132 21424 - A 09:27:50 0:00 oracleDBP1 (LOCAL=NO)
prepress 2109614 0.0 0.0 6720 104 - A Jan 09 0:00 /softp/oraprog/adman/proc/daem/lees_pipe -bpipe_wacht
prepress 2113750 0.0 0.0 516 248 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/amoncci -m/softp/inout/adman/amoncci_verwerkt -e/softp/inout/adman/amoncci_error -b* -x* -p/softp/oraprog/adman/proc/amverw1/amverw1.sh -s/softp/oraprog/adman/proc/amverw1/naamverw1.sh -i60
root 2097168 0.0 0.0 748 176 - A 05:24:11 0:00 telnetd -a
oracle 2056324 0.0 1.0 55960 26480 - A 07:05:54 0:00 oracleDBP1 (LOCAL=NO)
oracle 1618102 0.0 1.0 59176 22960 - A 06:58:05 0:00 oracleDBP1 (LOCAL=NO)
oracle 2085040 0.0 1.0 52756 25476 - A 07:56:15 0:00 oracleDBP1 (LOCAL=NO)
oracle 1867828 0.0 1.0 52656 21508 - A Jan 09 0:00 oracleDBP1 (LOCAL=NO)
prepress 1855592 0.0 0.0 492 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/readadp2 -m/softp/inout/adman/adp/backup -e/softp/inout/adman//readadp2_error -b* -xTXT -p/softp/oraprog/adman/proc/readadp2/readadp2.sh -s/softp/oraprog/adman/proc/readadp2/nareadadp2.sh -i10
root 1740882 0.0 0.0 660 20 - A Dec 19 0:00 /usr/sbin/muxatmd
oracle 1872012 0.0 1.0 56124 28852 - A 08:12:39 0:00 oracleDBP1 (LOCAL=NO)
oracle 1847306 0.0 1.0 52680 21976 - A Jan 06 0:00 oracleDBFS1 (LOCAL=NO)
oracle 1802316 0.0 1.0 52648 21508 - A Jan 09 0:00 oracleDBP1 (LOCAL=NO)
prepress 1790078 0.0 0.0 516 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/amis_verwerkt -m/softp/inout/adman/amis_verwerkt_na_2 -e/softp/inout/adman/amis_error_na_2 -b* -x* -p/softp/oraprog/adman/proc/amiverw2/amiverw2.sh -s/softp/oraprog/adman/proc/amiverw2/naamiverw2.sh -i500
prepress 1831136 0.0 0.0 516 208 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/ppverw1 -m/softp/inout/adman/ppverw1_verwerkt -e/softp/inout/adman/ppverw1_error -b* -xNDP -p/softp/oraprog/adman/proc/ppverw1/ppverw1.sh -s/softp/oraprog/adman/proc/ppverw1/nappverw1.sh -i15
oracle 1818834 0.0 1.0 52656 21424 - A Jan 06 0:00 oracleDBP1 (LOCAL=NO)
prepress 1712286 0.0 0.0 492 200 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/seqmedia -m/softp/inout/adman/seqmedia/backup -e/softp/inout/adman/seqmedia/backup -b* -xSOR -p/softp/oraprog/adman/proc/seqmedia/seqmedia.sh -s/softp/oraprog/adman/proc/seqmedia/naseqmedia.sh -i300
prepress 1929462 0.0 0.0 676 712 - A 08:15:02 0:00 sh -c exec /softp/oraprog/adman/proc/readadp/readadp.sh /Baskets/Out/AdStatus/529756A5-081408.status
oracle 1958036 0.0 1.0 56116 24116 - A 06:01:17 0:00 oracleDBP1 (LOCAL=NO)
oracle 1953982 0.0 0.0 804 236 pts/4 A Jan 06 0:00 -ksh
adm 1724666 0.0 0.0 224 232 - A 08:00:07 0:00 /usr/lib/sa/sadc 1200 3 /var/adm/sa/sa11
prepress 1888422 0.0 0.0 6760 6788 - A 08:15:02 0:00 ./readadp /Baskets/Out/AdStatus/529756A5-081408.status
root 1732626 0.0 0.0 84 16 - A Dec 17 0:00 /usr/lib/methods/tmssad -f /dev/tmssako
oracle 1917046 0.0 1.0 56208 23080 - A 06:55:38 0:00 oracleDBP1 (LOCAL=NO)
prepress 1908832 0.0 0.0 492 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/adsmon -m/softp/inout/adman/adsmon_verwerkt -e/softp/inout/adman/adsmon_error -b* -x* -p/softp/oraprog/adman/proc/adsverw1/adsverw1.sh -s/softp/oraprog/adman/proc/adsverw1/naadsverw1.sh -i10
oracle 360606 0.0 1.0 56128 28856 - A 08:13:42 0:00 oracleDBP1 (LOCAL=NO)
svcagent 389238 0.0 0.0 920 128 - A 00:22:03 0:00 /usr/bin/errpt -c -g -s0111002205
root 405684 0.0 0.0 748 32 - A Jan 06 0:00 telnetd -a
prepress 331888 0.0 0.0 492 192 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/wetber -m/softp/inout/adman/wetber_verwerkt -e/softp/inout/adman/wetber_error -b* -xWET -p/softp/oraprog/adman/proc/wetverw1/wetverw1.sh -s/softp/oraprog/adman/proc/wetverw1/nawetverw1.sh -i1800
root 229506 0.0 0.0 1720 452 pts/1 A 16:03:59 0:00 -ksh
oracle 241806 0.0 1.0 56088 28656 - A 07:27:46 0:00 oracleDBP1 (LOCAL=NO)
oracle 303232 0.0 1.0 56204 23528 - A 06:05:34 0:00 oracleDBP1 (LOCAL=NO)
root 524454 0.0 0.0 352 16 - A Nov 15 0:00 /usr/sbin/rpc.lockd -d 0
root 553128 0.0 0.0 200 16 - A Nov 15 0:00 /usr/sbin/biod 6
adm 725006 0.0 0.0 1284 780 - A 08:00:02 0:00 /bin/bsh /usr/lib/sa/sa1 1200 3
oracle 516200 0.0 1.0 56076 28804 - A 08:12:12 0:00 oracleDBP1 (LOCAL=NO)
oracle 418024 0.0 1.0 52428 21548 - A Dec 19 0:00 ora_arc1_DBFS1
oracle 430118 0.0 1.0 56180 28896 - A 08:10:29 0:00 oracleDBP1 (LOCAL=NO)
oracle 479234 0.0 1.0 52504 21880 - A Dec 19 0:00 ora_reco_DBFS1
root 221324 0.0 0.0 684 16 - A Nov 15 0:00 dtlogin <:0>
root 57382 0.0 0.0 52 32 - A Nov 15 0:00 lvmbb
root 90180 0.0 0.0 88 16 - A Nov 15 0:00 /usr/lib/methods/ssa_daemon -l ssa1
root 114756 0.0 0.0 168 144 - A Nov 15 0:00 /usr/ccs/bin/shlap64
root 49298 0.0 0.0 88 16 - A Nov 15 0:00 /usr/lib/methods/ssa_daemon -l ssa0
root 20490 0.0 0.0 52 40 - A Nov 15 0:00 xmdetd
root 24588 0.0 0.0 52 40 - A Nov 15 0:00 vmptacrt
root 40980 0.0 0.0 52 32 - A Nov 15 0:00 wlmsched
imnadm 172116 0.0 0.0 368 32 - A Nov 15 0:00 /usr/IMNSearch/httpdlite/httpdlite -r /etc/IMNSearch/httpdlite/httpdlite.conf
root 180314 0.0 0.0 472 16 - A Nov 15 0:00 /usr/dt/bin/dtlogin
oracle 205016 0.0 1.0 55968 28688 - A 08:06:17 0:00 oracleDBP1 (LOCAL=NO)
root 159824 0.0 0.0 348 176 - A Nov 15 0:00 /usr/lpp/diagnostics/bin/diagd
root 131296 0.0 0.0 500 216 - A Nov 15 0:00 /usr/sbin/inetd
svcagent 135302 0.0 0.0 748 236 - A Nov 15 0:00 -ksh /usr/svcagent/bin/ods
root 139362 0.0 0.0 724 16 - A Nov 15 0:00 /usr/bin/itesmdem itesrv.ini /etc/IMNSearch/search/
daemon 729154 0.0 0.0 996 16 - A Nov 15 0:00 /usr/sbin/rpc.statd -d 0
oracle 1347692 0.0 1.0 55736 28464 - A 08:05:01 0:00 oracleDBP1 (LOCAL=NO)
oracle 1409240 0.0 1.0 52868 25592 - A 08:05:13 0:00 oracleDBP1 (LOCAL=NO)
oracle 1413368 0.0 1.0 53220 21424 - A 15:04:45 0:00 oracleDBP1 (LOCAL=NO)
prepress 1319036 0.0 0.0 524 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman_amis -d1/softp/inout/adman/amis -m1/softp/inout/adman/amis_verwerkt -e1/softp/inout/adman/amis_error -b1* -x1* -p1/softp/oraprog/adman/proc/amiverw1/amiverw1.sh -s1/softp/oraprog/adman/proc/amiverw1/naamiverw1.sh -d2/softp/inout/adman/amisstop -m2/softp/inout/adman/amisstop_verwerkt -e2/softp/inout/adman/amisstop_error -b2* -x2* -p2/softp/oraprog/adman/proc/amiverw1/amiverw1.sh -s2/softp/oraprog/adman/proc/amiverw1/naamiverw1.sh -i300
oracle 1245264 Z 0:00 <defunct>
oracle 1302546 0.0 1.0 58040 26504 - A 07:06:15 0:00 oracleDBP1 (LOCAL=NO)
oracle 1315040 0.0 1.0 53680 21424 - A Jan 06 0:00 oracleDBFS1 (LOCAL=NO)
oracle 1499356 0.0 1.0 61392 34120 - A 08:12:41 0:00 oracleDBP1 (LOCAL=NO)
prepress 1523870 0.0 0.0 6704 104 - A Jan 09 0:00 /softp/oraprog/adman/proc/daem/lees_pipe -bpipe_agenda
prepress 1560800 0.0 0.0 516 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/catverw1 -m/softp/inout/adman/catverw1_verwerkt -e/softp/inout/adman/catverw1_error -b* -x* -p/softp/oraprog/adman/proc/catverw1/catverw1.sh -s/softp/oraprog/adman/proc/catverw1/nacatverw1.sh -i300
oracle 1490960 0.0 1.0 53500 21424 - A Jan 06 0:00 oracleDBFS1 (LOCAL=NO)
oracle 1421474 0.0 1.0 53648 22120 - A Jan 06 0:00 oracleDBFS1 (LOCAL=NO)
oracle 1433760 0.0 1.0 61384 34112 - A 08:13:38 0:00 oracleDBP1 (LOCAL=NO)
prepress 1441888 0.0 0.0 516 204 - A Jan 09 0:00 /softp/oraprog/adman/proc/scan/scan_adman -d/softp/inout/adman/bon -m/softp/inout/adman/bon_verwerkt -e/softp/inout/adman/bon_error -b* -xSES -p/softp/oraprog/adman/proc/bonverw1/bonverw1.sh -s/softp/oraprog/adman/proc/bonverw1/nabonverw1.sh -i300
oracle 1183794 0.0 1.0 58140 30600 - A 07:28:07 0:00 oracleDBP1 (LOCAL=NO)
oracle 839718 0.0 1.0 56288 29016 - A 08:08:30 0:00 oracleDBP1 (LOCAL=NO)
oracle 848026 0.0 1.0 52480 21548 - A Dec 19 0:00 ora_arc1_DBPS
oracle 884780 0.0 1.0 55952 28680 - A 08:08:05 0:00 oracleDBP1 (LOCAL=NO)
oracle 819264 0.0 1.0 61416 34144 - A 08:08:39 0:00 oracleDBP1 (LOCAL=NO)
root 753770 0.0 0.0 256 16 - A Nov 15 0:00 /usr/sbin/nfsd 3891
oracle 790764 0.0 1.0 56252 28972 - A 07:37:39 0:00 oracleDBP1 (LOCAL=NO)
root 807044 0.0 0.0 552 24 - A Dec 17 0:00 /usr/sbin/getty /dev/console
oracle 1085688 0.0 1.0 52764 25484 - A 07:48:52 0:00 oracleDBP1 (LOCAL=NO)
oracle 1102034 0.0 1.0 56272 29000 - A 08:06:33 0:00 oracleDBP1 (LOCAL=NO)
oracle 1138884 0.0 1.0 56204 28928 - A 08:04:43 0:00 oracleDBP1 (LOCAL=NO)
oracle 1028268 0.0 1.0 55884 28612 - A 07:57:42 0:00 oracleDBP1 (LOCAL=NO)
oracle 901172 0.0 1.0 56100 27492 - A 06:58:49 0:00 oracleDBP1 (LOCAL=NO)
oracle 905326 0.0 1.0 52648 25376 - A 08:15:02 0:00 oracleDBP1 (LOCAL=NO)
oracle 1019942 0.0 1.0 52512 21888 - A Dec 19 0:00 ora_reco_DBPS

greetz,

R.

 

[zaxxon]

Hi RMGBELGIUM,

hm ok, since startup; it's really strange. I have no idea, why vmstat shows that wrong average.

I can see following using up your CPU time, sorted by CPU usage:

USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 401564 57.7 1.0 68048 46476 - A Jan 06 3986:00 /softp/app/oracle/producti/ias902/jdk/bin/java -Xmx
256m -classpath :/softp/app/oracle/producti/ias902/ifs/cmsdk/custom_classes:/softp/app/oracle/producti/ias902/ifs/cmsdk
/samplecode:/softp/app/oracle/producti/ias902/ord/Annotator/lib/Annotator.jar:/softp/app/oracle/producti/ias902/ifs/cms
dk/settings:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/c
msdk_client.jar:/softp/app/oracle/producti/ias902/lib/servlet.jar:/softp/app/oracle/producti/ias902/lib/xmlparserv2.jar
:/softp/app/oracle/producti/ias902/jlib/javax-ssl-1_2.jar:/softp/app/oracle/producti/ias902/jlib/jssl-1_2.jar:/softp/ap
p/oracle/producti/ias902/jlib/swingall-1_1_1.jar:/softp/app/oracle/producti/ias902/jlib/jewt4.jar:/softp/app/oracle/pro
ducti/ias902/jlib/share.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/nis.jar:/softp/app/oracle/producti/ias902/j
lib/providerutil.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/aurora.zip:/softp/app/oracle/producti/ias902/ifs/c
msdk/lib/cmsdk.jar:/softp/app/oracle/producti/ias902/jdbc/lib/classes12.zip:/softp/app/oracle/producti/ias902/jdbc/lib/
nls_charset12.zip:/softp/app/oracle/producti/ias902/jlib/regexp.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/cms
dk.jar:/softp/app/oracle/producti/ias902/ifs/cmsdk/lib/inspect-4_1_6.jar:/softp/app/oracle/producti/ias902/classes:/sof
tp/app/oracle/producti/ias902/lib/vbjorb.jar:/softp/app/oracle/producti/ias902/jlib/help3.jar:/softp/app/oracle/product
i/ias902/jlib/help3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oracle_ice5.jar:/softp/app/oracle/producti/ias902/jl
ib/ewt3.jar:/softp/app/oracle/producti/ias902/jlib/ewt3-nls.jar:/softp/app/oracle/producti/ias902/jlib/oemlt-9_0_2.jar:
/softp/app/oracle/producti/ias902/jlib/emConfigInstall.jar:/softp/app/oracle/producti/ias902/jlib/repository.jar:/softp
/app/oracle/producti/ias902/jlib/ldap.jar:/softp/app/oracle/producti/ias902/dcm/lib/oc4j_deploy_tools.jar:/softp/app/or
acle/producti/ias902/dcm
root 8196 34.9 0.0 44 44 - A Nov 15 28516:03 wait
root 86136 2.1 0.0 684 288 - A Nov 15 1733:17 /usr/sbin/syncd 20
oracle 1896604 1.0 1.0 61912 34640 - A 08:03:49 0:07 oracleDBP1 (LOCAL=NO)
root 225412 0.7 0.0 1772 240 - A Nov 15 549:47 dtgreet
oracle 1290272 0.7 1.0 61428 34156 - A 08:12:30 0:01 oracleDBP1 (LOCAL=NO)
oracle 2080972 0.4 0.0 15896 1840 - A Dec 19 138:08 /softp/app/oracle/producti/920/bin/tnslsnr LISTENER -
inherit
oracle 1273862 0.4 1.0 55344 28064 - A 07:57:35 0:04 oracleDBP1 (LOCAL=NO)
oracle 1806460 0.3 1.0 52520 21796 - A Dec 21 74:01 ora_lgwr_DBP1
oracle 1511516 0.3 1.0 61620 34348 - A 08:02:43 0:02 oracleDBP1 (LOCAL=NO)
root 16392 0.2 0.0 44 40 - A Nov 15 170:26 lrud

I am not sure if this Java application is ok, using 57,7% of the CPU, you will know it, I guess.
34,9% are used up by a kernel wait, which is normally stated as idle time. The rest as you see by your other applications, which gives the correct idle time of 0%. An iostat at times when wait is high might be interessting too.
It could be, that the "wait" of the kernel is displayed wrong on your system.. Curious idea, I know but with a "vmstat 1 5" or something taken at same time like the "ps aux" might be interessting.

When you posted the other vmstats and the like, you had 0% wait time. So I am not sure what processes are using the CPU time then, but you can check it easy with "ps aux| head -1; ps aux

http://wwww|

sort -rnk 3,3| head -11|more". Change the fields from 3,3 to 4,4 to sort by memory.
If you see suspect processes, maybe check them in detail with a "svmon -P" or "filemon".

Looking for the usage of memory of your applications, each has a usage of 1,0% of mem, with none haveing less or more than 1,0%. Added all up resuls in 100% usage of your memory, as banders already said.

I am not sure if vmtune'ing helps a lot; maybe you might think about spending your box first some more RAM and then maybe an additional CPU.

Do your users have any problems with their application? Anything slow or laggy?

laters
zaxxon

 

[zaxxon]

The rest as you see by your other applications, which gives the correct idle time of 0%."

Forget this sentance - I changed part of text and made mistakes, sorry It would have been correct if you added up all your applications memory usage in percentage and included the 34,7% wait time to get to 100%, which would be incorrect, as the 34,7% belong to idle time.

I forgot to ask if the ML4 is installed correctly. When we update our AIX boxes, there are usually missing some filesets, even if there was no error in the installation process.
If you're not sure do a "instfix -i" to check if all 4 ML's are installed correctly. If there are messages like "Not all filesets of ML X have been found", do an "instfix -ciqk 5200-04_AIX_ML| grep ":-:", to check which filesets are not updated.

laters
zaxxon