vmstat fre value 0 and hdisk0 100% utilized 1

[arvibm]

Hi,

I am facing a performance problem on one of my 6H1 Server.
Oslevel-AIX 5.1+ ML-04,memory-4 GB.I have observed during day time for around 4-5 hours the server goes SLOW down and internal disks i.e. hdisk0 and hdisk1(mirrored) utlization is very high.I don't know why the internal harddisks get utilized 100% because we don't have any application filesystems on internal disks.pls find below attached output of vmstat,topas and iostat.Pls advise me how to resolve the issue and let me know if any more output is required for diagnosis.

===========================================================
Vmstat ouput:-
-------------
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
6 18 1252530 456 0 21 278 1471 2460 0 3864 6091 14622 10 15 3 72
1 11 1252531 475 0 31 283 2383 4122 0 3762 7160 15133 9 16 1 74
1 10 1252560 478 0 108 186 2215 3246 0 3402 7611 13510 16 14 2 68
2 4 1251078 1623 0 235 40 3068 5522 0 2656 9436 12100 19 19 5 58
1 9 1251342 44 0 66 287 5109 8528 0 2219 13446 16007 33 22 2 43
1 18 1249210 719 0 7 319 1357 1908 0 3201 6790 11745 7 15 0 78
1 12 1251570 0 0 29 404 3589 4764 0 3476 8278 12568 10 16 1 73
2 16 1251695 11 0 7 398 1864 2378 0 2813 5888 11023 8 14 0 78
0 21 1251748 0 0 9 299 1332 1882 0 2282 5505 10377 6 9 0 84
0 23 1251264 234 0 13 329 706 990 0 2277 7418 11763 8 10 0 81
2 20 1252450 0 0 20 314 1137 3840 0 1908 5282 8880 5 9 2 85
1 24 1252454 0 0 8 301 400 2498 0 1815 4992 10075 5 6 0 88
0 22 1252729 8 0 14 394 448 3030 0 1822 5251 10333 5 8 0 87
0 27 1253505 0 0 20 531 704 3390 0 1559 7302 10837 7 8 0 85
0 28 1253652 0 0 14 324 416 1860 0 1403 8343 10343 9 8 0 83
1 22 1252954 0 0 13 306 416 1792 0 1803 4833 8835 4 7 0 89
0 24 1253074 0 0 14 315 416 1772 0 1391 3475 7225 2 5 0 94
2 26 1252585 0 0 26 517 629 4115 0 1939 5563 8260 5 7 0 88
0 29 1252618 0 0 15 300 416 2097 0 1535 3685 7779 2 7 0 91
1 26 1252676 148 0 16 296 352 1678 0 1740 5819 7377 4 9 0 88
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 25 1252942 0 0 15 295 533 2670 0 1994 5675 11314 5 8 0 87
2 24 1252810 502 0 22 325 80 539 0 1753 6208 7971 4 9 0 88
1 19 1252346 408 0 79 300 477 1947 0 2425 8360 10863 11 11 0 78
3 30 1252451 430 0 120 226 528 2339 0 1767 8453 8497 12 10 0 78
1 18 1251034 1295 0 248 201 1005 3542 0 2101 12135 11709 21 15 0 64
1 13 1251678 418 0 267 125 841 1812 0 1897 11024 12306 32 15 3 50
2 14 1252301 544 0 171 231 891 1737 0 1703 37666 8952 46 22 0 32
3 13 1252238 2571 0 279 107 1616 5109 0 1791 17762 9310 47 16 1 36
1 14 1258187 495 0 188 179 2349 5914 0 1880 11776 8303 41 13 1 45
2 7 1256056 825 0 302 1 662 1297 0 2355 15158 10323 23 17 4 56
1 9 1250173 3611 0 252 0 0 0 0 1932 14920 6796 29 22 4 44
9 6 1249756 2780 0 224 0 0 0 0 1716 32147 6134 53 26 2 18
3 6 1247352 3903 0 222 1 0 0 0 1681 9233 8639 23 16 7 53
2 4 1245525 3902 0 248 0 0 0 0 1442 21237 16955 60 20 3 17
6 1 1244502 3278 0 71 0 0 0 0 1463 10557 13933 61 14 10 15
5 1 1244609 2874 0 57 0 0 0 0 1402 14875 15139 63 18 7 12
3 1 1244631 2035 0 107 0 0 0 0 1509 13219 14840 48 16 16 21
3 1 1244861 1036 0 45 0 0 0 0 1518 14107 14623 56 16 12 15
2 1 1246219 509 0 16 4 836 2189 0 1512 12837 15027 52 18 17 13
3 2 1244747 684 0 95 59 317 996 0 1706 15520 14524 59 16 10 16
===========================================================
Topas Output:-
--------------
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue
Sat Sep 27 16:46:24 2003 Interval: 2 Cswitch 17361 Readch 7364.6K
Syscall 15078 Writech 5531.1K
Kernel 11.6 |### | Reads 1616 Rawin 0
User 13.8 |#### | Writes 1572 Ttyout 0
Wait 64.5 |################## | Forks 2 Igets 0
Idle 10.0 |### | Execs 2 Namei 710
Runqueue 4.3 Dirblk 7
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 7.5
lo0 672.6 1743 1743 672.6 672.6
en0 20.2 182 153 18.5 22.0 PAGING MEMORY
en3 0.5 21 0 1.0 0.0 Faults 1055 Real,MB 4095
Steals 237 % Comp 41.5
Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 35 % Noncomp 59.3
hdisk1 99.9 1053.4 234 131.9 1974.9 PgspOut 244 % Client 0.5
hdisk0 73.2 590.7 141 159.9 1021.4 PageIn 164
hdisk5 72.2 5778.8 759 1039.4 10518.2 PageOut 1568 PAGING SPACE
Sios 1043 Size,MB 8192
Name PID CPU% PgSp Owner % Used 45.2
oracle 4653196 3.5 3.9 oracle NFS (calls/sec) % Free 54.7
f60webmx 1638468 2.6 10.5 oracle ServerV2 0
oracle 2875566 1.8 3.0 oracle ClientV2 0 Press:
aioserver 549046 1.4 0.0 root ServerV3 0 "h" for help
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 7.5
Sat Sep 27 16:46:26 2003 Interval: 2 Cswitch 8226 Readch 5862.7K
Syscall 20115 Writech 5105.3K
Kernel 9.7 |### | Reads 1295 Rawin 0
User 46.1 |############# | Writes 1587 Ttyout 0
Wait 34.5 |########## | Forks 0 Igets 0
Idle 9.6 |### | Execs 0 Namei 2541
Runqueue 2.0 Dirblk 7
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 4.0
lo0 212.1 1131 1131 212.1 212.1
en0 58.5 365 353 56.0 61.0 PAGING MEMORY
en3 1.0 32 0 2.0 0.0 Faults 825 Real,MB 4095
Steals 251 % Comp 41.3
Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 119 % Noncomp 59.3
hdisk1 72.0 602.6 131 508.3 697.0 PgspOut 79 % Client 0.5
hdisk5 65.5 5591.7 702 1052.7 10130.7 PageIn 252
hdisk0 50.0 356.5 84 452.3 260.7 PageOut 1346 PAGING SPACE
Sios 868 Size,MB 8192
Name PID CPU% PgSp Owner % Used 45.2
oracle 4284480 12.1 6.9 crptora NFS (calls/sec) % Free 54.7
oracle 2859242 8.1 6.2 crptora ServerV2 0
oracle 2998356 6.4 8.4 crptora ClientV2 0 Press:
oracle 4653196 3.4 3.9 oracle ServerV3 0 "h" for help
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 4.0
Sat Sep 27 16:46:28 2003 Interval: 2 Cswitch 7430 Readch 6409.6K
Syscall 32897 Writech 6255.5K
Kernel 7.8 |## | Reads 838 Rawin 0
User 68.8 |################### | Writes 1709 Ttyout 0
Wait 18.6 |##### | Forks 0 Igets 0
Idle 4.6 |# | Execs 0 Namei 30
Runqueue 3.5 Dirblk 0
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 1.0
en0 69.0 390 368 66.0 72.0
lo0 52.0 267 267 52.0 52.0 PAGING MEMORY
en3 0.5 30 0 1.0 0.0 Faults 448 Real,MB 4095
Steals 0 % Comp 41.2
Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 70 % Noncomp 59.4
hdisk5 78.5 6925.9 848 1264.3 12587.4 PgspOut 0 % Client 0.5
hdisk1 29.0 214.1 46 348.1 80.0 PageIn 226
hdisk0 14.5 150.0 30 220.1 80.0 PageOut 1550 PAGING SPACE
Sios 909 Size,MB 8192
Name PID CPU% PgSp Owner % Used 45.1
oracle 4284480 26.1 6.9 crptora NFS (calls/sec) % Free 54.8
oracle 2998356 21.6 8.4 crptora ServerV2 0
oracle 2859242 14.9 6.2 crptora ClientV2 0 Press:
oracle 4653196 3.9 3.9 oracle ServerV3 0 "h" for help
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 1.0
============================================================
iostat output:-
-----------------

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 723.5 3.9 5.6 32.8 57.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.0 1254.0 237.0 12 2496
hdisk0 69.0 628.0 131.0 12 1244
cd0 0.0 0.0 0.0 0 0
dac0 0.0 5734.0 696.0 3452 8016
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 74.5 5734.0 696.0 3452 8016

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 668.0 3.2 6.9 19.1 70.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.0 1224.0 234.0 56 2392
hdisk0 68.5 640.0 132.5 16 1264
cd0 0.0 0.0 0.0 0 0
dac0 0.0 5962.0 802.0 2036 9888
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 86.0 5962.0 802.0 2036 9888

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 675.0 9.4 8.2 16.5 65.9

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.0 1040.0 216.0 256 1824
hdisk0 66.5 586.0 121.5 188 984
cd0 0.0 0.0 0.0 0 0
dac0 0.0 4752.0 660.5 1872 7632
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 84.5 4752.0 660.5 1872 7632

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 730.6 3.2 6.6 0.0 90.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.4 1300.4 248.7 4 2600
hdisk0 74.9 681.1 142.3 20 1344
cd0 0.0 0.0 0.0 0 0
dac0 0.0 5890.6 700.1 3140 8656
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 79.4 5890.6 700.1 3140 8656

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 729.5 7.6 6.6 18.6 67.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.0 1090.0 233.0 24 2156
hdisk0 63.5 524.0 116.5 28 1020
cd0 0.0 0.0 0.0 0 0
dac0 0.0 7114.0 785.5 3476 10752
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 77.5 7114.0 785.5 3476 10752

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 685.1 4.5 5.6 33.6 56.3

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 99.9 1184.5 241.2 24 2348
hdisk0 68.4 595.3 131.3 12 1180
cd0 0.0 0.0 0.0 0 0
dac0 0.0 5483.1 627.2 3268 7712
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 84.9 5483.1 627.2 3268 7712

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 656.5 14.8 7.1 6.4 71.8

Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 100.0 1144.0 243.5 36 2252
hdisk0 74.0 632.0 142.5 36 1228
cd0 0.0 0.0 0.0 0 0
dac0 0.0 2160.0 300.5 2100 2220
dac1 0.0 0.0 0.0 0 0
hdisk2 0.0 0.0 0.0 0 0
hdisk3 0.0 0.0 0.0 0 0
hdisk4 0.0 0.0 0.0 0 0
hdisk5 42.5 2160.0 300.5 2100 2220

========================================================
# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging00 hdisk1 rootvg 4096MB 23 yes yes lv
hd6 hdisk0 rootvg 4096MB 23 yes yes lv
==========================================================
# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 128 256 2 open/syncd N/A
hd8 jfslog 1 2 2 open/syncd N/A
hd10opt jfs 1 2 2 open/syncd /opt
hd4 jfs 3 6 2 open/syncd /
hd2 jfs 93 186 2 open/syncd /usr
hd3 jfs 32 64 2 open/syncd /tmp
hd1 jfs 12 24 2 open/syncd /home
hd9var jfs 19 38 2 open/syncd /var
paging00 paging 128 128 1 open/syncd N/A


Regards

arvibm

 

[ademeter]

Hi,
some processes are "eating" your RAM memory and the system has no one free memory page. So, the VMM (virtual memory manager) is moving old pages to disk (hd6, paging00) and trying to get the reqested pages from paging space back to memory.

How to solve this?
1.) Check your applications (crptora) and the Oracle configuration for memory parameters
2.) Check the vmtune (AIX5.1) or vmo/schedo/ioo (AIX5.2). The default values should work, but for Oracle there is a recommendation to change the minperm=5-10%, maxclients=15-40%, maxperm=15-40%.
3.) Increase the RAM memory - buy more memory.

BTW, Why do you mirror the hd6 and you do not mirror the paging00?? You ould speed up your paging speed if:
1.) Use only one paging space per disk. AIX thinks it should stripe pages over all paging spaces and the disk head is doing more seeks then writes.
2.) Use one pgsp on hdisk0 (hd6) nonmirrored and the other on hdisk1 (paging00)

 

[arvibm]

Hi ademeter ,

Thanx for a quick response.here is my current vmtune parameters:-

# ./vmtune
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages maxrandwrt
98128 784208 2 32 480 512 65536 0

-M -w -k -c -b -B -u -l -d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket defps
838848 65536 16384 1 200 864 9 131072 1

-s -n -S -L -g -h
sync_release_ilock nokilluid v_pinshm lgpg_regions lgpg_size strict_maxperm
1 0 0 0 0 0

-t -j -J -z
maxclient j2_nPagesPer j2_maxRandomWrite j2_nRandomCluster
784208 32 0 0

-Z -q -Q -y
j2_nBufferPer j2_minPageReadAhead j2_maxPageReadAhead memory_affinity
512 2 8 0

-V -i
num_spec_dataseg spec_dataseg_int
0 512

PTA balance threshold percentage = 50.0%

number of valid memory pages = 1048560 maxperm=79.9% of real memory
maximum pinable=80.0% of real memory minperm=10.0% of real memory
number of file memory pages = 561910 numperm=57.3% of real memory
number of compressed memory pages = 0 compressed=0.0% of real memory
number of client memory pages = 0 numclient=0.0% of real memory
# of remote pgs sched-pageout = 0 maxclient=79.9% of real memory

which values i have to change to improve performance.i have created additional paging space on second harddisk i.e. hdisk1 and the paging space is not mirrored.i don't know how to unmirror default paging space.to unmirror the paging space i have to first deactivate it and then i have to recreate it.Is there any method or command through which i can unmirror the default paging space on the fly.

Regards

arvibm

 

[ademeter]

Hi,
I see there was some playing with the vmtune.
Param: YourVal DefaultVal
minpgahead 2 2
maxpgahead 32 8
minfree 480 120
maxfree 512 128
minperm 10% 20%
maxperm 80% 80%
maxclient 80% 80%

I expect from these modifications that you are running long sequential reads on the data.

I would change also both the maxperm and maxclient to something lower. maxperm=30, maxclient=30. Change it from 80 to 30 in steps (80-60-50-45-40-35-30).

I also have experienced a "strange" behaviour on AIX5.2 ML01 that is similar to the level of AIX5.1 ML04 that after I changed the minfree and maxfree in the way I used to do it on AIX 4.3.3 the VMM went "crazy" when running Oracle RDBMS v8/32bit. I got strange pgsp activity. I am also looking for the explanation of my experience. I had to change the vmtune back to default and then I changed only the minper/maxperm/maxclient and the system started to behave correctly.

About paging space mirroring:
If you mirror the underlying logical volume, the paging space mirorring is done and vice versa.
smit lv -> Set Characteristics of a Logical Volume ->
-> Add a Copy ...
-> Remove a Copy ...

 

[arvibm]

Hi ademeter ,


Thanx, ur tips helped me alot.today i have not faced the problem after unmirroring default paging space.
i have added the following entries in /etc/rc file
/usr/samples/kernel/vmtune -p 15 -P 30 -R 32 -f 480 -F 512 -s 1
/usr/samples/kernel/vmtune -b 200 -B 800

the vmtune values get activated automatically at the time of boot.

pls let me know if i have enetered any wrong values for vmtune.

do u have any command to identify the particular process which is consuming maximum amount of memory and slowing down the system.
If u will find any explanation of strange behaviour of systems after applying ML-04 then let me know.


Thanx and Regards

arvibm

 

[arvibm]

Hi,


Anyone knows the reason of strange paging activity with aix 5.1 + ML-04.


Regards

arvibm

 

[unixtechie]

If the rest of your rootvg is mirrored except the paging space and you lose your disk that contains the paging space and your system tries to access that paging space - BOOM!!!

 

[arvibm]

Hi Unixtechie,

If we mirror the paging space the I/O gets increases and the performance slows down.what is the solution for the problem which u have mentioned above.if one of my disk fails do i have to recreate the paging space on new disk before starting any application?do u have any command to trace out any particular process consuming maximum amount of memory?Any Reason for this strange behaviour of paging activity after applying ML-04 on AIX 5.1?

Regards

arvibm

 

[arvibm]

Hi Admeter/unixtechie,

i am waiting for ur reply.


Regards

arvibm

 

[ademeter]

Hi,
The values looks OK, but I cannot test them now. I am out of office.
I have experienced the same strange behaviour of AIX5.2 ML01, that is the coresponding level to AIX5.1 ML04, too. I am also looking for explanation of what has been changed in AIX5.2 ML01/AIX5.1 ML04. Can anyone help?