AIX 5.3 tuning question

[aix20080423]

Sorry for the long message:

######### Server Info ############

System Model: IBM,9117-MMA
Processor Type: PowerPC_POWER6
Number Of Processors: 4
Processor Clock Speed: 4704 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 greyhound
Memory Size: 16384 MB
Good Memory Size: 16384 MB

# oslevel -s
5300-06-02-0727

####### Changes we have been made to the server ######

ioo -p -o j2_dynamicBufferPreallocation=256
ioo -p -o j2_maxPageReadAhead=256
ioo -p -o j2_maxRandomWrite=32
ioo -p -o j2_nBufferPerPagerDevice=1024
ioo -p -o j2_nPagesPerWriteBehindCluster=64
ioo -p -o maxpgahead=16
ioo -p -o maxrandwrt=32
ioo -p -o numclust=8
ioo -p -o numfsbufs=2048
ioo -p -o pv_min_pbuf=2048
vmo -p -o lru_file_repage=0
vmo -p -o maxclient%=90
vmo -p -o maxperm%=90
vmo -p -o minperm%=5

MinServers - LEAST(#CPUs=8, 10)=8
MaxServers - (10*(#Disks=45)/#CPUs=8)=60

I configured page space as follows:

# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging00 hdisk3 pagingvg 8192MB 11 yes yes lv
paging01 hdisk2 pagingvg 8192MB 11 yes yes lv
hd6 hdisk1 rootvg 8192MB 10 yes yes lv

I also deactivated/activated all three page spaces.

################ Result ########################

Output from vmstat -v before the changes: (Uptime to this point ~180 days)

# vmstat -v | grep block
Mon Apr 7 19:13:00 EDT 2008
11624496 pending disk I/Os blocked with no pbuf
85953253 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
41362018 external pager filesystem I/Os blocked with no fsbuf

After changing the parameters and rebooted the server: (Uptime to this point 10 days)

# vmstat -v | grep block
Tue Apr 22 11:42:13 EDT 2008
1267 pending disk I/Os blocked with no pbuf
13513203 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
159202 external pager filesystem I/Os blocked with no fsbuf

As you can see, the "paging space I/Os blocked with no psbuf" is still increasing a lot. I'd like to know how to fix this.

Thank you!!!

Wei

 

[zaxxon]

What does "vmstat 1 20" and a "svmon -G" show when there is traffic on the box?
What does "svmon -P | grep -p Pid" say in the colum of Pgsp?
What kind of application is there running?
What does

svmon -P| grep -p Pid| grep -vE "^---|^$|Pid"| sort -rn -k 4,4| head -20

show?
And "vmo -x| grep lru" ?
Maybe increase numpsbufs by factor 2 - seemed to help with the blocked, but I am curious what kind of application is running there. Let's see the output of above commands or else you might consider to get more RAM.

laters
zaxxon

 

[aix20080423]

Thank you for the reply.

Here are the answers to your questions:

# vmstat 1 20

System configuration: lcpu=8 mem=16384MB

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 10 5387143 12567 0 1229 1215 31242 858344 0 3125 8890 10374 9 15 34 43
2 8 5387146 12799 0 1237 1268 23267 451644 1 2787 16098 16639 16 12 25 47
6 15 5387147 12690 0 1438 1365 18645 421601 2 3744 80561 62476 62 10 3 24
10 15 5387147 12462 0 1394 1091 18418 359688 0 3554 78019 61617 61 10 3 25
13 12 5387146 12648 0 1008 1144 20574 280842 0 2764 31393 24686 23 11 28 38
8 6 5387149 12757 0 1014 787 32204 394290 0 2706 15765 11364 10 15 34 41
7 10 5387144 13089 0 1274 1182 21401 596433 4 2866 36286 21253 23 11 29 36
4 16 5387145 12847 0 1321 1072 19548 541767 3 3676 83908 59858 64 11 3 23
7 12 5387144 12560 0 717 529 18196 612121 1 2799 88865 49529 55 10 10 25
2 3 5387143 12508 0 573 612 24285 502036 2 1682 16098 11762 12 12 41 35
0 6 5387146 12485 0 676 526 26766 1654604 1 1711 14585 9483 10 15 43 32
4 16 5393549 11970 0 401 5422 25740 669341 2 2639 10288 9711 9 13 31 46
10 11 5393776 12784 0 524 1472 12419 297682 0 2753 52774 28883 37 7 1 55
10 15 5393945 12473 0 526 578 19592 697213 0 2982 85080 54168 64 10 2 24
0 9 5387157 12995 0 704 388 12731 310770 0 2144 43777 30291 34 9 22 36
1 5 5387158 12476 0 554 78 23545 635804 2 1245 19030 9386 10 12 44 34
0 7 5387155 12672 0 589 0 25048 402879 0 1453 15908 9915 11 12 42 35
9 5 5387155 12653 0 685 0 20664 276262 1 2441 66095 36515 48 10 18 24
2 19 5387156 12470 0 535 0 18507 278081 3 2573 83352 50726 62 9 4 25
14 13 5387158 12579 0 203 0 23192 453023 1 1768 42228 28037 30 12 28 30

# svmon -G
size inuse free pin virtual
memory 4194304 4181479 12825 1961553 5389857
pg space 6291456 1783159

work pers clnt
pin 1961553 0 0
in use 3962918 0 218561

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 4056151 1747335 1880497 5247873
m 64 KB - 7833 2239 5066 8874

# svmon -P| grep -p Pid| grep -vE "^---|^$|Pid"| sort -rn -k 4,4| head -20
2200004 oracle 1434088 1421364 2975 1429709 Y N N
2187676 oracle 1434316 1421364 3100 1430004 Y N N
2154964 oracle 1434137 1421364 3092 1429864 Y N N
2040262 oracle 1434570 1421364 2986 1430197 Y N N
2023840 oracle 1641944 1421364 3174 1430953 Y N N
2003124 oracle 1522430 1421364 16854 1452718 Y N N
1994928 oracle 1533560 1421364 8619 1440523 Y N N
1978826 oracle 1448115 1421364 3222 1430258 Y N N
1958344 oracle 1448519 1421364 2997 1430410 Y N N
1949872 oracle 1533987 1421364 9759 1442783 Y N N
1937664 oracle 1551532 1421364 4985 1470174 Y N N
1929564 oracle 1640899 1421364 3439 1430254 Y N N
1913158 oracle 1447825 1421364 3071 1429790 Y N N
1880346 oracle 1434036 1421364 2903 1429571 Y N N
1847444 oracle 1523671 1421364 15985 1453563 Y N N
1843230 oracle 1514588 1421364 10318 1440512 Y N N
1827026 oracle 1554152 1421364 6943 1474286 Y N N
1716300 oracle 1434079 1421364 2815 1429515 Y N N
1659038 oracle 1532065 1421364 6423 1470174 Y N N
1634458 oracle 1434505 1421364 3677 1430767 Y N N

# vmo -x | grep lru
lru_file_repage,0,1,0,0,1,boolean,D,
lru_poll_interval,10,10,10,0,60000,milliseconds,D,
lrubucket,131072,131072,131072,65536,4194304,4KB pages,D,

The application we are running is database warehouse, ETL.

We do consider adding another 16 GB memory, it will be couple of weeks from now.

Thanks again!

 

[aix20080423]

One more piece of info (vmstat -v | grep block)

you can see how much the counter increased during the time this was running (30 minutes).


Wed Apr 23 12:01:00 EDT 2008
1267 pending disk I/Os blocked with no pbuf
15180098 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
168121 external pager filesystem I/Os blocked with no fsbuf



Wed Apr 23 12:33:34 EDT 2008
1267 pending disk I/Os blocked with no pbuf
15243929 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
168121 external pager filesystem I/Os blocked with no fsbuf


Thanks!

 

[zaxxon]

I have no idea what helpful parameter can be tuned VMM wise which you isn't already. I guess there is no other way than getting more RAM.

laters
zaxxon

 

[zaxxon]

Just curious to know what happened so far - did you find another way to tune or did you get your RAM yet?

laters
zaxxon

 

[aix20080423]

Just got the additional 16 GB memory, will see how it improves the performance...

 

[zaxxon]

I am no Oracle DBA but could it be that the size of the SGA might have been a bit too large compared to the installed RAM? I bet there are some recommendations how to size it. So I guess it should be not more than about 2/3 to 3/4 of your built in RAM, if it's running somewhat alone on that machine.

Just found this http://www.dba-oracle.com/t_sga_sizing.htm
He talks about "sga_max_size" and "sga_target" - maybe check these too.

laters
zaxxon