Performance problems of a p690 LPAR

[RiazAhamed]

Hi,

I need Urgent help in resolving performance issues on a LPAR (pSeries p690) running AIX 5L ver 5.2, ML4, JFS2 filesystems and DB2.

This LPAR has 20 processors; 54 GB Memory and consistently it is RUNNING SLOW for the past few months.

CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 55296 MB
Processor Clock Speed: 1100 MHz

# vmstat -v
14155776 memory pages
13645476 lruable pages
42967 free pages
4 memory pools
857065 pinned pages
80.1 maxpin percentage
3.0 minperm percentage
10.0 maxperm percentage
37.0 numperm percentage
5056149 file pages
0.0 compressed percentage
0 compressed pages
37.0 numclient percentage
10.0 maxclient percentage
5053589 client pages
0 remote pageouts scheduled
237 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
27442 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
13160 external pager filesystem I/Os blocked with no fsbuf

And...

# vmstat 5
System Configuration: lcpu=20 mem=55296MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
7 21 9048650 22517 0 0 0 4167 9888 0 8990 68283 41520 17 13 20 49
12 47 9048720 33556 0 0 0 23692 38067 0 15020 218167 138203 41 19 0 39
16 48 9049104 33959 0 0 0 19885 35170 0 14733 176964 134771 46 20 0 33
12 48 9048896 19801 0 0 0 18607 27337 0 14722 140453 144841 40 21 1 39
16 49 9048945 26710 0 0 0 22837 34880 0 15164 217362 137209 43 20 1 37
14 49 9049036 39515 0 0 0 23348 37338 0 14505 142268 137738 42 20 0 38
16 43 9049113 29740 0 0 0 19641 26499 0 14588 133223 138848 42 19 0 38
14 51 9049812 24093 0 0 0 19282 32294 0 14746 179873 139821 39 21 1 39
8 54 9049365 34704 0 0 0 22309 29113 0 15491 223155 139550 40 19 1 40
11 48 9049466 26799 0 0 0 17192 27072 0 15425 261954 140777 39 20 1 41
8 42 9049532 38821 0 0 0 11794 20446 0 7481 128557 123959 23 23 1 53
9 47 9049636 19601 0 0 0 14265 25433 0 14786 139700 140813 37 18 1 44
15 49 9050206 22118 0 0 0 20124 40855 0 15384 142294 140295 38 20 1 42
14 50 9049851 19408 0 0 0 18509 43938 0 15868 221585 141426 40 19 1 41
11 49 9049984 30651 0 0 0 24060 45054 0 14616 180971 139272 39 19 1 41
13 52 9050086 32471 0 0 0 23127 32627 0 14488 142934 139406 37 20 1 42
9 48 9050319 19615 0 0 0 20013 28600 0 14762 222597 139666 39 20 1 41
11 51 9050352 33204 0 0 0 24616 32932 0 13888 143170 139895 40 20 1 40
^C#

Please advise me to make this system perform better.

Thanks.

Riaz Ahamed

 

[hirschaj]

ML04 of 5.2 has multiple AIO and paging algorithm issues. I'm surprised your server has not crashed yet (as did some where I work). Get to ML05 ASAP.


Jim Hirschauer

http://www.aixexpert.com

 

[RiazAhamed]

Hi hirschaj,

Thanks for your reply.

Can I go for installing ML6 directly? Or you want me to work on implementing ML5 first?

Please advise.

 

[hirschaj]

ML06 is fairly new so that is your call. I know we are running ML05 without issues now. But you can go straight to ML06 if you want to, no need to go ML05 then ML06.


Jim Hirschauer

http://www.aixexpert.com

 

[AIX5L]

Hi

How is your Database Disk Layout ?
There are around 40 Percent of I/O Waits...

What kind off Storage is assigned to your LPAR.
How many Adapters are attached to your Storage.
Where is your Paging Space ?
Did you are working with aio ? ( How is the setup )

 

[MoshiachNow]

Also,to add to AIX5L words:

Since memory does not seem to be an issue - Networking and storage should be also inspected here.

Please attach the output of "iostat -a 2 5" and at least 1 topas screen.
Also,I would run couple of network performance tests - to verify things are tuned there.

Long live king Moshiach !

http://www.7for70.com/

 

[RiazAhamed]

Hi,
Good morning & Thanks for your reply.

I understand this posting is several screens, I can email the output to you, if you want.

AIX5L, we are using IBM ESS Shark in our environment for all the LPARs.

We have couple of Gigabit ethernet adapters & Fiber channel scsi adapters installed for this LPAR.

We are using IBM ESS for the applications/databases running on this LPAR, therefore the paging space is distributed to 37 disks (36 in IBM ESS) including "hd6" (Internal scsi).

# lsps -s
Total Paging Space Percent Used
97280MB 1%
#

There are 2 internal scsi disks:
hdisk0 002680babd93dcf6 rootvg active
hdisk1 002680ba2d04d9d1 rootvg active

And lot many disks from the IBM ESS shark:

# lspv | grep active | wc -l
52

Yes, we do have AIO properly setup in all LPARs. Please note that, this is the only LPAR having performance problem.

============================================================

For MoshiachNow;

# uname -a
AIX ax-ohp-db01 2 5 002680BA4C00
# oslevel -r
5200-04

The topas screen output:

Topas Monitor for host: ax-ohp-db01 EVENTS/QUEUES FILE/TTY
Thu Jun 16 10:18:06 2005 Interval: 2 Cswitch 176.1K Readch 59.8M
Syscall 666.8K Writech 794.0K
Kernel 21.6 |####### | Reads 7777 Rawin 0
User 54.6 |################ | Writes 180 Ttyout 424
Wait 23.6 |####### | Forks 9 Igets 3
Idle 0.3 |# | Execs 8 Namei 250
Runqueue 20.5 Dirblk 1491
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 63.5
en7 234.0 363.0 221.0 446.8 21.3
lo0 0.3 6.0 6.0 0.3 0.3 PAGING MEMORY
Faults 8463 Real,MB 55295
Disk Busy% KBPS TPS KB-Read KB-Writ Steals 15532 % Comp 66.6
hdisk344 99.0 3984.0 172.5 7968.0 0.0 PgspIn 0 % Noncomp 34.1
hdisk33 98.5 3364.0 170.0 6728.0 0.0 PgspOut 0 % Client 34.1
hdisk290 98.0 3056.0 133.5 6112.0 0.0 PageIn 25455
hdisk340 97.0 5608.0 228.5 11216.0 0.0 PageOut 68 PAGING SPACE
hdisk44 96.5 2962.0 127.5 5924.0 0.0 Sios 25476 Size,MB 97280
hdisk26 96.5 4308.0 233.0 8616.0 0.0 % Used 1.0
hdisk28 96.0 4958.0 225.0 9916.0 0.0 NFS (calls/sec) % Free 98.9
hdisk39 95.5 3886.0 243.0 7772.0 0.0 ServerV2 0
hdisk37 94.5 4154.0 268.5 8308.0 0.0 ClientV2 0 Press:
hdisk30 93.5 3524.0 151.5 7048.0 0.0 ServerV3 0 "h" for help
hdisk244 93.5 5522.0 294.5 11044.0 0.0 ClientV3 5 "q" to quit
hdisk298 93.0 5176.0 267.5 10352.0 0.0

Name PID CPU% PgSp Owner
db2sysc 7135448 5.0 80.1 wfdm07dp
db2sysc 14819504 4.8 5.7 wfdm07dp
db2sysc 14545138 4.3 8.7 wfdm07dp
db2sysc 3252276 2.7 39.8 wfdm07dp
db2sysc 3338334 2.0 43.3 wfdm07dp
db2sysc 3489960 1.9 42.2 wfdm07dp
db2sysc 2773066 1.5 42.2 wfdm07dp
lrud 143430 1.5 0.1 root
db2sysc 2814046 1.5 81.5 wfdm07dp
db2sysc 1417396 1.5 79.6 wfdm07dp
db2sysc 2674714 1.4 77.1 wfdm07dp
db2sysc 1626138 1.4 37.6 wfdm07dp
db2sysc 3457176 1.4 78.5 wfdm07dp
Exiting
#

========================================================

The output of "iostat -a 2 5" is huge and I am pasting the output here;

System configuration: lcpu=20 disk=196

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.4 70.8 17.9 14.8 15.0 52.3

Adapter: Kbps tps Kb_read Kb_wrtn
scsi2 37.9 3.2 373931 8988822

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1_Path0 1.8 37.9 3.2 373931 8988822

Adapter: Kbps tps Kb_read Kb_wrtn
scsi0 34.3 3.6 1655113 6813502

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0_Path0 2.0 34.3 3.6 1655113 6813502

Adapter: Kbps tps Kb_read Kb_wrtn
fcs0 2239.7 3865.7 4294967295 552615279

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
vpath33_hdisk212 0.0 0.0 0.0 0 0
vpath33_hdisk220 0.0 0.0 0.0 0 0
vpath0_hdisk24 5.7 444.8 10.9 109417816 325992
vpath1_hdisk25 0.9 116.3 3.3 12616174 16084428
vpath11_hdisk46 1.0 41.7 2.1 10280464 0
vpath2_hdisk26 46.6 2431.2 123.1 599882174 0
vpath29_hdisk206 0.0 0.1 0.0 23693 28
vpath20_hdisk349 0.0 0.3 0.0 42221 38224
vpath21_hdisk350 7.3 181.6 38.7 11991869 32825348
vpath22_hdisk351 1.4 17.3 3.8 1822398 2440328
vpath46_hdisk417 7.0 431.4 27.7 89693766 16739728
vpath49_hdisk423 0.2 4.0 0.6 353725 631548
vpath3_hdisk27 40.9 2253.8 103.1 556092181 0
vpath12_hdisk47 0.0 0.0 0.0 11 0
vpath30_hdisk93 0.0 0.0 0.0 10 0
vpath31_hdisk210 0.0 0.0 0.0 30 0
vpath32_hdisk211 0.0 0.0 0.0 12 0
vpath34_hdisk213 0.0 2.2 0.0 528646 9300
vpath35_hdisk214 0.0 0.3 0.0 15864 46552
vpath36_hdisk216 0.1 1.4 0.3 1057 333844
vpath4_hdisk28 37.3 1897.3 94.2 468135071 0
vpath5_hdisk29 0.0 0.0 0.0 12 0
vpath17_hdisk98 0.0 0.2 0.0 61028 0
vpath18_hdisk99 0.7 29.1 4.6 5580125 1588396
vpath19_hdisk100 0.0 0.0 0.0 11 0
vpath45_hdisk101 4.8 115.1 20.1 12431468 15965527
vpath6_hdisk30 50.0 2758.4 133.2 680592036 0
vpath7_hdisk31 0.0 0.0 0.0 9 0
vpath8_hdisk32 25.7 1356.2 62.7 334271264 345720
vpath13_hdisk48 0.0 0.0 0.0 11 0
vpath23_hdisk352 1.0 13.3 3.1 1331504 1941936
vpath24_hdisk353 6.7 614.4 8.3 151292011 302940
vpath27_hdisk397 2.9 53.8 10.7 5139329 8140148
vpath28_hdisk398 44.2 1074.2 160.8 217511448 47539172
vpath37_hdisk399 48.0 1199.6 191.9 267998932 27996280
vpath38_hdisk400 50.1 1235.5 195.0 276970209 27880424
vpath39_hdisk401 52.6 1366.5 216.5 325181420 11992088
vpath47_hdisk418 5.0 253.8 22.0 48781955 13841787
vpath9_hdisk33 48.4 2677.9 129.3 660235288 498328
vpath14_hdisk49 0.0 0.0 0.0 1057 0
vpath15_hdisk50 0.0 0.0 0.0 1061 0
vpath10_hdisk34 0.0 0.1 0.0 22631 4
vpath16_hdisk51 0.8 36.2 3.7 6764554 2173308
vpath25_hdisk354 4.5 502.5 6.7 123844793 138592
vpath26_hdisk355 0.1 12.1 0.2 2578077 404640
vpath40_hdisk402 25.2 1007.9 167.3 235292397 13388516
vpath41_hdisk403 17.7 633.7 127.0 145991110 10376808
vpath42_hdisk404 9.1 231.3 50.1 47261511 9820960
vpath43_hdisk405 2.6 61.9 10.9 9148960 6124636
vpath44_hdisk406 0.0 0.0 0.0 2106 0
vpath48_hdisk419 0.9 64.2 3.3 9107520 6730856
vpath0_hdisk35 5.9 443.4 10.9 109072675 325280
vpath1_hdisk36 1.0 116.5 3.2 12579710 16177444
vpath11_hdisk52 1.0 41.4 2.1 10218677 0
vpath2_hdisk37 46.9 2427.9 123.0 599066424 0
vpath29_hdisk207 0.0 0.1 0.0 34599 32
vpath20_hdisk356 0.0 0.3 0.0 43007 38664
vpath21_hdisk357 7.4 181.3 38.6 12052552 32691756
vpath22_hdisk358 1.4 17.1 3.8 1791450 2429915
vpath46_hdisk420 7.0 431.9 27.7 89839119 16735495
vpath49_hdisk424 0.2 4.0 0.6 343430 633700
vpath3_hdisk38 41.2 2255.2 103.3 556443271 0
vpath12_hdisk53 0.0 0.0 0.0 2108 0
vpath30_hdisk217 0.0 0.0 0.0 1060 0
vpath31_hdisk218 0.0 0.0 0.0 38 4
vpath32_hdisk219 0.0 0.0 0.0 8 0
vpath34_hdisk221 0.0 2.1 0.0 521169 9052
vpath35_hdisk222 0.0 0.3 0.0 22590 40560
vpath36_hdisk224 0.1 1.4 0.3 21 335748
vpath4_hdisk39 37.7 1895.5 94.1 467699960 0
vpath5_hdisk40 0.0 0.0 0.0 10 0
vpath17_hdisk115 0.0 0.4 0.0 92554 0
vpath18_hdisk116 0.8 29.0 4.6 5559045 1590479
vpath19_hdisk117 0.0 0.0 0.0 8885 0
vpath45_hdisk118 4.8 114.9 20.1 12438525 15922113
vpath6_hdisk41 50.3 2755.3 133.2 679829672 0
vpath7_hdisk42 0.0 0.0 0.0 1057 0
vpath8_hdisk43 26.1 1356.5 62.7 334353711 344364
vpath13_hdisk54 0.0 0.0 0.0 1060 0
vpath23_hdisk359 1.0 13.3 3.1 1348025 1938952
vpath24_hdisk360 6.8 613.5 8.3 151062957 298951
vpath27_hdisk407 3.0 53.6 10.6 5124578 8094624
vpath28_hdisk408 44.5 1073.3 160.3 217452301 47364476
vpath37_hdisk409 48.3 1197.5 191.2 267518277 27958944
vpath38_hdisk410 50.5 1233.1 194.4 276447855 27807124
vpath39_hdisk411 52.9 1363.1 215.7 324385422 11937324
vpath47_hdisk421 5.1 254.2 22.0 48889118 13822188
vpath9_hdisk44 48.7 2675.7 129.2 659706366 496972
vpath14_hdisk55 0.0 0.0 0.0 10 0
vpath15_hdisk56 0.0 0.0 0.0 1056 0
vpath10_hdisk45 0.0 0.1 0.0 16332 8
vpath16_hdisk57 0.8 36.0 3.7 6723409 2163560
vpath25_hdisk361 4.6 502.1 6.7 123760753 133560
vpath26_hdisk362 0.1 12.1 0.2 2585450 392612
vpath40_hdisk412 25.6 1003.9 166.4 234366631 13326524
vpath41_hdisk413 18.2 630.4 126.1 145233971 10303800
vpath42_hdisk414 9.4 230.8 50.0 47147151 9795352
vpath43_hdisk415 2.6 61.9 10.9 9176464 6106048
vpath44_hdisk416 0.0 0.0 0.0 10 0
vpath48_hdisk422 0.9 63.7 3.3 8982737 6733268

Adapter: Kbps tps Kb_read Kb_wrtn
fcs1 1984.8 3837.6 4294967295 489738474

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
vpath33_hdisk251 0.0 0.0 0.0 0 0
vpath51_hdisk426 0.0 0.0 0.0 0 0
vpath33_hdisk305 0.0 0.0 0.0 0 0
vpath51_hdisk428 0.0 0.0 0.0 0 0
vpath0_hdisk241 6.0 450.2 11.7 110747957 331124
vpath1_hdisk242 0.9 117.1 3.2 12693236 16209232
vpath11_hdisk243 1.0 41.8 2.1 10303558 0
vpath2_hdisk244 47.0 2439.5 124.0 601915423 0
vpath29_hdisk245 0.0 0.2 0.0 49140 44
vpath20_hdisk363 0.0 0.3 0.0 39118 40292
vpath21_hdisk364 7.4 186.8 39.6 12617847 33461884
vpath22_hdisk365 1.8 39.0 4.2 7038038 2588116
vpath3_hdisk246 42.0 2298.8 107.3 567212071 0
vpath12_hdisk247 0.0 0.0 0.0 16 0
vpath30_hdisk248 0.0 0.0 0.0 1063 0
vpath31_hdisk249 0.0 0.0 0.0 48 4
vpath32_hdisk250 0.0 0.0 0.0 11 0
vpath34_hdisk252 0.0 2.6 0.0 619964 9320
vpath35_hdisk253 0.0 0.3 0.0 18170 49368
vpath36_hdisk255 0.1 1.4 0.3 1065 338004
vpath4_hdisk256 37.7 1907.0 95.0 470541060 0
vpath5_hdisk257 0.0 0.0 0.0 1062 0
vpath17_hdisk262 0.0 0.5 0.0 115378 0
vpath18_hdisk263 0.7 29.0 4.6 5584673 1580380
vpath19_hdisk264 0.0 0.0 0.0 14 0
vpath6_hdisk286 51.3 2809.8 137.9 693283753
0
vpath7_hdisk287 0.0 0.0 0.0 15 0
vpath8_hdisk288 26.1 1364.8 63.4 336396384 342408
vpath13_hdisk289 0.0 0.0 0.0 13 0
vpath23_hdisk366 1.3 27.0 3.4 4577311 2090291
vpath24_hdisk367 6.7 621.5 8.4 153070430 283216
vpath27_hdisk377 3.1 55.6 11.1 5186069 8529560
vpath28_hdisk378 45.9 1125.2 168.9 225711668 51912840
vpath37_hdisk379 49.7 1249.7 200.3 275606477 32734820
vpath38_hdisk380 51.8 1282.3 202.6 284489536 31893844
vpath39_hdisk381 54.2 1416.3 225.1 332189338 17266780
vpath50_hdisk425 0.1 1.9 0.2 414578 58684
vpath9_hdisk290 49.4 2719.6 133.0 670528703 506028
vpath14_hdisk291 0.0 0.0 0.0 17 0
vpath15_hdisk292 0.0 0.0 0.0 18 0
vpath10_hdisk293 0.0 0.1 0.0 15253 8
vpath16_hdisk294 0.8 40.3 3.9 7653154 2279196
vpath25_hdisk368 4.6 510.3 6.8 125763951 137076
vpath26_hdisk369 0.1 12.2 0.2 2612538 406044
vpath40_hdisk382 25.6 1041.2 172.3 241528034 15371276
vpath41_hdisk383 17.9 646.7 129.0 149120320 10457512
vpath42_hdisk384 9.2 234.1 50.7 47885438 9877556
vpath43_hdisk385 2.6 62.9 11.1 9272360 6253632
vpath44_hdisk386 0.0 0.0 0.0 13 0
vpath0_hdisk295 6.2 449.8 11.7 110654304 324680
vpath1_hdisk296 1.0 115.9 3.2 12463075 16136028
vpath11_hdisk297 1.0 41.7 2.1 10300568 0
vpath2_hdisk298 47.3 2434.8 123.8 600752386 0
vpath29_hdisk299 0.0 0.2 0.0 40397 28
vpath20_hdisk370 0.0 0.3 0.0 38656 37032
vpath21_hdisk371 7.5 186.5 39.5 12593301 33428656
vpath22_hdisk372 1.8 38.8 4.2 7004143 2577568
vpath3_hdisk300 42.3 2297.0 107.1 566750228 0
vpath12_hdisk301 0.0 0.0 0.0 10 0
vpath30_hdisk302 0.0 0.0 0.0 11 0
vpath31_hdisk303 0.0 0.0 0.0 2164 0
vpath32_hdisk304 0.0 0.0 0.0 2106 0
vpath34_hdisk306 0.0 2.4 0.0 590841 9052
vpath35_hdisk307 0.0 0.3 0.0 18065 47748
vpath36_hdisk309 0.1 1.3 0.3 14 332852
vpath4_hdisk310 38.1 1903.6 94.9 469698697 0
vpath5_hdisk311 0.0 0.0 0.0 1061 0
vpath17_hdisk316 0.0 0.5 0.0 124962 0
vpath18_hdisk317 0.8 29.3 4.6 5603414 1636768
vpath19_hdisk318 0.0 0.0 0.0 11 0
vpath6_hdisk340 51.5 2800.8 137.6 691072414 0
vpath7_hdisk341 0.0 0.0 0.0 1065 0
vpath8_hdisk342 26.5 1363.3 63.3 336033312 344376
vpath13_hdisk343 0.0 0.0 0.0 1062 0
vpath23_hdisk373 1.3 27.0 3.4 4580254 2080740
vpath24_hdisk374 6.8 620.2 8.4 152732539 295112
vpath27_hdisk387 3.1 55.4 11.1 5170345 8508756
vpath28_hdisk388 46.3 1124.6 168.7 225644026 51846780
vpath37_hdisk389 50.2 1248.2 199.9 275276314 32702696
vpath38_hdisk390 52.2 1281.5 202.3 284305564 31887668
vpath39_hdisk391 54.6 1415.8 224.7 332095884 17246952
vpath50_hdisk427 0.1 1.9 0.2 418152 58864
vpath9_hdisk344 49.7 2718.2 132.9 670169062 507176
vpath14_hdisk345 0.0 0.0 0.0 1061 0
vpath15_hdisk346 0.0 0.0 0.0 13 0
vpath10_hdisk347 0.0 0.1 0.0 13195 0
vpath16_hdisk348 0.8 40.1 3.9 7627369 2273476
vpath25_hdisk375 4.6 510.7 6.8 125859580 138220
vpath26_hdisk376 0.1 12.3 0.2 2620284 411011
vpath40_hdisk392 26.1 1039.1 171.8 241047986 15338376
vpath41_hdisk393 18.4 645.9 128.7 148931904 10436436
vpath42_hdisk394 9.5 234.0 50.7 47871308 9874948
vpath43_hdisk395 2.7 62.9 11.1 9273892 6247936
vpath44_hdisk396 0.0 0.0 0.0 19 0

tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 1058.8 70.8 20.4 0.2 8.6
Adapter: Kbps tps Kb_read Kb_wrtn
scsi2 0.0 0.0 0 0

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1_Path0 0.0 0.0 0.0 0 0

Adapter: Kbps tps Kb_read Kb_wrtn
scsi0 8.0 2.0 16 0

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0_Path0 0.5 8.0 2.0 16 0

Adapter: Kbps tps Kb_read Kb_wrtn
fcs0 95719.9 4733.2 190636 756

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
vpath33_hdisk212 0.0 0.0 0.0 0 0
vpath33_hdisk220 0.0 0.0 0.0 0 0
vpath0_hdisk24 54.0 4189.0 67.0 8376 0
vpath1_hdisk25 0.0 0.0 0.0 0 0
vpath11_hdisk46 18.5 1000.3 14.5 2000 0
vpath2_hdisk26 95.5 7992.0 203.1 15980 0
vpath29_hdisk206 0.0 0.0 0.0 0 0
vpath20_hdisk349 0.0 0.0 0.0 0 0
vpath21_hdisk350 0.0 0.0 0.0 0 0
vpath22_hdisk351 0.0 0.0 0.0 0 0
vpath46_hdisk417 0.0 0.0 0.0 0 0
vpath49_hdisk423 0.0 0.0 0.0 0 0
vpath3_hdisk27 97.5 8650.2 149.5 17296 0
vpath12_hdisk47 0.0 0.0 0.0 0 0
vpath30_hdisk93 0.0 0.0 0.0 0 0
vpath31_hdisk210 0.0 0.0 0.0 0 0
vpath32_hdisk211 0.0 0.0 0.0 0 0
vpath34_hdisk213 0.0 0.0 0.0 0 0
vpath35_hdisk214 0.0 0.0 0.0 0 0
vpath36_hdisk216 0.0 0.0 0.0 0 0
vpath4_hdisk28 84.0 4919.2 115.0 9836 0
vpath5_hdisk29 0.0 0.0 0.0 0 0
vpath17_hdisk98 0.0 0.0 0.0 0 0
vpath18_hdisk99 0.0 0.0 0.0 0 0
vpath19_hdisk100 0.0 0.0 0.0 0 0
vpath45_hdisk101 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 0.0 69.0 22.7 0.1 8.3

Adapter: Kbps tps Kb_read Kb_wrtn
scsi2 4.0 1.0 0 8

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1_Path0 0.5 4.0 1.0 0 8

Adapter: Kbps tps Kb_read Kb_wrtn
scsi0 10.0 2.5 12 8

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0_Path0 2.0 10.0 2.5 12 8

Adapter: Kbps tps Kb_read Kb_wrtn
fcs0 95898.5 4724.4 191768 988

 

[AIX5L]

Hmmmm....

We see more Disks over fcs1 than over fcs0...
Is your Paging Space on the Same Disks, as your DB ?
I prefer to use the Own Disks for PS.
Can you send vmstat -v and vmo -a ?
lsattr -El aio0 ?
db2 get dbm cfg and db2 get db cfg ?
How often do you sync the I/O per Minute.
I prefer to let the syncd to run every 10 Seconds.
There are some Ways to tune the I/O Performance.
What is your sdd Level ?
Can you send the datapath query adapter output to see how the Adapters will be used.

 

[AIX5L]

Hi

What Type of Adapter FC Adapters are in the LPAR.
I only see two Adapters in the iostat Output.

For this Configuration i think you need 4 2Gbit Adapters and a good Layout on the ESS Ranks.

Did you know nmon and nmon Analyser ?
You can collect the data over one day analyse it and send it to me. ( post your mailadress please )

 

[RiazAhamed]

Hi AIX5L,

Thanks for your fast response.

Yes, the paging space is kept on the same disk as the DB.

Yes, I know about the nmon analyser. we are using Q Logic FC adapters in this LPAR for connecting to the SAN.

Oh... do we need to have four 2 GB adapters? May I know how did you arrive at this number?

I know the disks layout is good on the IBM ESS side. I am sure, since we don't have this type of performance problems in other LPARs which uses the IBM ESS shark.

here you go:

# vmstat -v
14155776 memory pages
13645476 lruable pages
36777 free pages
4 memory pools
890908 pinned pages
80.1 maxpin percentage
3.0 minperm percentage
10.0 maxperm percentage
33.9 numperm percentage
4631152 file pages
0.0 compressed percentage
0 compressed pages
33.9 numclient percentage
10.0 maxclient percentage
4629114 client pages
0 remote pageouts scheduled
237 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
27790 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
14543 external pager filesystem I/Os blocked with no fsbuf
#

# vmo -a
memory_frames = 14155776
pinnable_frames = 13264064
maxfree = 10120
minfree = 5000
minperm% = 3
minperm = 409362
maxperm% = 10
maxperm = 1364546
strict_maxperm = 0
maxpin% = 80
maxpin = 11324621
maxclient% = 10
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 24903680
npskill = 194560
npswarn = 778240
v_pinshm = 0
pta_balance_threshold = n/a
pagecoloring = n/a
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = n/a
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
low_ps_handling = 1
mbuf_heap_psize = 4096
strict_maxclient = 0
cpu_scale_memp = 8
#

# lsattr -El aio0
autoconfig available STATE to be configured at system restart True
fastpath enable State of fast path True
kprocprio 39 Server PRIORITY True
maxreqs 20000 Maximum number of REQUESTS True
maxservers 20 MAXIMUM number of servers per cpu True
minservers 10 MINIMUM number of servers True
#

========================================================
Here is the sdd level;

# lslpp -L | grep ibm
bos.adt.libm 5.2.0.42 C F Base Application Development
bos.terminfo.ibm.data 5.2.0.30 C F IBM Terminal Definitions
devices.pci.ibmccm.rte 5.2.0.0 C F Common Character Mode Graphics
ibm2105.rte 32.6.100.13 C F IBM 2105 Disk Device
ibm2105cli.rte 1.5.2.9 C F IBM 2105 Command Line
ibmSdd_510.rte

# lslpp -L | grep ibm2105
ibm2105.rte 32.6.100.13 C F IBM 2105 Disk Device
ibm2105cli.rte 1.5.2.9 C F IBM 2105 Command Line
#

Thanks.
Riaz Ahamed
email address: MgRiazAhamed@excite.com

 

[RiazAhamed]

Sorry, I missed couple of stuff:

First,
# datapath query adapter

Active Adapters :2

Adpt# Adapter Name State Mode Select Errors Paths Active
0 fscsi0 NORMAL ACTIVE 807659848 0 100 98
1 fscsi1 NORMAL ACTIVE 799429811 0 94 90
#

Second;

db2 get dbm cfg ?

host:$ /medstat/advantage/wfdm06dp/db_build:db2 get dbm cfg

Database Manager Configuration

Node type = Enterprise Server Edition with local and remote clients

Database manager configuration release level = 0x0a00

CPU speed (millisec/instruction) (CPUSPEED) = 7.557509e-07
Communications bandwidth (MB/sec) (COMM_BANDWIDTH) = 1.000000e+02

Max number of concurrently active databases (NUMDB) = 1
Data Links support (DATALINKS) = NO
Federated Database System Support (FEDERATED) = NO
Transaction processor monitor name (TP_MON_NAME) =

Default charge-back account (DFT_ACCOUNT_STR) =

Java Development Kit installation path (JDK_PATH) = /usr/java13_64

Diagnostic error capture level (DIAGLEVEL) = 3
Notify Level (NOTIFYLEVEL) = 3
Diagnostic data directory path (DIAGPATH) = /medstat/advantage/wf dm06dp/home/wfdm06dp/sqllib/db2dump

Default database monitor switches
Buffer pool (DFT_MON_BUFPOOL) = ON
Lock (DFT_MON_LOCK) = ON
Sort (DFT_MON_SORT) = ON
Statement (DFT_MON_STMT) = ON
Table (DFT_MON_TABLE) = ON
Timestamp (DFT_MON_TIMESTAMP) = ON
Unit of work (DFT_MON_UOW) = ON
Monitor health of instance and databases (HEALTH_MON) = OFF

SYSADM group name (SYSADM_GROUP) = DBA
SYSCTRL group name (SYSCTRL_GROUP) =
SYSMAINT group name (SYSMAINT_GROUP) =
SYSMON group name (SYSMON_GROUP) =

Client Userid-Password Plugin (CLNT_PW_PLUGIN) =
Client Kerberos Plugin (CLNT_KRB_PLUGIN) =
Group Plugin (GROUP_PLUGIN) =
GSS Plugin for Local Authorization (LOCAL_GSSPLUGIN) =
Server Plugin Mode (SRV_PLUGIN_MODE) = UNFENCED
Server List of GSS Plugins (SRVCON_GSSPLUGIN_LIST) =
Server Userid-Password Plugin (SRVCON_PW_PLUGIN) =
Server Connection Authentication (SRVCON_AUTH) = NOT_SPECIFIED
Database manager authentication (AUTHENTICATION) = SERVER
Cataloging allowed without authority (CATALOG_NOAUTH) = NO
Trust all clients (TRUST_ALLCLNTS) = YES
Trusted client authentication (TRUST_CLNTAUTH) = CLIENT
Bypass federated authentication (FED_NOAUTH) = NO

Default database path (DFTDBPATH) = /medstat/advantage/wf dm06dp/home/wfdm06dp

Database monitor heap size (4KB) (MON_HEAP_SZ) = 90
Java Virtual Machine heap size (4KB) (JAVA_HEAP_SZ) = 2048
Audit buffer size (4KB) (AUDIT_BUF_SZ) = 0
Size of instance shared memory (4KB) (INSTANCE_MEMORY) = AUTOMATIC
Backup buffer default size (4KB) (BACKBUFSZ) = 4096
Restore buffer default size (4KB) (RESTBUFSZ) = 4096

Sort heap threshold (4KB) (SHEAPTHRES) = 20000

Directory cache support (DIR_CACHE) = YES

Application support layer heap size (4KB) (ASLHEAPSZ) = 512
Max requester I/O block size (bytes) (RQRIOBLK) = 32767
Query heap size (4KB) (QUERY_HEAP_SZ) = 2048

Workload impact by throttled utilities(UTIL_IMPACT_LIM) = 100

Priority of agents (AGENTPRI) = SYSTEM
Max number of existing agents (MAXAGENTS) = 200
Agent pool size (NUM_POOLAGENTS) = 100(calculated)
Initial number of agents in pool (NUM_INITAGENTS) = 60
Max number of coordinating agents (MAX_COORDAGENTS) = 60
Max no. of concurrent coordinating agents (MAXCAGENTS) = MAX_COORDAGENTS
Max number of client connections (MAX_CONNECTIONS) = MAX_COORDAGENTS

Keep fenced process (KEEPFENCED) = YES
Number of pooled fenced processes (FENCED_POOL) = MAX_COORDAGENTS
Initial number of fenced processes (NUM_INITFENCED) = 0

Index re-creation time and redo index build (INDEXREC) = RESTART

Transaction manager database name (TM_DATABASE) = 1ST_CONN
Transaction resync interval (sec) (RESYNC_INTERVAL) = 180

SPM name (SPM_NAME) = ax_ohp_1
SPM log size (SPM_LOG_FILE_SZ) = 256
SPM resync agent limit (SPM_MAX_RESYNC) = 20
SPM log path (SPM_LOG_PATH) =

TCP/IP Service name (SVCENAME) = db2cwfdm06dp
Discovery mode (DISCOVER) = SEARCH
Discover server instance (DISCOVER_INST) = ENABLE

Maximum query degree of parallelism (MAX_QUERYDEGREE) = ANY
Enable intra-partition parallelism (INTRA_PARALLEL) = NO

No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) = 20000
Number of FCM request blocks (FCM_NUM_RQB) = 30000
Number of FCM connection entries (FCM_NUM_CONNECT) = 22500
Number of FCM message anchors (FCM_NUM_ANCHORS) = 22500

Node connection elapse time (sec) (CONN_ELAPSE) = 10
Max number of node connection retries (MAX_CONNRETRIES) = 5
Max time difference between nodes (min) (MAX_TIME_DIFF) = 60

db2start/db2stop timeout (min) (START_STOP_TIME) = 10

db2 get db cfg ?

 

[DukeSSD]

BEFORE YOU UPGRADE TO ML4/5/6 MAKE SURE YOUR SYSTEM FIRMWARE IS AT THE LATEST LEVEL - yes I meant to shout that !

http://techsupport.services.ibm.com/server/mdownload//download.html

3H050405 is the latest level and seems stable.
lsmcode will tell you your current level.

 

[p5wizard]

You might want to read this SDD flash:

http://www-1.ibm.com/support/docvie...0&uid=ssg1S1002161&loc=en_US&cs=utf-8&lang=en

It's about SDD queue_depth_logic feature and a.o. DB2 and possible perf degradation...

Also 10/20 min/max AIO servers per CPU seems a bit low, though of course on a 20 CPU LPAR that still adds up to 200/400. But I'd double that at least.



HTH,

p5wizard

 

[hirschaj]

One other thing, I am hearing reports about problems with ML06 such as not being able to boot into full partition mode after upgrading so I would go to ML05 for now.


Jim Hirschauer

http://www.aixexpert.com

 

[p5wizard]

if

ps -lk|grep aio

shows all your aioservers consuming CPU time (i.e. all have time 0:01 or more) then it can help to have more servers configured. If your app performs an AIO operation and all aioservers are busy - your application WAITs...

But you need to reboot for that change to become effective...

HTH,

p5wizard

 

[AIX5L]

Hi

Ok..

Increase the aio maxservers to 256 and minimum to 64
Maxrequests to 15872.

What are your Values. ( ioo -a )

j2_maxPageReadAhead ( my initial Values 128 )
j2_minPageReadAhead ( 2 )
numfsbufs ( 500 )

Change syncd rate to 10 Seconds.
move the Paging Space to own Disks.
Are the Logs and the Tablespaces on the same disks ?

Update to ML5 and SDD 1.6.0.2

I dont have alot of experience with qlogic Adapters on PPC.
But If you have a big DB and so many CPUs and Mem you can have some Problem to bring the IO out. If you have 4 Adapters put them on different Switches. You can see the Statistics on your SAN Switch for each Port.

Is your sort heap enough ? 80 MB

 

[RiazAhamed]

Thanks All for your valuable input.

I am looking into this & trying to get detail answers. I"ll keep posting further about the results.

 

[RiazAhamed]

Hi AIX5L,

Here is:

# ioo -a | pg
minpgahead = 64
maxpgahead = 256
pd_npages = 65536
maxrandwrt = 0
numclust = 1
numfsbufs = 2048
sync_release_ilock = 0
lvm_bufcnt = 64
j2_minPageReadAhead = 64
j2_maxPageReadAhead = 256
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_maxRandomWrite = 0
j2_nRandomCluster = 0
j2_non_fatal_crashes_system = 0
jfs_clread_enabled = 0
jfs_use_read_lock = 1
hd_pvs_opn = 52
hd_pbuf_cnt = 7168
j2_inodeCacheSize = 400
j2_metadataCacheSize = 400
j2_dynamicBufferPreallocation = 16
j2_maxUsableMaxTransfer = 512

I don't know whether the logs & tables are on the same disk or not.


For p5wizard,

I see there are 400 "aioserver" processes each consuming CPU time of 3 to 4 minutes in this LPAR. Is this too much?

For hirschaj,

This is an LPAR (p690 series) and this will not be booting in Full partition mode. Also, I installed ML06 on other production LPARs and it's working fine. No problem.

 

[RiazAhamed]

Hi DukeSSD,

Here is the system firmware level:

# lsmcode

DISPLAY MICROCODE LEVEL 802811
IBM,7040-681

Platform Firmware level is 3H041021
System Firmware level is RG041021_d78e05_sfw138
SPCN Firmware level is 0000RHE11193

Use Enter to continue.

 

[RiazAhamed]

Hi All,

The performance seems to degrade further

# sudo vmstat 5
Password:
System Configuration: lcpu=20 mem=55296MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
8 28 10096654 27155 0 0 0 7719 9754 0 10416 81140 47853 19 15 11 54
5 54 10096828 43101 0 0 0 17435 44791 0 11082 36159 42437 7 11 2 80
4 55 10101966 25908 0 2 0 13458 33624 0 10899 204590 42993 10 13 1 75
3 53 10101143 37362 0 0 0 17696 48763 0 10543 128241 41548 9 15 2 75
4 56 10102095 31793 0 0 0 13282 32339 0 10769 124075 47211 10 12 2 76
13 51 10102544 19309 0 0 0 10207 29079 0 9175 38775 35253 7 16 1 75
3 54 10102635 34176 0 0 0 15775 45211 0 9288 48584 34738 9 16 1 74
3 57 10102898 36182 0 0 0 15739 47439 0 11192 39755 46528 8 12 1 79
3 56 10103161 30047 0 0 0 12906 31578 0 11117 39138 44907 9 13 2 77
2 52 10099880 26537 0 0 0 13688 33708 0 11449 35913 45395 8 12 2 79
3 54 10100105 33892 0 0 0 17564 44821 0 11296 130041 45200 10 13 1 75
3 52 10100372 23624 0 0 0 12886 38086 0 11121 42388 44674 8 13 1 77
6 55 10101167 48863 0 0 0 19354 34246 0 11413 41934 46285 8 12 1 78
7 52 10108375 34671 0 1 0 13216 25657 0 10922 189471 47556 12 20 2 66
7 57 10106760 36492 0 0 0 14255 34008 0 10795 149861 42329 14 25 2 60
^C#

Please help. Let me know, if you need more inputs.

Thanks.
Riaz Ahamed
email: MgRiazAhamed@excite.com