Another memory management problem

[marco17]

Hi,
(sorry for my english)

I'have an AIX 5.3 server with 16G of memory.
I run some oracle database and tuxedo on it.
And I'm experiencing page in/page out i can't explain.

I have listed all the command using working segments with

svmon -Pw

Then i have listed all the segments for those commands with:

svmon -C <command name> -w

Then i sum the size of all those segments adding each vsid only one time (am i wrong ?).

The result is 2545724 pages ( near 10G ).

I have set maxperm% to 14.
So 86% of my memory (near 14,5G) is dedicated to working segments, isnt it ?

But i still have page in/page out in vmstat, and i don't understand why.
Anyone can explain me what i'm missing ?

Here vmstat output :

----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 3767702 11115 0 1 0 0 0 0 1177 48124 6727 27 7 59 7
0 0 3767709 11123 0 15 0 130 1639 0 666 29449 5278 20 7 67 6
0 0 3767675 11201 0 0 1 259 2738 0 456 20321 3185 19 4 74 3
1 0 3767697 11155 0 1 0 0 0 0 403 24335 3659 18 4 77 1
1 0 3767684 11162 0 1 0 0 0 0 458 8285 1745 4 13 80 3
0 0 3767820 11061 0 1 0 130 1614 0 233 5835 1243 5 2 89 4
0 2 3765626 13123 0 25 0 0 0 0 297 13164 1372 11 3 82 4
0 0 3765336 13407 0 2 0 0 0 0 873 21741 4145 22 4 71 3
1 0 3765339 13389 0 4 0 0 0 0 1448 13682 3551 17 3 79 1
1 0 3765339 13387 0 2 0 0 0 0 1424 12870 3478 16 3 80 1
0 2 3765518 13168 0 0 0 0 0 0 564 11474 2229 10 3 83 4
1 0 3765432 13283 0 6 0 131 1608 0 976 22177 3479 20 11 64 5
0 0 3765735 12940 0 11 0 0 0 0 582 19898 2531 20 6 72 3
1 0 3765677 12853 0 36 0 0 0 0 758 30465 3231 49 9 36 6
1 0 3765955 12401 0 6 0 0 0 0 489 24109 2957 45 6 46 4
4 1 3766120 12119 0 45 0 129 1777 0 649 183010 6533 68 19 10 2
1 0 3765272 12903 0 3 0 129 1397 0 363 33677 6031 53 6 36 5
0 0 3764828 13317 0 9 0 129 2088 0 417 20112 3107 15 3 81 1
0 2 3765476 12139 0 6 0 129 1095 0 880 25594 4442 31 8 47 14
1 0 3764552 12872 0 11 0 129 876 0 534 23930 3872 26 5 65 4
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 3762909 14515 0 0 0 0 0 0 232 7364 1446 6 2 92 0
0 0 3762778 14646 0 0 0 0 0 0 763 16662 3505 14 5 81 0
1 0 3764559 12740 0 14 0 0 0 0 347 82126 2246 24 9 58 10
0 1 3764646 12618 0 5 0 0 0 0 831 18417 2804 11 4 84 1
2 0 3774603 2633 0 8 176 281 416 0 485 18341 2129 21 10 60 9
2 0 3766643 10458 0 11 105 1961 18485 0 741 20744 3737 32 7 53 8
2 2 3766010 11058 0 2 0 0 0 0 299 27028 3285 56 8 35 1
3 0 3765770 11180 0 16 0 0 0 0 323 25111 2552 44 5 48 3
0 0 3765690 11254 0 0 0 0 0 0 161 4842 960 3 2 95 1
0 0 3764118 12802 0 1 0 0 0 0 261 4928 1085 3 2 92 3
0 0 3764127 12827 0 3 0 129 7737 0 231 5022 1188 3 2 94 2
0 0 3764034 12919 0 0 0 0 0 0 969 11411 2802 11 5 84 0
0 1 3764515 12341 0 26 0 130 8214 0 387 12139 1350 11 3 77 8
0 0 3764901 11890 0 12 0 129 9783 0 408 24970 4242 24 4 67 4
1 0 3765641 11083 0 11 0 0 0 0 385 25628 3924 38 4 56 2
1 0 3765929 10770 0 9 0 0 0 0 283 20008 1583 42 3 54 1
0 0 3765916 10770 0 0 0 0 0 0 359 28091 2338 51 4 44 0
1 0 3765922 10772 0 12 0 131 4865 0 275 21216 2398 37 3 57 3


kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 3776242 10175 0 0 0 0 0 0 260 103540 2246 23 15 59 2
0 0 3776277 10136 0 0 0 0 0 0 891 12397 2577 11 3 86 0
0 0 3776277 10095 0 0 0 0 0 0 114 3900 1061 3 1 95 1
0 0 3776121 10233 0 1 0 0 0 0 410 25231 3702 23 8 68 1
1 0 3776439 9887 0 15 0 0 0 0 160 23457 4426 26 4 69 1
3 0 3779237 7025 0 13 0 0 0 0 371 16539 1477 47 4 45 3
1 0 3779234 6762 0 1 0 132 1009 0 432 19758 1866 51 2 45 1
0 0 3779400 6571 0 2 0 0 0 0 164 18094 1417 35 2 62 1
1 0 3778977 6985 0 0 0 0 0 0 670 23732 2874 46 4 50 1
5 0 3778206 7712 0 2 0 0 0 0 853 48373 6402 72 7 20 1
0 0 3778409 7574 0 3 0 129 936 0 661 52668 5553 72 6 22 0
1 1 3783083 2874 0 0 0 0 0 0 486 164508 19653 65 14 20 1
4 0 3778593 9406 0 3 1526 2060 2796 0 1738 101889 3413 61 12 24 3
1 0 3778510 9479 0 3 0 0 0 0 1253 94046 3114 57 8 34 0
1 0 3785906 2137 0 5 27 515 769 0 618 28394 3214 55 11 29 4
2 0 3782482 9246 0 3 2372 3353 10919 0 817 18938 1753 87 5 7 0
3 0 3787038 4683 0 1 0 0 0 0 184 24597 1962 98 2 0 0
5 0 3788978 2741 0 1 0 0 0 0 761 25695 2693 97 2 0 0
3 0 3778553 14626 0 1 206 1486 12367 0 502 22716 2954 96 4 1 0
6 0 3778320 14813 0 5 0 0 0 0 1045 44226 5779 91 5 4 0
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 0 3778418 14694 0 18 0 0 0 0 201 16371 1201 77 2 21 0
3 0 3776393 16700 0 2 0 0 0 0 509 32606 3100 79 4 18 0
3 0 3776445 16599 0 1 0 0 0 0 303 16033 1986 60 5 33 2
1 0 3776438 16447 0 156 0 0 0 0 269 4689 602 40 7 47 6
0 1 3776314 16142 0 430 0 0 0 0 512 11517 1381 18 8 55 19
0 1 3778104 14058 0 287 0 0 0 0 854 18150 3329 41 4 43 12
0 0 3777884 14185 0 65 0 0 0 0 190 12747 1086 35 2 58 5
3 0 3777989 14064 0 0 0 0 0 0 575 63311 2855 50 9 40 0
1 0 3777738 14285 0 1 0 0 0 0 330 39264 4768 58 5 36 1
1 0 3777623 14353 0 8 0 0 0 0 591 29536 5888 63 6 29 2
0 0 3777595 14331 0 6 0 0 0 0 351 12799 2032 9 4 86 1
2 0 3778760 13123 0 11 0 0 0 0 623 11164 2624 16 3 64 18
0 0 3777911 13949 0 0 0 0 0 0 220 4915 788 45 2 50 2
0 0 3777703 14156 0 0 0 0 0 0 101 5675 691 1 1 97 1
0 0 3777854 13571 0 6 0 0 0 0 1080 11443 2858 13 5 68 14
0 11 3777907 13513 0 4 0 0 0 0 1250 14546 3364 18 4 77 2
0 0 3778035 13299 0 2 0 0 0 0 318 16624 2403 10 3 83 4

this is the svmon -G output taken at the same time:

size inuse free pin virtual
memory 4079616 4067404 12212 990282 3778381
pg space 2162688 230737

work pers clnt
pin 990282 0 0
in use 3645399 0 422005

PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 3869548 226593 881130 3577293
m 64 KB - 12366 259 6822 12568