High paging in 5.2, not seen in 5.1

[kjsys1]

This is on our Oracle unix server---memory use is not really the problem, per se.(16 processors, 9216MB of real memory, 6144MB of paging space).
The problem is that computational pages are being paged out/in based on what we calculate as computational pages at about 50%, and not (no where near either) on the maxperm% setting of 20% (minperm% is at 10%).
Is this not correct?? Computational pages should NOT be paged out? If the working set of computational pages as a % of real memory is less than the MINPERM setting, then computational pages ARE fair game for stealing (paging out) to replenish the free list. But in our case they are greater than MAXPERM they should not be stolen. I've done daily svmon -G throughout the day and see computational as about 50%. Now, the very day we upgraded from 5.1 to 5.2 this started happening---whereas (under 5.1) this was not the behavior.

I don't believe setting strict_maxperm to 1 is what we want to do?? We believe that we would either waste memory (maxperm setting) if set too low, or cause computational pages to be paged out if set too high. In theory if strict_maxperm is not set, the file pages (numperm) should sort of dynamically adjust based on the requirements set for %computational pages?

Basicly how do we get back to the 5.1 behavior without actually downgrading to 5.1? Does anyone know what parm changes might make this happen?

Many thanks to any help you can give me!

 

[RMGBELGIUM]

is your kernel running in 32 bit or 64 bit? (bootinfo -K )

regards,

R.

 

[hirschaj]

Are you running AIX 5.2 ML4? If so upgrade to anything newer and I bet your paging problem will disappear. There were major problems with AIO and paging in ML4.


Jim Hirschauer

http://www.aixexpert.com

 

[kjsys1]

Here is the requested info:
/tmp $s bootinfo -K
32
/tmp $oslevel -r
5200-05
/tmp $

 

[hirschaj]

Can you provide output from vmstat -v?


Jim Hirschauer

http://www.aixexpert.com

 

[kjsys1]

/tmp $vmstat -v
2359296 memory pages
2282865 lruable pages
6977 free pages
3 memory pools
130333 pinned pages
80.1 maxpin percentage
10.0 minperm percentage
20.0 maxperm percentage
53.6 numperm percentage
1224760 file pages
0.0 compressed percentage
0 compressed pages
0.0 numclient percentage
20.0 maxclient percentage
0 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
448426 paging space I/Os blocked with no sbuf
37518 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os locked with no fsbuf


Here are some vmstat outputs around 6:00 AM, when Oracle comes up:
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 1 0 674136 6976 28 9 0 0 0 0 1787 1760 359 2 1 96 1 05:55:03
1 1 0 673550 7644 29 3 0 0 34 246 1789 5157 459 2 0 97 1 05:55:18
1 1 0 673561 9172 0 6 0 0 102 1167 1755 9366 359 3 1 95 0 05:55:33
1 0 0 673775 8956 0 1 0 0 0 0 1745 1012 207 3 0 97 0 05:55:48
1 0 0 673736 8961 0 22 0 0 0 0 1780 3515 279 1 1 98 0 05:56:03
1 0 0 673577 9106 0 3 0 0 0 0 1753 996 241 3 0 97 0 05:56:18
1 0 0 673579 9092 0 2 0 0 0 0 1749 637 217 3 0 97 0 05:56:33
1 0 0 673579 9090 0 1 0 0 0 0 1743 474 202 0 0 99 0 05:56:48
1 0 0 673581 9058 0 10 0 0 0 0 1754 1430 219 3 0 97 0 05:57:03
1 0 0 673585 9050 0 2 0 0 0 0 1766 873 286 3 0 97 0 05:57:18
1 0 0 673585 9048 0 1 0 0 0 0 1743 708 208 1 0 99 0 05:57:33
1 0 0 673587 9042 0 2 0 0 0 0 1742 451 208 2 0 98 0 05:57:48
1 0 0 673589 9002 0 8 0 0 0 0 1752 1291 217 3 0 96 0 05:58:04
1 0 0 673591 8998 0 1 0 0 0 0 1752 885 243 2 0 98 0 05:58:19
1 0 0 673593 8992 0 2 0 0 0 0 1754 863 248 1 0 99 0 05:58:34
1 0 0 673597 8980 0 2 0 0 0 0 1743 622 210 3 0 97 0 05:58:49
1 0 0 673405 9135 0 8 0 0 0 0 1749 1256 212 3 0 97 0 05:59:04
0 0 0 673407 9129 0 2 0 0 0 0 1746 775 209 0 0 99 0 05:59:19
1 0 0 673409 9115 0 2 0 0 0 0 1763 803 284 3 0 97 0 05:59:34
2 1 0 683756 5834 103 12 0 0 581 2225 1966 33592 1233 5 6 85 4 05:59:49
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
2 1 0 685639 9722 255 210 0 0 698 2896 2774 46788 3654 3 7 81 8 06:00:04
1 1 0 683838 8176 181 122 0 0 67 193 2565 23578 4130 5 1 88 6 06:00:19
1 4 0 704653 5836 671 57 0 0 1884 6913 2339 23674 3684 4 2 71 22 06:00:34
1 6 0 718982 6044 988 56 0 6 1999 13383 2640 1678 3444 2 1 61 36 06:00:49
1 5 0 728590 5884 632 172 2 434 1375 8399 2389 4579 2165 2 2 62 34 06:01:04
1 5 0 741646 5791 907 69 1 376 1827 9179 2603 2477 3470 4 1 60 35 06:01:19
1 5 0 754086 5985 880 219 11 402 1928 7829 2594 10399 3260 3 3 59 35 06:01:34
1 5 0 764684 5788 759 70 4 310 1513 15281 2459 1850 2738 2 1 61 37 06:01:49
1 6 0 775002 5937 728 261 11 299 1667 7224 2459 3014 2969 4 1 58 37 06:02:04
1 6 0 788180 6116 911 128 10 413 1912 8302 2607 2493 3497 4 1 57 39 06:02:19
1 7 0 798421 5958 709 102 7 301 1476 6755 2409 2256 2836 1 1 55 43 06:02:34
1 7 0 808082 6140 683 246 10 307 1578 10088 2416 2380 2779 4 1 52 43 06:02:49
1 7 0 818323 5856 698 160 20 369 1509 7863 2439 4130 3128 4 2 52 42 06:03:04
1 4 0 832885 5886 986 288 15 449 2237 10425 2704 2857 4001 2 1 65 31 06:03:19
1 7 0 844332 5957 782 106 13 345 1655 11432 2493 2514 3195 3 1 52 44 06:03:34
1 8 0 854868 5918 720 254 18 365 1672 7747 2468 2409 3017 4 1 52 43 06:03:49
1 7 0 865456 6205 722 153 21 302 1606 9358 2447 3034 3028 3 1 51 45 06:04:04
1 7 0 876090 6155 680 197 22 338 1570 6741 2409 4408 3268 3 2 52 43 06:04:19
1 6 0 894050 5816 1261 299 28 573 2737 14598 2987 3901 5046 5 2 56 38 06:04:34
1 5 0 915547 5833 1421 332 43 681 3200 21092 3170 3564 5680 4 2 56 38 06:04:49
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 8 0 927803 5846 879 193 37 435 1900 13252 2640 3947 3837 2 1 51 45 06:05:04
1 7 0 942324 6019 970 375 44 483 2320 11917 2828 4065 4634 5 2 51 43 06:05:19
2 5 0 959550 5936 1149 355 52 548 2660 14036 2908 12299 5014 5 3 57 35 06:05:34
1 4 0 996697 5762 2330 377 61 1141 5189 23883 4085 3277 8123 2 3 69 25 06:05:49
1 2 0 1013184 7176 763 262 276 767 2460 13212 2599 8575 3424 5 2 78 15 06:06:04
1 2 0 1013187 5784 318 239 223 7 649 4921 2195 3588 2504 2 1 84 12 06:06:19
1 2 0 1013700 5835 280 362 298 137 915 7990 2304 3820 2966 4 1 81 15 06:06:34
1 2 0 1013704 5938 355 359 239 112 889 6991 2291 3560 2954 4 1 80 16 06:06:49
1 2 0 1013834 5809 321 360 186 88 803 2909 2226 4421 2717 3 1 83 14 06:07:04
1 3 0 1013837 5885 322 467 247 128 912 4111 2345 3998 3380 5 1 79 15 06:07:19
1 2 0 1013840 5826 300 632 273 155 992 6764 2395 4057 3715 2 1 84 12 06:07:34
2 2 0 1013844 5996 360 456 204 83 880 3990 2315 3706 3320 5 1 78 16 06:07:49
1 2 0 1013847 5919 337 545 207 98 922 4645 2311 4278 3189 2 1 84 13 06:08:04
1 3 0 1013851 5997 389 423 236 111 931 6339 2359 4109 3343 5 1 77 17 06:08:19
1 2 0 1013855 5829 296 448 248 94 846 4011 2307 3468 3076 2 1 84 13 06:08:34
1 2 0 1013860 5781 333 250 189 85 701 2645 2208 3066 2523 4 1 77 18 06:08:49
1 2 0 1013872 6063 310 436 247 175 880 5013 2326 4706 3019 2 1 83 14 06:09:04
1 3 0 1013875 5911 336 388 255 44 863 5055 2320 4089 3201 4 1 77 18 06:09:19
1 2 0 1013877 5824 270 501 337 158 956 4132 2411 5612 3548 3 2 82 14 06:09:34
1 3 0 1014868 5792 452 512 258 152 1162 7015 2460 17524 4227 5 3 70 23 06:09:49
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 2 0 1014874 5835 430 549 237 110 1083 6792 2396 20099 4053 3 4 77 16 06:10:04
1 2 0 1014871 5770 394 565 276 124 991 4002 2528 30404 3616 5 7 72 17 06:10:19
1 2 0 1013910 5803 328 517 294 142 938 6269 2404 19495 4048 3 3 81 12 06:10:34
1 2 0 1013898 5890 397 531 179 36 1000 6531 2306 4874 3866 5 1 79 16 06:10:49
1 1 0 1013902 5767 376 485 204 67 949 4305 2319 8097 3492 3 1 83 12 06:11:04
1 3 0 1013906 5886 428 643 164 64 1027 7522 2416 5181 4161 4 2 77 18 06:11:19
1 2 0 1013914 5953 443 477 98 26 923 5124 2209 5191 3500 4 1 82 14 06:11:34
1 3 0 1013920 5820 427 574 149 64 975 6621 2324 4625 4110 3 1 79 17 06:11:49
1 2 0 1014004 6400 568 610 80 86 1127 6552 2418 6179 4262 5 1 77 16 06:12:04
1 2 0 1014004 5951 496 575 12 0 862 4170 2229 4672 3918 2 1 81 16 06:12:19
1 3 0 1014004 5991 603 633 9 0 1058 9818 2297 5230 4025 4 1 75 20 06:12:34
1 3 0 1014004 6090 519 649 9 0 991 5645 2235 5100 3945 3 1 79 17 06:12:49
1 2 0 1014005 5792 537 556 8 0 931 7794 2225 5973 3733 3 1 79 18 06:13:04
1 2 0 984249 36513 449 652 23 3 1016 4986 2229 6174 4305 3 2 80 15 06:13:19
1 3 0 987441 19159 493 551 51 0 0 0 2365 4960 4632 3 1 78 18 06:13:34
1 4 0 988465 5877 617 482 59 0 187 1158 2418 4284 4951 3 1 64 32 06:13:49
1 2 0 988465 5810 771 823 84 0 1263 7931 2640 5926 5880 5 1 75 19 06:14:04
1 3 0 988465 5761 778 918 48 0 1402 5786 2649 5737 7063 5 1 76 18 06:14:19
1 3 0 988465 5854 802 947 41 0 1382 8897 2662 5780 6267 4 2 76 17 06:14:35
1 3 0 988465 6063 1293 950 4 0 1881 8038 3086 5662 9010 6 2 69 23 06:14:50
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 3 0 1015104 6167 1442 790 2 179 3723 20843 3227 7443 9474 4 3 75 19 06:15:05
1 2 0 1015104 5892 656 637 2 0 1084 7962 2445 6678 5850 6 1 79 14 06:15:20
2 2 0 1015104 5850 542 524 3 0 913 6167 2289 14279 5368 5 3 79 13 06:15:35
1 2 0 1015104 6002 542 671 1 0 1033 7645 2319 6015 5018 4 1 82 13 06:15:50
1 2 0 1015104 5938 604 695 1 0 1092 8226 2358 9631 6013 6 2 79 13 06:16:05
1 1 0 1015104 5923 561 640 1 0 1023 7312 2288 6107 5555 3 1 84 12 06:16:20
1 1 0 1015104 6011 570 762 2 0 1093 7285 2357 6585 5098 5 2 80 12 06:16:35
1 3 0 1015104 6106 510 625 1 0 972 5570 2258 5882 5390 5 1 81 12 06:16:50
1 2 0 1015104 5965 579 672 0 0 1042 6499 2297 6772 4656 3 1 83 13 06:17:05
1 1 0 1015104 5961 572 659 2 0 1033 7436 2281 5636 5195 5 1 81 13 06:17:20
1 2 0 1015104 6002 596 700 1 0 1093 7070 2365 5949 5805 6 2 79 14 06:17:35
1 1 0 1015104 5929 624 494 1 0 974 6965 2288 4646 4843 3 1 83 13 06:17:50
1 2 0 1015112 5964 631 630 1 0 1074 7321 2359 7177 4919 5 1 79 14 06:18:05
1 3 0 1015112 5867 761 652 51 0 1254 8470 2565 5774 6879 6 1 75 18 06:18:20
1 2 0 1015112 5895 598 512 72 0 1024 7372 2409 5551 6015 4 2 78 17 06:18:35
1 1 0 1015112 5961 524 499 8 0 887 6645 2276 4492 4571 4 1 81 14 06:18:50
2 3 0 1015117 5917 1261 2496 30 0 3592 23135 2619 6973 6666 6 2 69 24 06:19:05
1 2 0 1015137 6112 1778 3583 20 0 5191 31518 2685 5665 5649 3 2 75 20 06:19:20
1 4 0 1015137 5986 1366 3030 95 46 4302 28052 2617 4925 4711 3 3 70 24 06:19:35
2 4 0 1016124 6102 1260 2936 88 27 4179 25340 2702 14824 3846 3 3 72 22 06:19:50
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 3 0 1016159 5806 1308 2832 109 68 4098 26252 2799 11713 4963 5 4 68 23 06:20:05
2 2 0 987511 8696 964 2366 106 0 1576 11136 2518 30653 4182 6 7 71 16 06:20:20
2 2 0 986488 5816 880 2476 91 0 2882 13137 2585 26753 3679 10 6 68 16 06:20:35
1 1 0 986488 5764 731 2459 63 0 3110 13597 2376 3411 3902 11 1 76 12 06:20:50
1 3 0 986511 7008 1171 2722 28 0 3855 16780 2412 8459 4641 10 2 68 20 06:21:05
2 5 0 991235 5914 1338 2840 10 0 3924 18830 3116 5870 6854 7 2 63 28 06:21:20
1 3 0 1011965 5887 1938 3524 2 98 6320 36784 3304 5697 9650 7 3 66 24 06:21:35
2 3 0 1011946 5929 1803 2693 2 0 4049 26790 2881 6174 9683 5 4 65 25 06:21:50
1 3 0 1012084 7447 1327 1729 18 0 2629 19346 3001 6019 9007 4 2 72 22 06:22:05
1 3 0 1013071 5811 1034 741 107 39 1487 11002 2938 4246 8890 6 2 70 23 06:22:20
1 3 0 1013101 5761 725 844 69 41 1323 11138 2638 3906 9115 5 2 74 19 06:22:35
1 4 0 1022632 5835 1054 808 108 332 2152 13616 3078 4250 6572 4 3 74 20 06:22:50
1 3 0 1022892 6056 1261 1411 115 58 2057 15673 3340 6044 8743 6 2 70 22 06:23:05
2 4 0 1023061 6067 1386 1627 119 57 2810 16312 3397 5692 8610 6 2 71 21 06:23:20
1 3 0 1025063 5998 1302 716 113 108 2130 18819 3175 5043 6433 4 2 75 20 06:23:35
1 3 0 1026251 6025 1179 647 134 107 1821 12868 3105 5566 7540 6 2 70 22 06:23:50
1 3 0 1027985 5946 1142 566 89 118 1732 12824 3018 6750 7208 5 2 70 23 06:24:05
1 5 0 1030671 6036 1216 733 70 74 1971 14620 3137 5828 7063 4 2 70 24 06:24:20
1 3 0 1032103 5820 661 674 113 100 1316 9733 2675 6897 6082 6 2 70 22 06:24:35
1 3 0 1033849 5915 1005 588 112 91 1618 11030 2971 4427 6549 4 2 69 25 06:24:50
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 3 0 1035927 6113 1298 654 115 128 1996 15247 3265 5700 7031 4 2 71 23 06:25:05
2 3 0 1036393 5822 1029 1468 147 74 1854 14399 3091 5522 7218 6 2 71 22 06:25:20
2 3 0 1036479 5850 1171 1729 90 113 1921 11703 3048 14219 7372 7 3 72 18 06:25:35
1 2 0 1036495 6251 386 729 121 0 1075 9399 2349 5229 4400 7 2 79 13 06:25:50
1 3 0 1036543 5764 452 647 140 48 987 5692 2474 8943 5807 9 1 78 12 06:26:05
2 2 0 990810 39735 425 574 233 40 273 1489 2513 5363 6881 5 1 79 14 06:26:20
1 2 0 991834 22327 601 565 97 0 0 0 2534 5151 8465 5 1 77 17 06:26:35
1 3 0 992906 6051 815 696 19 0 289 1973 2704 6528 11041 6 2 73 19 06:26:50
1 2 0 993682 5818 691 618 35 0 1175 4063 2568 7307 11471 5 2 77 16 06:27:05
1 3 0 1000368 5832 953 729 42 0 1928 10144 2824 6134 8665 5 2 72 21 06:27:20
1 2 0 1003960 6033 994 571 45 0 1697 6928 2804 5725 8320 7 2 72 19 06:27:35
1 3 0 1004444 5983 1048 678 41 0 1572 9770 2883 5977 8365 5 2 71 22 06:27:50
1 3 0 1004458 5853 1935 554 37 0 2332 13574 3672 5782 10624 5 3 73 20 06:28:05
1 3 0 1008902 5874 1722 593 85 0 2514 15960 3511 5686 10375 7 2 72 18 06:28:20
1 3 0 1023767 5791 1095 2022 125 0 2668 15296 3088 5061 7420 6 2 74 18 06:28:35
1 1 0 976088 41992 186 563 109 0 76 413 2124 10158 3620 4 1 87 8 06:28:50
1 1 0 976088 31565 195 700 19 0 0 0 2093 6724 4868 5 2 88 5 06:29:05
1 1 0 974905 22130 211 655 22 0 0 0 2096 6101 4750 5 1 89 6 06:29:20
1 1 0 974905 12068 183 664 18 0 0 0 2077 6432 4174 3 1 90 6 06:29:35
2 1 0 975889 5982 230 699 16 0 375 1631 2119 18643 4916 5 3 84 8 06:29:50
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
2 3 0 975890 6709 274 748 15 0 801 3548 2208 22778 4967 6 5 80 9 06:30:05
1 1 0 975902 5776 222 700 19 0 647 1862 2124 32506 4649 3 6 85 7 06:30:20
1 1 0 974899 5890 196 646 15 0 598 2572 2068 17759 4084 5 3 87 6 06:30:36
1 1 0 974899 5829 190 688 17 0 674 2890 2078 5796 4801 5 1 89 5 06:30:51
1 1 0 974889 5813 196 918 18 0 735 3937 2211 9746 5309 3 2 89 5 06:31:06
1 1 0 974889 5787 197 721 16 0 640 2662 2090 6435 4330 5 1 90 5 06:31:21
2 3 0 974889 5773 183 755 15 0 649 3307 2090 6689 5370 5 1 89 5 06:31:36
1 1 0 974889 6140 196 743 11 0 683 1870 2091 6631 5241 4 1 90 5 06:31:51
1 1 0 973910 6361 177 699 17 0 615 1817 2075 6283 3966 4 2 89 5 06:32:06
1 1 0 973910 6103 184 785 13 0 656 2707 2094 7344 5595 5 1 89 5 06:32:21
1 1 0 973910 5855 189 736 13 0 656 2608 2066 6199 5041 5 1 90 5 06:32:36
1 1 0 974063 5961 198 735 27 0 700 2876 2127 6097 4252 3 1 90 6 06:32:51
1 2 0 974077 6583 189 688 21 0 647 2645 2089 6645 4558 5 2 88 5 06:33:06
1 1 0 975204 6040 180 757 16 0 683 3303 2099 6459 5077 5 1 89 5 06:33:21
1 1 0 975204 5806 160 552 12 0 495 1392 2008 5234 3419 2 1 92 5 06:33:36
1 1 0 975204 5992 185 515 20 0 530 1645 2040 5375 4220 5 1 87 7 06:33:51
1 1 0 975204 5990 183 686 17 0 614 2257 2066 7104 4966 5 2 88 5 06:34:06
1 1 0 975204 6154 180 694 14 0 622 2968 2060 6447 4858 3 1 91 6 06:34:21
1 1 0 975204 6151 206 721 17 0 657 3759 2086 5524 4084 5 1 88 6 06:34:36
1 1 0 975211 5878 197 657 16 0 596 2765 2073 5917 4684 5 1 88 6 06:34:51
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
2 1 0 975211 5767 179 684 11 0 588 3262 2070 6504 4652 3 1 90 6 06:35:06
1 1 0 976463 6173 191 583 30 0 716 2074 2076 6199 3860 5 1 87 6 06:35:21
2 1 0 976470 5856 189 860 13 0 665 2136 2138 14868 5490 6 2 86 5 06:35:36
1 1 0 976470 6080 191 726 14 0 656 2716 2079 5936 4871 4 1 90 5 06:35:51
1 1 0 974519 7245 197 834 16 0 699 3355 2140 10509 4537 5 1 88 5 06:36:06
1 2 0 974519 5915 184 813 11 0 563 1921 2127 6812 5418 5 2 88 5 06:36:21
1 1 0 973302 6066 197 678 13 0 546 2547 2069 6551 5078 4 1 90 5 06:36:36
1 1 0 973302 5782 197 751 12 0 648 2601 2085 6727 4481 4 1 91 5 06:36:51
2 1 0 973302 6004 202 774 15 0 681 2088 2116 6972 5101 6 1 88 5 06:37:06
1 1 0 973302 6082 187 792 3 0 657 2452 2097 6789 5352 5 2 88 5 06:37:21
1 1 0 973302 5823 200 730 9 0 630 2589 2075 6125 4209 3 1 91 5 06:37:36
1 1 0 973302 6037 209 775 7 0 699 3756 2098 6616 5284 5 1 89 5 06:37:51
1 2 0 973302 6040 195 677 8 0 615 2436 2069 6818 4775 6 1 89 5 06:38:06
1 1 0 973302 5847 192 769 6 0 632 3347 2098 5865 4247 3 1 91 5 06:38:21
1 1 0 973302 5967 176 758 4 0 647 1858 2073 6571 5174 6 1 88 5 06:38:36
1 1 0 973302 6094 205 718 2 0 655 1980 2074 6088 4914 5 1 86 8 06:38:51
1 4 0 973306 5880 354 888 0 0 869 3526 2264 8550 6281 4 1 68 27 06:39:06
1 2 0 973307 5977 237 815 0 0 717 3907 2165 6477 4736 5 2 83 11 06:39:21
2 3 0 973305 5964 215 768 1 0 674 2860 2086 9282 5728 6 2 87 6 06:39:36
2 1 0 975484 6348 287 814 3 0 955 3984 2177 24026 5637 4 4 83 9 06:39:51
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 1 0 975611 5841 220 866 17 0 717 2277 2162 36700 5144 5 7 82 6 06:40:06
2 2 0 974590 6137 239 953 0 0 657 2847 2278 20396 5699 6 4 83 7 06:40:21
1 1 0 974596 6019 207 802 0 0 683 3340 2103 15411 5422 5 2 88 5 06:40:36
1 1 0 974596 5824 202 711 0 0 653 3076 2046 6249 4156 4 1 90 5 06:40:51
1 1 0 974695 5765 203 883 5 0 729 4062 2149 10422 5592 6 1 87 5 06:41:06
1 2 0 974695 5800 195 755 0 0 648 1949 2098 7155 5444 5 1 89 5 06:41:21
1 1 0 974695 5872 202 787 0 0 690 2423 2095 6913 4572 4 1 90 6 06:41:36
1 1 0 974695 5795 197 780 0 0 657 2791 2092 6706 5328 5 1 88 5 06:41:51
1 1 0 974705 6414 200 700 0 0 672 3571 2054 7525 4826 6 1 88 5 06:42:06
1 1 0 973488 6019 202 811 0 0 588 2840 2092 7002 4699 3 1 91 5 06:42:21
1 1 0 973488 6069 195 735 0 0 630 3796 2102 6543 5151 6 2 87 5 06:42:36
1 1 0 973488 5829 187 769 0 0 630 2039 2076 6128 5023 5 1 88 5 06:42:51
1 1 0 974525 6197 458 1545 1 0 1831 7598 2256 6456 5074 3 1 88 8 06:43:06
2 1 0 974528 5854 894 2836 0 0 3415 15629 2468 5928 4892 5 2 82 11 06:43:21
2 1 0 974528 5957 942 2868 0 0 3551 15349 2423 5809 5422 5 3 80 11 06:43:36
1 1 0 974528 5888 926 2730 0 0 3426 14699 2402 6090 5429 3 2 83 12 06:43:51
1 1 0 974528 5810 1020 3172 0 0 3959 16516 2517 7504 5120 5 2 82 11 06:44:06
1 1 0 974528 5897 565 2070 0 0 2367 9376 2336 6183 5566 6 1 82 11 06:44:21
1 2 0 974528 6087 648 2278 0 0 2683 11277 2388 6621 5684 4 2 82 11 06:44:36
1 1 0 974528 6061 911 2752 0 0 3415 20317 2413 6148 4919 4 2 83 11 06:44:51
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 2 0 974542 5851 1018 2839 0 0 3621 21661 2428 8158 5799 6 2 80 13 06:45:06
1 1 0 974542 5845 849 2845 0 0 3445 16071 2509 6722 6175 5 2 82 12 06:45:21
1 1 0 974551 6053 851 2882 1 0 3488 14201 2565 15481 5521 4 4 79 13 06:45:36
1 2 0 974562 5777 1620 4766 0 0 6141 26746 2780 6110 6176 6 3 80 12 06:45:51
1 2 0 972357 5882 1368 3384 0 0 4440 28404 2422 8646 5055 5 3 83 9 06:46:06
1 1 0 972359 5812 204 712 0 0 733 4027 2072 6309 4255 3 1 91 6 06:46:21
1 1 0 972359 6222 201 771 0 0 793 4450 2110 5644 4851 5 2 88 5 06:46:36
1 1 0 972363 5852 206 810 1 0 794 4772 2121 6559 5308 5 1 88 5 06:46:51
1 1 0 972363 6141 193 694 0 0 751 2568 2050 7079 4927 3 1 91 5 06:47:07
1 1 0 973653 5837 236 778 47 0 922 3839 2171 6389 4532 5 1 85 9 06:47:22
1 2 0 973659 6196 198 646 139 0 837 4224 2198 5523 4837 5 2 82 11 06:47:37
1 1 0 973660 5832 166 656 139 0 786 3206 2169 5514 4785 3 1 85 11 06:47:52
1 1 0 973795 5861 343 704 37 0 921 4045 2191 7516 4340 5 1 82 12 06:48:07
1 1 0 973796 5908 214 806 0 0 818 2776 2104 6072 5093 5 1 87 6 06:48:22
1 1 0 973796 5997 152 686 0 0 640 2875 2048 6029 4861 4 1 90 5 06:48:37
1 1 0 973796 6103 184 719 0 0 708 3642 2082 5951 4109 4 1 90 5 06:48:52
2 4 0 973796 6039 216 798 0 0 810 4238 2117 7545 5325 6 1 88 5 06:49:07
1 1 0 972366 6084 176 794 0 0 656 3403 2093 6305 5219 5 1 89 5 06:49:22
1 1 0 972366 6120 196 796 0 0 794 2855 2089 6203 4345 3 1 91 5 06:49:37
2 1 0 973375 5822 201 720 0 0 734 2995 2097 12921 5008 5 3 84 8 06:49:52
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 2 0 973387 6078 256 833 0 0 845 5019 2182 13224 5714 6 2 82 10 06:50:07
1 1 0 973401 5830 293 741 0 0 801 3784 2168 14550 4516 3 3 85 10 06:50:22
2 1 0 973432 5869 223 759 0 0 734 3144 2119 43006 5214 6 8 80 6 06:50:37
1 3 0 972414 5956 274 887 0 0 717 2402 2268 16748 5251 6 3 83 9 06:50:52
1 1 0 972428 6054 209 778 1 0 751 3048 2091 10344 5058 3 1 90 5 06:51:07
1 1 0 972428 6119 196 705 0 0 699 4560 2060 7054 4447 5 1 89 5 06:51:22
1 1 0 972428 6118 207 800 0 0 750 3835 2100 5973 5094 6 1 89 5 06:51:37
1 1 0 972428 6184 189 805 0 0 726 3377 2100 8015 5854 4 2 90 4 06:51:52
1 1 0 972428 5962 199 846 0 0 750 2561 2107 7402 4675 5 1 90 5 06:52:07
1 2 0 972428 5761 203 744 0 0 708 3675 2092 6203 4979 5 1 88 6 06:52:22
1 1 0 972428 5936 205 805 0 0 768 4638 2096 7126 5560 5 1 90 4 06:52:37
1 1 0 973668 6052 228 837 0 0 861 4654 2151 6362 4487 4 2 89 5 06:52:52
1 1 0 973672 5903 197 797 0 0 743 3463 2091 7484 5330 6 1 89 4 06:53:07
1 1 0 973672 6075 211 867 0 0 803 3323 2140 7386 5764 5 1 88 6 06:53:22
1 1 0 973672 5798 230 728 0 0 708 3439 2104 6400 4593 3 1 83 13 06:53:37
2 4 0 973672 6040 264 739 0 0 768 4566 2164 6095 5387 5 2 77 17 06:53:52
1 1 0 972471 6203 244 745 0 0 698 4217 2102 8617 5467 6 1 87 6 06:54:07
1 1 0 972471 5969 201 831 0 0 752 2725 2111 6807 4953 3 1 91 5 06:54:22
1 1 0 972471 6231 203 846 0 0 784 3211 2123 9378 5671 6 2 88 5 06:54:37
1 1 0 972471 6137 197 848 2 0 768 3938 2116 7145 5703 6 1 89 5 06:54:52
kthr memory page faults cpu time
-------- ----------- ------------------------ ------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 1 0 973702 6024 211 812 0 0 829 4291 2121 7420 5245 4 2 90 5 06:55:07
1 3 0 973702 6024 207 823 0 0 768 3915 2127 6652 4609 5 1 89 5 06:55:22
2 1 0 973706 5847 210 820 0 0 761 2728 2127 15225 5386 6 2 87 5 06:55:37
1 1 0 973706 5826 203 901 0 0 811 3899 2135 6504 5511 4 1 90 5 06:55:52
2 1 0 973804 5955 204 798 1 0 734 4132 2138 10950 4844 5 2 88 5 06:56:07
1 1 0 973804 6115 205 777 0 0 767 3609 2074 7281 5489 6 1 89 5 06:56:22
1 1 0 973804 6120 199 865 0 0 786 3104 2129 7056 5580 5 1 90 5 06:56:37
2 2 0 973804 6000 208 780 0 0 726 2238 2093 6618 4527 4 1 91 5 06:56:52
1 1 0 972590 6252 190 805 0 0 665 3033 2097 7608 5095 6 2 87 5 06:57:07
1 1 0 972590 5811 190 856 0 0 716 4066 2114 7235 5703 6 1 89 5 06:57:22
1 1 0 973970 5903 204 833 1 0 872 3964 2137 6856 4624 3 1 91 5 06:57:37
1 1 0 974056 6119 192 828 0 0 761 2678 2112 7353 5583 6 1 88 5 06:57:52
1 1 0 974092 5985 199 799 0 0 726 2237 2105 6649 4947 6 2 88 5 06:58:07
1 1 0 974092 6005 208 730 0 0 692 3116 2077 6080 4953 3 1 92 5 06:58:22
1 2 0 974092 5861 180 685 0 0 622 3815 2050 5826 3960 5 1 89 5 06:58:37
1 1 0 974092 5815 179 714 0 0 650 2926 2063 7320 5300 6 1 89 5 06:58:52
1 1 0 972683 5822 196 776 0 0 598 2915 2096 7347 5244 4 2 90 5 06:59:07
1 1 0 972683 6122 180 732 0 0 683 1936 2067 5790 4029 4 1 90 5 06:59:22
1 1 0 974282 5901 199 816 3 0 837 2968 2186 7592 5506 6 1 88 5 06:59:37
1 2 0 987210 6046 635 754 45 0 2099 10701 2734 27763 6854 6 5 74 15 06:59:52

 

[metzlej]

Can you post the ouput of vmo -L and ioo -L

Jarrett

 

[p5wizard]

post a topas screen also please (first let it run for some time)

HTH,

p5wizard

 

[kjsys1]

VMO -L
------
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 2304K 2304K 4KB pages S
--------------------------------------------------------------------------------
pinnable_frames 2176K 2176K 4KB pages S
--------------------------------------------------------------------------------
maxfree 2K 128 2K 16 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
minfree 1920 2040 1920 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm% 10 20 10 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
minperm 228285 228285 S
--------------------------------------------------------------------------------
maxperm% 20 80 20 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxperm 456571 456571 S
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
maxpin 1843K 1843K S
--------------------------------------------------------------------------------
maxclient% 20 80 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4G-1 uid D
--------------------------------------------------------------------------------
numpsblks 1536K 1536K 4KB pages S
--------------------------------------------------------------------------------
npskill 12K 12K 12K 1 1M-1 4KB pages D
--------------------------------------------------------------------------------
npswarn 48K 48K 48K 0 1M-1 4KB pages D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
pta_balance_threshold 50 50 50 0 99 % pta segment D
--------------------------------------------------------------------------------
pagecoloring 0 0 0 0 1 boolean B
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
mempools 1 1 1 1 16 B
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 256M bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 B
lgpg_size
--------------------------------------------------------------------------------
num_spec_dataseg n/a 0 0 0 B
--------------------------------------------------------------------------------
spec_dataseg_int n/a 512 512 0 B
--------------------------------------------------------------------------------
memory_affinity 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 0 4KB pages D
lgpg_size
--------------------------------------------------------------------------------
large_page_heap_size n/a 0 0 0 8E-1 bytes B
lgpg_size
--------------------------------------------------------------------------------
kernel_heap_psize n/a 4K 4K 4K 16M bytes B
lgpg_size
--------------------------------------------------------------------------------
soft_min_lgpgs_vmpool 0 0 0 0 90 % D
lgpg_size
--------------------------------------------------------------------------------
vmm_fork_policy 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
mbuf_heap_psize n/a 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
strict_maxclient 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
lru_poll_interval 1 0 1 0 60000 milliseconds D
--------------------------------------------------------------------------------
lru_file_repage 0 1 0 0 1 boolean D
--------------------------------------------------------------------------------

n/a means parameter not supported by the current platform or kernel

Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented

Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60

IOO -L
-------
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
memory_frames 2304K 2304K 4KB pages S
--------------------------------------------------------------------------------
minpgahead 2 2 2 0 4K 4KB pages D
maxpgahead
--------------------------------------------------------------------------------
maxpgahead 8 8 8 0 4K 4KB pages D
minpgahead
--------------------------------------------------------------------------------
pd_npages 64K 64K 64K 1 512K 4KB pages D
--------------------------------------------------------------------------------
maxrandwrt 0 0 0 0 512K 4KB pages D
--------------------------------------------------------------------------------
numclust 1 1 1 0 2G-1 16KB/cluster D
--------------------------------------------------------------------------------
numfsbufs 5000 186 5000 1 2G-1 M
--------------------------------------------------------------------------------
sync_release_ilock 1 0 1 0 1 boolean D
--------------------------------------------------------------------------------
lvm_bufcnt 9 9 9 1 64 128KB/buffer D
--------------------------------------------------------------------------------
j2_minPageReadAhead 2 2 2 0 64K 4KB pages D
--------------------------------------------------------------------------------
j2_maxPageReadAhead 128 128 128 0 64K 4KB pages D
--------------------------------------------------------------------------------
j2_nBufferPerPagerDevice 512 512 512 0 2G-1 M
--------------------------------------------------------------------------------
j2_nPagesPerWriteBehindCluster
32 32 32 0 64K D
--------------------------------------------------------------------------------
j2_maxRandomWrite 0 0 0 0 64K 4KB pages D
--------------------------------------------------------------------------------
j2_nRandomCluster 0 0 0 0 64K 16KB clusters D
--------------------------------------------------------------------------------
j2_non_fatal_crashes_system
0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
j2_syncModifiedMapped 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
jfs_clread_enabled 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
jfs_use_read_lock 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
hd_pvs_opn 57 57 S
--------------------------------------------------------------------------------
hd_pbuf_cnt 8960 3904 5440 0 2G-1 I
--------------------------------------------------------------------------------
j2_inodeCacheSize 400 400 400 1 1000 D
--------------------------------------------------------------------------------
j2_metadataCacheSize 400 400 400 1 1000 D
--------------------------------------------------------------------------------
j2_dynamicBufferPreallocation
16 16 16 0 256 16k slabs M
--------------------------------------------------------------------------------
j2_maxUsableMaxTransfer 512 512 512 1 4K pages M
--------------------------------------------------------------------------------
pgahd_scale_thresh 0 0 0 0 200K 4KB pages D
memory_frames
--------------------------------------------------------------------------------

n/a means parameter not supported by the current platform or kernel

Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented

Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60

 

[kjsys1]

Topas Monitor for host: kingbird EVENTS/QUEUES FILE/TTY
Thu Aug 25 10:05:07 2005 Interval: 2 Cswitch 48642 Readch 5792.7K
Syscall 148.9K Writech 3332.3K
Kernel 8.8 |### | Reads 24206 Rawin 0
User 28.8 |######### | Writes 24149 Ttyout 313
Wait 10.9 |#### | Forks 7 Igets 2
Idle 51.5 |############### | Execs 7 Namei 991
Runqueue 4.5 Dirblk 1299
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 1.5
en0 1389.6 1486.4 1442.4 673.6 716.0
lo0 0.0 0.0 0.0 0.0 0.0 PAGING MEMORY
Faults 3567 Real,MB 9215
Disk Busy% KBPS TPS KB-Read KB-Writ Steals 1344 % Comp 47.8
hdisk0 16.0 182.4 34.5 2.0 180.4 PgspIn 0 % Noncomp 52.8
hdisk1 13.5 180.4 34.0 0.0 180.4 PgspOut 3 % Client 0.5
hdisk377 12.0 104.1 18.5 104.1 0.0 PageIn 605
hdisk56 11.0 116.1 18.5 116.1 0.0 PageOut 255 PAGING SPACE
hdisk235 8.5 112.1 18.0 112.1 0.0 Sios 581 Size,MB 6144
% Used 25.8
Name PID CPU% PgSp Owner NFS (calls/sec) % Free 74.1
oracle 874696 4.0 6.9 psoft ServerV2 0
oracle 952622 3.1 13.6 psoft ClientV2 0 Press:
oracle 946752 3.0 10.8 psoft ServerV3 0 "h" for help
oracle 926324 2.7 5.9 psoft ClientV3 0 "q" to quit
sqr 839430 2.4 9.1 psoft
sqr 963296 2.3 9.0 psoft
oracle 960146 2.1 4.8 psoft

 

[p5wizard]

Did you set vm/io tunables with vmtune or vmo/ioo? The program vmtune in /usr/samples/kernel must be exactly matched with your kernel, if not expect problems!

On a 5.2-05 Oracle/SAP LPAR with 4GB, I have minperm%=3, maxperm%=8 and maxclient%=8, but that LPAR uses a lot more than 8% noncomputational (44%). But it is a smaller ORACLE system, less users, and I have no paging activity...

Perhaps your oracle memory parameters (SGA size) make the system behave differently as compared to 5.1? In any case, if Oracle is taking care of DB buffering, any pct of memory that you allow the system to use as LVM buffer - you have maxperm%=20 - is wasting system memory.

HTH,

p5wizard

 

[kjsys1]

We used the vmo and ioo commands to set the tuning parms