Paging when there's plenty of free physical memory

[headaix]

I am seeing sporadic paging, very minimal, but somewhat confusing. The server in question has about 18GB of RAM free. Is this a VMO setting or settings that needs to be addressed?

Minperm is 10%, Max is 20%.
Minfree is 1920, Max is 2048
16 CPUs, Power5
48GB RAM
40GB Paging space is about 2% full.

Thanks in advance,

Christopher

$ vmstat 1 60
System Configuration: lcpu=16 mem=49152MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
10 1 5689000 4790883 0 0 1 222 762 0 2777 86801 45892 15 10 73 2
4 0 5690358 4789508 0 0 0 0 0 0 81188 112913 57964 16 19 64 1
6 1 5689005 4790860 0 1 0 0 0 0 88563 133802 69844 25 16 57 2
69 1 5690373 4789487 0 0 0 0 0 0 81797 94043 56633 28 12 59 0
16 0 5689052 4790808 0 0 0 0 0 0 79614 127343 53118 39 14 46 2
7 0 5689053 4790803 0 0 0 0 0 0 70606 116498 57794 37 11 50 1
12 0 5689053 4790803 0 0 0 0 0 0 146315 221129 84863 46 23 29 1
7 0 5689380 4790460 0 3 0 0 0 0 210448 181979 77364 46 22 31 1
4 1 5689305 4790487 0 12 0 0 0 0 126311 214927 104567 38 26 32 3
7 0 5689931 4789864 0 8 0 0 0 0 105493 125701 66100 26 17 56 2
14 1 5689305 4790470 0 0 0 0 0 0 73550 93349 49182 27 10 60 3
7 1 5689053 4790742 0 0 0 0 0 0 116141 126674 68502 33 17 48 2
15 1 5689433 4790350 0 1 0 0 0 0 125036 180120 97831 30 25 44 1
9 1 5689443 4790332 0 0 0 0 0 0 100380 216343 77309 34 19 44 3
6 0 5689147 4790635 0 0 0 0 0 0 62870 134440 70743 20 15 63 2
4 0 5689148 4790629 0 0 0 0 0 0 74628 144767 90160 18 17 64 1
0 0 5689148 4790629 0 0 0 0 0 0 93637 152679 84721 23 19 58 1
6 0 5689148 4790628 0 0 0 0 0 0 66521 116175 68113 19 12 68 2
3 0 5690505 4789260 0 0 0 0 0 0 72123 114387 60731 16 12 71 1
39 0 5689148 4790616 0 0 0 0 0 0 91088 158590 72082 29 15 54 2
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
214 0 5689148 4790567 0 45 0 0 0 0 161567 290584 102220 45 25 29 1
28 0 5689148 4790567 0 0 0 0 0 0 209921 232839 96944 45 27 27 1
10 0 5689148 4790567 0 0 0 0 0 0 148075 228944 102303 39 24 35 1
8 1 5689148 4790567 0 0 0 0 0 0 148825 203350 93353 41 18 40 1
10 1 5689148 4790567 0 0 0 0 0 0 114896 164206 73125 42 18 39 1
6 0 5689148 4790563 0 0 0 0 0 0 128580 200794 79046 34 19 46 2
6 0 5689148 4790562 0 0 0 0 0 0 174687 233294 111183 40 26 34 1
32 1 5689148 4790554 0 1 0 0 0 0 263842 240376 74498 66 30 4 0
18 1 5690502 4789190 0 0 0 0 0 0 183665 220314 69937 70 22 8 0
30 2 5689150 4790538 0 2 0 0 0 0 152371 240374 91266 64 25 9 1
49 2 5688705 4790915 0 29 0 0 0 0 128926 233181 62885 66 23 9 2
28 1 5689006 4790629 0 1 0 0 0 0 135084 182301 63624 69 19 11 1
28 0 5687612 4792012 0 4 0 0 0 0 221735 229300 54125 69 25 6 0
32 0 5688962 4790650 0 0 0 0 0 0 185268 209386 62989 74 23 3 0
26 1 5687612 4791988 0 1 0 0 0 0 151501 215598 66597 74 24 2 0
43 0 5687617 4791972 0 0 0 0 0 0 151638 200042 63181 75 20 5 0
19 1 5688131 4791424 0 0 0 0 0 0 103867 188338 64905 54 26 19 1
32 0 5688438 4791102 0 0 0 0 0 0 138112 221022 70382 57 23 18 2
7 0 5687941 4791578 0 17 0 0 0 0 157039 227564 83852 40 23 34 3
10 2 5688975 4790551 0 0 0 0 0 0 129058 176104 73738 34 19 43 3
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
6 0 5687918 4791600 0 0 0 0 0 0 84086 200427 63477 28 15 55 2
12 1 5687647 4791870 0 0 0 0 0 0 140031 184016 83682 33 18 48 1
7 1 5687865 4791623 0 0 0 0 0 0 123722 188798 69026 27 18 52 3
12 0 5687865 4791613 0 0 0 0 0 0 74319 223952 76704 28 21 50 1
4 0 5689219 4790241 0 0 0 0 0 0 117451 170397 96930 32 19 49 0
7 0 5688980 4790464 0 0 0 0 0 0 108093 152337 74727 33 22 43 2
13 2 5690359 4789059 0 0 0 0 0 0 153091 235661 81974 40 29 30 2
7 2 5690103 4789291 0 0 0 0 0 0 135637 237858 90375 43 27 26 4
39 1 5687838 4791538 0 0 0 0 0 0 154284 228212 66854 66 27 6 1
9 2 5688200 4791164 0 0 0 0 0 0 126402 173690 63962 52 16 30 2
10 0 5688314 4791038 0 0 0 0 0 0 134504 154242 62688 53 17 28 2
144 1 5687613 4791738 0 0 0 0 0 0 120867 235069 97372 47 27 25 1
28 0 5687613 4791736 0 0 0 0 0 0 154059 231345 86649 65 22 12 1
26 0 5687613 4791736 0 0 0 0 0 0 130003 182467 88973 43 18 37 1
9 0 5687613 4791730 0 0 0 0 0 0 122525 163768 84480 40 16 42 2
41 1 5687613 4791726 0 1 0 0 0 0 85940 177291 73370 34 13 50 3
11 1 5687627 4791711 0 1 0 0 0 0 140346 167367 96369 28 21 49 2
51 0 5688998 4790330 0 0 0 0 0 0 104896 210303 100661 37 30 31 2
7 2 5687628 4791696 0 0 0 0 0 0 157977 207420 86957 50 25 22 4
21 1 5687629 4791692 0 0 19 0 0 0 192578 243293 100320 48 28 21 2

 

[khalidaaa]

I don't see any paging in your output! pi and po are always zeros?!?!

Your (CPU) run queue reached 214 at one time which is relatively high for 16 CPUs but it is not constant so it should be fine!

Regards,
Khalid

 

[DukeSSD]

Look again khalidaaa it is paging.
Even on the 214 runnable threads line you mention it paged in 45 pages!

Yes headaix, it could do with some (more) tuning.

Even at a first glance, paging with that amount of free memory is not normal for AIX.

Typically a well tuned AIX system will keep just about all memory in use, even if it is not paging.

From your settings it has been tuned but not too well.

Please supply the oslevel -r because the settings are somewhat dependent on the level and ML / TL.

 

[headaix]

5200-08 from oslevel -r

here's the output of vmo -a

cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 0
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 20
maxfree = 2048
maxperm = 2421932
maxperm% = 20
maxpin = 10066330
maxpin% = 80
mbuf_heap_psize = 4096
memory_affinity = 1
memory_frames = 12582912
mempools = 1
minfree = 1920
minperm = 1210965
minperm% = 10
nokilluid = 0
npskill = 81920
npswarn = 327680
num_spec_dataseg = 0
numpsblks = 10485760
pagecoloring = n/a
pinnable_frames = 11948101
pta_balance_threshold = n/a
relalias_percentage = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 1
v_pinshm = 0
vmm_fork_policy = 0

and ioo -a

hd_pbuf_cnt = 2992
hd_pvs_opn = 16
j2_atimeUpdateSymlink = 0
j2_dynamicBufferPreallocation = 16
j2_inodeCacheSize = 400
j2_maxPageReadAhead = 128
j2_maxRandomWrite = 0
j2_maxUsableMaxTransfer = 512
j2_metadataCacheSize = 400
j2_minPageReadAhead = 2
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_nRandomCluster = 0
j2_nonFatalCrashesSystem = 0
j2_syncModifiedMapped = 1
jfs_clread_enabled = 0
jfs_use_read_lock = 1
lvm_bufcnt = 20
maxpgahead = 8
maxrandwrt = 0
memory_frames = 12582912
minpgahead = 2
numclust = 1
numfsbufs = 600
pd_npages = 65536
pgahd_scale_thresh = 0
sync_release_ilock = 0

 

[khalidaaa]

Well, to my knowledge, paging is the main principle of virtual memory so this shouldn't be an indicator of memeory bottleneck unless it is paging more often!

This is part of the course i attended two years ago on performance tuning:

When determining if a system may be short on memory or if some memory tuning
needs to be done, run the vmstat command over a set interval and examine the pi
and po columns. These columns indicate the number of paging space page-ins/per
second and the number of paging space page-outs/per second. If the values are
constantly non-zero, there is most likely a memory bottleneck. Occasional non-zero
values don’t necessarily indicate a problem since paging is the main principle of virtual
memory.
Notice the high I/O wait in the output and also the number of threads on the blocked
queue. The I/O is likely due to the paging in/out from paging space.

One way to determine the appropriate amount of RAM for a system is to look at the
largest value for avm as reported by vmstat. Multiply that by 4 KB to get the number of
bytes and then compare that to the number of bytes of RAM on the system. Ideally, avm
should be smaller than total RAM. If not, some amount of virtual memory paging will
occur. How much will depend on the difference between the two values. Remember, the
idea of virtual memory is that it gives us the capability of addressing more memory than
we have (some of the memory is in RAM and the rest is in paging space). But, if there is
far more virtual memory than real memory, this could cause excessive paging which
then results in delays. If avm is lower than RAM, then paging space paging could be
caused by RAM being filled up with file pages. In that case, tuning the
minperm/maxperm values could reduce the amount of paging space paging.
As a rule of thumb, memory is over committed if (%avm + 5%) > 100%
where:
%avm=(avm*4096/RAM installed)*100
The value for RAM installed can be found with the bootinfo -r command.
(Note: this rule of thumb might not apply for database servers using raw logical volumes
instead of file systems).

In this case the %avm=(5689148*4096/49152*2048)*100=45.21%

So it is hardly half of the RAM is being used!

Regards,
Khalid

 

[headaix]

Thanks for the thorough explanation, Khalid. This makes sense.

I know that VM works by "touching" pages from time to time, which can appear as sporadic paging. Just wanted to make certain.

Best,

headaix