aix 5.2 ml3 memory utilization

[rondebbs]

Hello, we are running our 5.2 aix on two different p670s. One is in Texas and handles the eastern half and the other is in California and handles the western half of our business. Both systems shutdown the progress database at around mid-night to do a database backup. After the backup the progress db is brought back up. At that time we have lots of free memory. Users start logging on around 5:00am and memory utilization starts to creep up. At around 10:00 or 11:00 am memory maxes out at about 94% utilized. It used to max out at around 100% but we recently upgraded from 24GB ram to 32GB.

As long as we are below 90% memory utilization we really don't get any page scans, page cycles or page steals in nmon. At about 93% utilization we start seeing many page scans, page cycles and page steals.

The two boxes are configured almost identical but the ioo parameter j2_nBufferPerPagerDevice is at 512 (default) in ca but at 4096 in tx. The tx box gets much fewer page scans, page cycles and page steals even thought both boxes have a similar load. Our plan is to change the ca j2 parameter to match the 4096 in tx.

Another problem that we have is when we try to copy lots of files from on directory to another directory on the same box the cp statement seems to hang all other processes until it completes. All file systems are jfs2. This issue may be more problematic on the ca box then it is on the tx box.

Finally our nightly file system backup on the ca box jumped from 4 hours to 8 hours since we added the additional 8GB of memory.

Below are some screen shots from nmon and ioo. The first nmon at 9:15 shows memory utilization at 66% with no page scans, page cycles and page steals. The second nmon at 10:45 shows memory utilization at 94% and you can see that we start getting page scans, page cycles and page steals.

Any thoughts on how to better manage our memory?

--nmon-v10r---h=Help-------------Host=wfscr04--------Refresh=11secs---09:16.43----------------
+-Memory-Use---------------------Paging------------------------Stats-------------------------+
¦ Physical PagingSpace pages/sec In Out FileSystemCache ¦
¦% Used 66.4% 8.7% to Paging Space 0.0 0.0 Process & 28.5% ¦
¦% Free 33.6% 91.3% to File System 5379.1 911.9 System 37.9% ¦
¦MB Used 21771.9MB 474.3MB Page Scans 0.0 ¦
¦MB Free 10996.1MB 4965.7MB Page Cycles 0.0 Free 33.6% ¦
¦Total(MB) 32768.0MB 5440.0MB Page Steals 0.0 ------ ¦
¦ Page Faults 9902.6 Total 100.0% ¦
¦Min/Maxperm 1567MB( 5%) 18809MB( 57%) note: % of memory ¦
¦Min/Maxfree 120 248 Total Virtual 37.3GB ¦
¦ Pinned 24.0% ¦
+--------------------------------------------------------------------------------------------+



??nmon-v10r???d=Disk-Graph???????Host=wfscr04????????Refresh=10secs???10:45.26????????????????
??Memory-Use?????????????????????Paging????????????????????????Stats??????????????????????????
? Physical PagingSpace pages/sec In Out FileSystemCache ?
?% Used 94.9% 8.8% to Paging Space 0.0 0.0 Process & 55.5% ?
?% Free 5.1% 91.2% to File System 1723.4 849.2 System 39.3% ?
?MB Used 31081.6MB 476.8MB Page Scans 5786211.2 ?
?MB Free 1686.4MB 4963.2MB Page Cycles 22.1 Free 5.1% ?
?Total(MB) 32768.0MB 5440.0MB Page Steals 622.6 ------ ?
? Page Faults 27493.7 Total 100.0% ?
?Min/Maxperm 1567MB( 5%) 18809MB( 57%) note: % of memory ?
?Min/Maxfree 120 248 Total Virtual 37.3GB ?
? Pinned 24.1% ?
??????????????????????????????????????????????????????????????????????????????????????????????


root@wfscr04:/home/nmon#ioo -a
minpgahead = 2
maxpgahead = 128
pd_npages = 65536
maxrandwrt = 0
numclust = 1
numfsbufs = 4096
sync_release_ilock = 1
lvm_bufcnt = 9
j2_minPageReadAhead = 2
j2_maxPageReadAhead = 128
j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
j2_maxRandomWrite = 0
j2_nRandomCluster = 0
jfs_clread_enabled = 0
jfs_use_read_lock = 1
hd_pvs_opn = 233
hd_pbuf_cnt = 30464
j2_inodeCacheSize = 400
j2_metadataCacheSize = 400
root@wfscr04:/home/nmon#
root@wfscr04:/home/nmon#vmo -a
memory_frames = 8388608
pinnable_frames = 6371475
maxfree = 248
minfree = 120
minperm% = 5
minperm = 401255
maxperm% = 60
maxperm = 4815065
strict_maxperm = 1
maxpin% = 80
maxpin = 6710887
maxclient% = 60
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 1392640
npskill = 10880
npswarn = 43520
v_pinshm = 1
pta_balance_threshold = n/a
pagecoloring = n/a
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = 0
spec_dataseg_int = 512
memory_affinity = 1
htabscale = n/a
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = 0
kernel_heap_psize = 4096
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
root@wfscr04:/home/nmon#

 

[RodKnowlton]

Short answer: As long as the "to Paging Space" numbers stay at zero, you don't really have anything to worry about.

Long answer: The AIX Virtual Memory Manager (VMM) paging mechanism serves double duty. It manages virtual memory, as you would expect, but it also manages file caching. Nmon only shows the totals (meaning it doesn't distinguish between virtual memory or file pages) for paging cycles, steals and faults. It does, however, break out the numbers for paging space versus file system. Your posted stats show no paging space activity, so all of the scans, steals, and cycles are file related. The first reading shows zero scans, steals, and cycles because there was still real memory to be allocated and therefore no need to free any cache up.

For more info on the VMM/File Cache relationship, search this forum for "maxperm".

- Rod


IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+

A Simple Code for Posting on the Web

 

[rondebbs]

Yeah, I see what you mean. It looks like "to Paging Space" generally shows 0.0 in and 0.0 out. On the other hand "to File system" usually shows numbers in the thousands.

Do you think it makes sense to lower maxperm% and maxclient% on this 5.2 Progress DB box? Currently both are set to 60. It looks like these changes do not require a reboot or remounting of file systems so they would be fairly easy to test (we have a difficult time with these 24/7 systems when we need to reboot or remount).

Do you think we could benifit by using the jfs2 cio/dio parameter or the rbrw parameter for our file systems. These are more difficult to change as they require a remount but maybe they would help with the problem where the large file system copies hang the system until they are complete.