Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations SkipVought on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
MEM stays high on AIX
- Thread starter mgauteng
- Start date
Can you provide a little more detail.
What do you mean high? What are you looking at?
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
What do you mean high? What are you looking at?
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
- Thread starter
- #3
RodKnowlton
MIS
- Apr 26, 2000
- 1,005
- 0
- 0
mgauteng,
AIX uses all available memory that is not used for programs as file cache.
For the most part, all AIX machines will show all memory in use except right after boot.
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
AIX uses all available memory that is not used for programs as file cache.
For the most part, all AIX machines will show all memory in use except right after boot.
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
- Thread starter
- #5
Thanks for this info, so what causes the server to start swapping when busy and high i/o on the database disk The memory stays at 99% and then we get page faults in and out. lsps -a shows useage at 99% as well. User affected being unable to access the server or when logged in its very slow. Disks are san based on network is fine.
thanks for the input
thanks for the input
RodKnowlton
MIS
- Apr 26, 2000
- 1,005
- 0
- 0
Sounds like you either have too little memory or you need to adjust maxperm.
How much physical memory does the machine have, and what's the output of vmstat (with no arguments)?
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
How much physical memory does the machine have, and what's the output of vmstat (with no arguments)?
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
- Thread starter
- #7
4 procs 4gb mem vm stat shows :
System configuration: lcpu=4 mem=4096MB ent=2.00
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 2 837257 8774 0 0 0 46 142 0 135 3120 1159 6 0 91 3 0.14 6.8
#
vmo output
lru_file_repage = 1
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 80
maxfree = 1088
maxperm = 781407
maxperm% = 80
maxpin = 852463
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 1048576
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 2
minfree = 960
minperm = 195351
minperm% = 20
Thaks for the help
System configuration: lcpu=4 mem=4096MB ent=2.00
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 2 837257 8774 0 0 0 46 142 0 135 3120 1159 6 0 91 3 0.14 6.8
#
vmo output
lru_file_repage = 1
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 80
maxfree = 1088
maxperm = 781407
maxperm% = 80
maxpin = 852463
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 1048576
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 2
minfree = 960
minperm = 195351
minperm% = 20
Thaks for the help
This may help.
The AIX Virtual Memeory anager (VMM) kernel tunables minperm% and maxperm% govern the allocation of pages of physical memory to either computational (process) memory or file (cache) pages.
The AIX O/S uses physical to cache files thereby reducing (in most cases) I/O.
The amount of physical memory being used for file caching can be determined at any point by executing the command
vmstat -v
and viewing the numperm% value.
When memory becomes constrained, with the number of free pages dropping below the minfree value, the VMM scans the page frames and reclaims stale pages.
The rate of page scanning can be examined using the vmstst command (see column with heading sr - scan rate - number of pages scaned per second).
When not enough stale pages can be found, the VMM must begin to page active pages out.
The determination of which pages to page out is regulated by the minperm% and maxperm% settings.
When the value of numperm% (which is dynamic) is greater than maxperm% (Static Setting) the VMM steals only file pages by paging the information in memory back to disk.
The paging of thesefile pages does not cause the suspension of process execution that often results from paging/swapping to page space.
It has the potential of increasing the I/O a bit as file pages must be paged back in if/when needed.
When the value of numperm% drops below minperm%, the VMM steals one page of computational (process) memory for every page of file memory written out. The stealing of computational pages is swapping.
As for the correct initial seetings, there are as many rules of thumb as there are thumbs.
The approach that I have often taken is to set the minperm/maxperm/maxclient settings of 15/50/50 respectively for database servers.
This allows for more of the database files to be cached in physical memory.
The reduction of minperm% from the default of 20% to 15% postpones the onset of swapping.
That said this is an iteractive tuning effort. You will need to watch the system behaviour for swapping and/or excessive I/O. Additionally you should monitor the value of numperm% during your periods of peak utilisation.
I would suggest that after monitoring numperm% for a period that you consider decreasing maxperm% to 10% less than your numperm% value.
Doing so may further reduce the onset of paging.
A final note any excess physical memory can be used for file caching. You may therefore see the value of numperm% to be very high during periods of light applcation load.
The value of maxclient% must be less than or equal to maxperm%. I generally set it to be equal.
As I stated above there is no magic wand or hard set rules to system tuning,it may also be worth getting an evaluation copy of sarcheck as this can help you understand and fix the bottlenecks on your system if your not happy with the trail and error approach.
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
The AIX Virtual Memeory anager (VMM) kernel tunables minperm% and maxperm% govern the allocation of pages of physical memory to either computational (process) memory or file (cache) pages.
The AIX O/S uses physical to cache files thereby reducing (in most cases) I/O.
The amount of physical memory being used for file caching can be determined at any point by executing the command
vmstat -v
and viewing the numperm% value.
When memory becomes constrained, with the number of free pages dropping below the minfree value, the VMM scans the page frames and reclaims stale pages.
The rate of page scanning can be examined using the vmstst command (see column with heading sr - scan rate - number of pages scaned per second).
When not enough stale pages can be found, the VMM must begin to page active pages out.
The determination of which pages to page out is regulated by the minperm% and maxperm% settings.
When the value of numperm% (which is dynamic) is greater than maxperm% (Static Setting) the VMM steals only file pages by paging the information in memory back to disk.
The paging of thesefile pages does not cause the suspension of process execution that often results from paging/swapping to page space.
It has the potential of increasing the I/O a bit as file pages must be paged back in if/when needed.
When the value of numperm% drops below minperm%, the VMM steals one page of computational (process) memory for every page of file memory written out. The stealing of computational pages is swapping.
As for the correct initial seetings, there are as many rules of thumb as there are thumbs.
The approach that I have often taken is to set the minperm/maxperm/maxclient settings of 15/50/50 respectively for database servers.
This allows for more of the database files to be cached in physical memory.
The reduction of minperm% from the default of 20% to 15% postpones the onset of swapping.
That said this is an iteractive tuning effort. You will need to watch the system behaviour for swapping and/or excessive I/O. Additionally you should monitor the value of numperm% during your periods of peak utilisation.
I would suggest that after monitoring numperm% for a period that you consider decreasing maxperm% to 10% less than your numperm% value.
Doing so may further reduce the onset of paging.
A final note any excess physical memory can be used for file caching. You may therefore see the value of numperm% to be very high during periods of light applcation load.
The value of maxclient% must be less than or equal to maxperm%. I generally set it to be equal.
As I stated above there is no magic wand or hard set rules to system tuning,it may also be worth getting an evaluation copy of sarcheck as this can help you understand and fix the bottlenecks on your system if your not happy with the trail and error approach.
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
Hi mike,
I don't find anything about strict_maxperm in your post.
It was set to 0 in one of my servers ( vmtune AIX 4.3.3 )and the pages allocated for buffer files grows more than maxperm% .
When I set it to 1, maxperm% has never been exceeded.
Now with vmo, strict_maxperm still acts the same way or not ?
Ali
I don't find anything about strict_maxperm in your post.
It was set to 0 in one of my servers ( vmtune AIX 4.3.3 )and the pages allocated for buffer files grows more than maxperm% .
When I set it to 1, maxperm% has never been exceeded.
Now with vmo, strict_maxperm still acts the same way or not ?
Ali
In 5.3 setting the strict_maxperm option to 1 places a hard limit on memory used for persistant cache and makes maxperm the upper limit for file cache.
I'd give it a wide-berth as improper use of stric_maxperm can cause more problems as it changes the VMMM method of page replacement.
I believe the only difference is an increase in the default value of minfree to 960 per pool and maxfree to 1088.
There was a problem with strict_maxperm in 4.3.3 that caused page-replacement to run for other page types.
see
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
I'd give it a wide-berth as improper use of stric_maxperm can cause more problems as it changes the VMMM method of page replacement.
I believe the only difference is an increase in the default value of minfree to 960 per pool and maxfree to 1088.
There was a problem with strict_maxperm in 4.3.3 that caused page-replacement to run for other page types.
see
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
RodKnowlton
MIS
- Apr 26, 2000
- 1,005
- 0
- 0
mgauteng,
You really need to add memory to that machine, if you can.
The AVM number in vmstat is the number of active 4K pages of virtual memory.
That 837257 figure means that your programs are using (837257 * 4 / 1024) = 3270.54MB, which is almost 80% of your physical memory.
I'd do minperm%=15, maxperm%=20, maxclient%=20 to start. That should keep file caching mostly out of your programs' way. That should have a dramatic effect on the perceived speed of the system (i.e. it'll feel more responsive to interactive users).
The problem then becomes that there's only 800MB available for file page caching, which could hurt DB2 performance, depending on the nature of your data and queries.
If you add memory, and you should if at all possible, you'll need to adjust maxperm%/maxclient% upward appropriately. I'd probably just raise them by 5 each day until I get responsiveness complaints, then back it down 5.
HTH,
Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
You really need to add memory to that machine, if you can.
The AVM number in vmstat is the number of active 4K pages of virtual memory.
That 837257 figure means that your programs are using (837257 * 4 / 1024) = 3270.54MB, which is almost 80% of your physical memory.
I'd do minperm%=15, maxperm%=20, maxclient%=20 to start. That should keep file caching mostly out of your programs' way. That should have a dramatic effect on the perceived speed of the system (i.e. it'll feel more responsive to interactive users).
The problem then becomes that there's only 800MB available for file page caching, which could hurt DB2 performance, depending on the nature of your data and queries.
If you add memory, and you should if at all possible, you'll need to adjust maxperm%/maxclient% upward appropriately. I'd probably just raise them by 5 each day until I get responsiveness complaints, then back it down 5.
HTH,
Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
With lru_file_repage you don't need to set maxclient%=20 - but I have seen conflicting docuements on this. Some continue to set maxclient% and some leave the default. I have left mine at the default for Oracle 10g on AIX 5300-04, but will continue to monitor and if needed I would lower maxclient% if the need arises. So far it hasn't. I have nmon collecting every 15 minutes and every 30 seconds during a busy hour.
vmstat -v will give you more info and will help you determine if you need to change pbufs, etc.
Also, you don't want numperm% below minperm%.
Good Luck!
vmstat -v will give you more info and will help you determine if you need to change pbufs, etc.
Also, you don't want numperm% below minperm%.
Good Luck!
- Thread starter
- #14
hi guys thanks for the updates and the picture is getting clearer now .Just one question when running topas or memory usage after the aplied changes to the memory tuning parameters , will I get less memory beig used?how will know the difference besides not being called by users of a slow system ?
Thanks
Thanks
RodKnowlton
MIS
- Apr 26, 2000
- 1,005
- 0
- 0
In topas, you'll still see all memory being used, but you should the Comp and Noncomp percentages change, with Comp going up and Noncomp dropping. What tuning calls perm pages, topas calls noncomp, so under full load the noncomp percentage should drop to roughly maxperm%.
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
- Rod
IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+
Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.
Couple of useful docs about tuning
lru_file_repage & lru_poll_interval
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
lru_file_repage & lru_poll_interval
Mike
"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
- Status
