Memory Percent Used increases 4

[thechickster]

I need some guidence, I am new to AIX realm and have been tasked to figure out why several of our servers are reaching 80 to 90 percent utilzation over a course of 4 weeks, we upgraded the servers to AIX 5.2 (no ML), obviously when we reboot the servers the go to 20%.

Need help
Each Server is a Web Server running Netscape

%-)

 

[aixmurderer]

I assume you are talking about paging space usage.

From 5.1 we found PS management being pretty good, the only problem we encounter now is that Java apps aren't doing garbage collection as it should, so we get paging space creep, requiring a weekly reboot.

To prove that it was the java apps I did a stop of the apps, PS % usage dropped by over 20%. So you could apply the same technic to find the culprit application.

IBM Certified Confused - MQSeries
IBM Certified Flabbergasted - AIX 5 pSeries System Administration
MS Certified Windblows Rebooter

 

[raztaboule]

sorry for my english

run :
svmon -P | grep -p Pid
and you will see the paging space pages use by process in the column pgsp.

If any perf specialist can confirm this it will be nice.

bye.

 

[MoshiachNow]

I would also check for zombies,they take up on the PS:

ps -ef |grep defunct

"Long live king Moshiach !"

http://www.7for70.com/

 

[theycallmetim]

If you are not looking at paging space, but memory like you say, this is normal. The more files you touch, the more they will reside in memory (persistant storage) and gradually "use" up all your memory. This is not a big deal. When working memory is needed, you file cache is then released. Try

svmon -G

See where the memory is going. working or pers? If pers? Don't worry about it unless you start to use paging space, then there are some tuning options you can use to deal with memory better.

 

[thechickster]

All yours answers have helped.

When using the svmon -g this is the output ( after the machine was rebooted)
Size inuse free pin virtual
memory 786432 191225 595207 56860 142493
pgspace 108344 1274

work pers clnt lpage
pin 56860 0 0 0
in use 142516 48709

before the box was rebooted the Memory used was 79%
now the memory is 15%

so are you saying 83% to 92% of memory used is fine? me personally being new to this relam does not sound normal but I'm new with a big learning curve.(windblows env)
but I'm still concerned that these are production Web boxes that need to be rebooted every three weeks or so.


%-)

 

[theycallmetim]

Yes.

From the output above, you have 3 GB of memory.

556 Meg is used for running applications (working storage)
109 Meg is used for file system cache.

When your memory usage gets large, do this command again (svmon -G)
if the increase is in the pers (in use) column, don't worry about it.(unless you are actively paging) If it's in the work(in use) column, then you may need to buy more memory. The combination of these 2 numbers is the total memory used you are seeing. This is normal to increase in time as you will be accessing more files and they will stay in memory. They will automatically be flushed out if your programs need more memory. Please let your system run for a while and post the output again.

Thanks

 

[uptodate]

Run:

svmon -Pau 10 | more

Look in the "Type" column for "work" and in the "Description" column for "private" and check how many 4kb pages are used under the "Pgspace" column. This is the minimum number of working pages this segment is using in all of virtual memory. A "Pgspace" number that grows but never decreases could indicate a memory leak.

 

[thechickster]

I kinda understand the memory thing, but why does the memory increase and stay after a mksysb on the box. We have to reboot the box. 5.2.2

thanks

 

[theycallmetim]

mksysb reads a lot of files. You shouldn't have to reboot. Before you reboot show the command svmon -G again.

 

[thechickster]

SIZE INUSE FREE PIN VIRTUAL
Memory 786432 780673 5759 59199 168702
pg space 1073152 1756

work pers clnt lpage
pin 59199 0 0 0
in use 168646 612027 0 0

We went from 42% to 99% used since the mksysb
being new to AIX , having 4G of RAM 4 procs Pseries 660, something just doesn't seem right.

I appreciate all your help
C

 

[raztaboule]

can you please run
/usr/samples/kernel/vmtune
and
lsps -a
and post the outputs.

what kind of application are you running ? oracle ?

 

[MoshiachNow]

Also,still did not get the output of the defunct check.
Thanks

"Long live king Moshiach !"

http://www.7for70.com/

 

[theycallmetim]

This is purely the way AIX works. You see that you are only using 658 Meg of memory for your programs and 2.9 GB as persistant storage. NOT A BIG DEAL. Just make sure your vm parameters are set so you do not page out to paging space, flush the persistant storage.

since you only need about 20% of memory for working storage, I'd be save and set

vmo -o minperm%=20
vmo -o maxclient%=50
vmo -o maxperm%=50

You are NOT running out of memory. What command are you using to see this? The command you are doing must be giving you the combination of working and persistant storage. After a mksysb you are reading in a lot of files and they are put into the "buffer" (persistant storage). This will automatically get cleaned up if your programs need more memory.

 

[thechickster]

wlmstat

ClASS CPU MEM DKIO
Unmanaged 0 6 0
Default 11 6 0
Shared 0 1 0
System 1 93 0
TOTAL 12 100 0

 

[thechickster]

Can I flush the persistant storage?

I Believe we have 4G of PS

 

[raztaboule]

do a
vmstat 1
and check if there is a lot of pgin/pgout (pi / po columns )
if yes you need memory tunning , if not , take a coffee and have fun surfing on the web

why do you think yoy have a problem ? is someone complaining ?

 

[WiccaChic]

Hi raztaboule. I have always been fuzzy on this and I guess this type of thing is knowledge you gain with experience. What would you categorize as a lot of pi/po?

 

[raztaboule]

I'am not so experienced in it. but i always heard that no paging should occur..... i don't think there is really rules just what you use to see on your system and try to get it better.

 

[theycallmetim]

the syncd flushes the persistant storage.

It writes all dirty pages to disk every 60 seconds by default. You can decrease the interval if you want. But to fully answer your question, no, you can't flush the cache and see your memory stats decrease in wlmstat. Everything appears to be fine as long as you are not paging to paging space.

svmon will give you a more detailed snapshot of your memory usage and who is using it.