AIX vsz vs rss and a memory consumption issue

[mensafool]

Hey guys,
I have been tasked with trying to figure out why an application keeps consuming more and more memory but I've kind of hit a brick wall. I am one of the java programmers that develops solutions for this AIX LPAR and it seems like the process in question keeps consuming more and more memory. I'm talking to the vendor and seeing if I can get some consultant help but I was curious what your opinions are of the following processes. We are running an egate application and I am wondering the following thing, what could I do to solve the memory consumption issue on this box? Killing the process and restarting the process? Any tips or troubleshooting tips to try? Thanks in advance.

Yes we are running the same process a few times which is required in the eGate application. My virtual memory consumption is a major issue here right? not the physical memory?

COMMAND %MEM RSS VSZ %CPU USER PID
stcms.exe 3 524716 2405696 0.6 jcap 581802
is_domain1 3 465788 620680 1.2 jcap 663588
stcms.exe 3 435664 4290360 0.7 jcap 430290
stcms.exe 2 375444 3864648 0.6 jcap 552976
stcms.exe 2 366000 3740976 0.4 jcap 393408

 

[sjm2]

Why do you think you have a memory issue? Memory is there to be used.

 

[mensafool]

4gb of memory consumption is a little high isnt it? The physical memory isn't a concern of mine as much as the Virtual memory consumption. From what I've read about VSZ and the way it is calculated, it doesn't take into account shared libraries accross processes which maybe why I am getting the bloated numbers.

The bigger concern (you are correct) is that this AIX server is constantly paging and never stops.

MEMORY
Real,MB 15168
% Comp 81
% Noncomp 18
% Client 18

PAGING SPACE
Size,MB 20992
% Used 76
% Free 24

 

[sbix]

mensafool,
I used something like this to detect the physical memory usage:

#!/usr/bin/ksh
GetTop10 () {
ps gl |awk '{print $4" "$11" "0}'|sort -nr +1|tr " " ":"|head
}
ME=$(basename $0)
typeset -i counter=0 oldcounter=0 MEM=0
OldTop10=$(GetTop10)
TRESHOLD=1048576
while true
do
sleep 600
Top10=""
for i in $(GetTop10)
do
echo $i |IFS=":" read PID MEM counter
old_i=$(echo $OldTop10| tr " " "\n"|grep "$PID:")
if [ x$old_i = x ];then
Top10=$Top10" $i"
else
echo $old_i|IFS=":" read PID MEM oldcounter
counter=$oldcounter+1
Top10=$Top10" $PID:$MEM:$counter"
if [ $MEM -gt $THRESHOLD -a $counter -gt 6 ];then
dax=$(date +%d/%m%/%Y-%T)
echo "$dax - Proc. PID=$PID - is using $MEM kb from ${counter}0 minutes"
fi
fi
done
OldTop10=$Top10
done

This script was intended to find which processes were using more than 1Gb for more than an hour.
You can adjust it to "search & kill" (and eventually restart) the huge ones

 

[mrn]

You answered your own question in the " I am one of the java programmers that develops solutions for this AIX LPAR"

I've yet to see in 20+ years as an AIX Admin a first time java deployment that didn't leak memory like a BP oil well in the gulf of mexico (Sorry couldn't think of a better metaphor)

http://www.ibm.com/developerworks/aix/library/au-javaonaix_memory.html

also google for "aix memory leaks post" top link





Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[mensafool]

Hey guys, I did quite a bit more research and you will be delighted to hear that it really has nothing to do with Java. The application we are running, which is a delivered application currently has a defect which we are trying to remedy. It really has nothing to do with Java, although I share your sentiments that it is an easy language to code without memory leaks. I would hope that the hotfixes we received from the vendor will solve this but it was more of a question about how to appropriately set metrics and set expectations for memory allocation and release in AIX. Should I expect applications to consume and release memory as in the Windows OS or is AIX a different beast and should I forget everything about Windows and Memory Management.

Thanks again for the replies.

 

[sjm2]

Q Should I expect applications to consume and release memory as in the Windows OS or is AIX a different beast and should I forget everything about Windows and Memory Management.

A False question. It is up to an application in either OS (or indeed any) to give up memory if it id no longer needed.

 

[capsusr]

Can you provide a list of the hot fix or ESR (I am assuming eGate 4.5.x)

Greatly appreciated

 

[mensafool]

After our initial investigation it appeared that each of the 3 ESR's also had prerequisites in order to install each of the 3 ESRs. Such that the list is currently:

1. 6676321 Repository
2. 6802097 Repository
3. 6863213 eDesigner
4. 109110 eDesigner
5. 6718549 eDesigner
6. 6721126 LH
7. 6799649 LH
8. 6835732 eManager
9. 2170230 LH JMS

Careful with the Logical Host ESR's they performed a CHOWN on our directories after the install to a different user.

 

[mensafool]

We are on 5.1.2

 

[capsusr]

Thanks so much.

I had seen issues with JVM leaks on 5.1.2 and we had a large issue with it crashing at a previous site.

Have to preface I am not a Unix guru.

We broke out to multiple LH and that stopped the crashing.

This if very helpful thanks again,