Memory Utilization 1

[mezanine]

I have been watching my system closely these last few days and noticed some interesting things. Users were complaining that the system was slow. I have x4 2.8 ghz xeon processors, 4GB of RAM. With these specs I should not run into any problems, however the system is indeed slow. I have run TOP, IOSTAT, FREE, and VMSTAT to try and isolate the problem, all of which point to memory utilization. I rebooted the system yesterday and everthing ran fine, and my memory utilization was moderate. After looking at the system this morning the available memory has shrunk to almost nothing. It appears that a program is grabbing memory and not releasing it, we are also using entirely way to much swap.
Does anyone have any suggestions on how to determine were the memory is alocated?

11:32am up 1 day, 5:03, 182 users, load average: 1.11, 1.08, 1.26
832 processes: 831 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.3% user, 0.4% system, 0.0% nice, 98.3% idle
CPU1 states: 0.1% user, 0.1% system, 0.0% nice, 99.3% idle
CPU2 states: 0.1% user, 0.1% system, 0.0% nice, 99.3% idle
CPU3 states: 1.2% user, 6.0% system, 0.0% nice, 92.1% idle
Mem: 3863192K av, 3858096K used, 5096K free, 124K shrd, 332748K buff
Swap: 2096400K av, 125296K used, 1971104K free 3212548K cached

 

[Castor66]

In 'top' press 'M' (uppercase 'm') and the process on top will be the process that's hogging the most memory, e.g. on one of my DB servers I get :

 18:47:50  up 175 days,  1:21,  1 user,  load average: 0.33, 0.26, 0.16
65 processes: 64 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  799.2%
           cpu00    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu01    0.0%    0.0%    0.0%   0.2%     0.0%    0.0%   99.8%
           cpu02    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu03    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu04    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu05    0.2%    0.0%    0.0%   0.0%     0.0%    0.0%   99.8%
           cpu06    0.2%    0.0%    0.0%   0.0%     0.0%    0.0%   99.8%
           cpu07    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
Mem:  3934208k av, 3502572k used,  431636k free,       0k shrd,   75660k buff
                   2644212k actv,   17236k in_d,   41628k in_c
Swap: 2044072k av,       0k used, 2044072k free                 3143756k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
24238 mysql     15   0 28796  28M  2228 S     0.2  0.7  1773h   2 mysqld
 1651 xfs       15   0  3012 3012   608 S     0.0  0.0   0:00   4 xfs

Now I can see that mysqld is using the most memory. I cna then use this info to start looking at why I'm running out of memory (as you can see - I'm not)

Give that a shot and see what happens. Tracking down a program which may have a memory leak is a nightmare.

 

[mezanine]

Thanks for the tip, unfortunatley nothing pops up hogging mem, which leads me to believe that their may be a memory leak somewhere. I know you said it could be a nightmare, do you have any starting points?

Thanks

 

[Yrrk]

The initial output you posted above doesn't indicate any problem IMO. It's not unusual to see all memory being utilized and your swap is mostly free.

If you had a memory leak, it's usually pretty easy to determine with what because using TOP with shift-M will show the process consuming more and more memory over time. Eventually swap utilization grows, swap free dwindles, and the system begins to crawl as things begin "swapping". If you reach this state it's usually obvious by doing a vmstat 5 and watching the "si" and "so" columns. On a normal system it's not unsual for them to be 0 most of the time. On a swapping system you'll see your disk LED's pegged, and your vmstat output showing large values in si and so.

Also with a memory leak you'll often see a reboot fix it for a while.. then the system performance begins to degrade over time until swapping occurs and it can become unresponsive. Again easy to put all these indicators together.

cat /proc/meminfo is useful to see how memory is being used overall and googling it will teach you how to read whats in it.

But to get specific process level memory info TOP is ok but i always prefer ps ultimately. e.g.

ps awxfmo f,user,uid,pid,ppid,pri,ni,size,rss,share,stat,%cpu,$mem,cpu,time,command

Can be useful in a lot of ways. Play with using ps.. you can see how much memory things are using, cpu, whatever you want really.

And don't forget sar. sar can show you whats been going on with memory and/or swapping over time..

sar -B is great. You can see if you're getting a lot of page ins and page outs over time. Again an indicator of whether things are degrading and swapping is occuring (lots of page ins and page outs can indicate swapping).

sar -r shows you general memory info over time.

All in all if you are not observing this degrading type of behavior then you do not have a memory leak.

 

[Kozusnik]

Also by running top -c, you can see the full command to the program you're running. Maybe you've got a program creating multiple child processes and they are eating up your memory. This is not unusual, but can cause a problem.
While running top, you may want to press "i" to see the processes actually using cpu. It's a starting point anyway.

Mark