facing problem with momory usage 2.6.9-55.ELsmp

[sudhakar1217]

Hi,

i am using linux Red Hat Enterprise Linux AS release 4 (Nahant Update 5)

kernal 2.6.9-55.ELsmp

i am having 32 GB of RAM.


total used free shared buffers cached
Mem: 32911084 32850520 60564 0 601740 5205144
-/+ buffers/cache: 27043636 5867448
Swap: 2097144 3604 2093540


if i take all the process memory usage its around 10 GB only.

i am not able to check what happend to the rest of the momory.

can some body help me how to investigate this issue

Regards,
Sudhakar

 

[Annihilannic]

You probably have some HugePages reserved. Check the last few lines of /proc/meminfo.

If you need to adjust this, they are configured using the vm.nr_hugepages option in /etc/sysctl.conf.

These are often configured for Oracle databases or similar to reserve large areas of memory. Because they are so large they require a lot less kernel overhead to manage the memory, and free up "Low" memory for the kernel to use for other purposes.

Annihilannic.

 

[sudhakar1217]

Hi Annihilannic,

find the below memorey info under

/proc/meminfo

i am not able to understand what need to refer form this

MemTotal: 32911084 kB
MemFree: 39452 kB
Buffers: 485692 kB
Cached: 4106076 kB
SwapCached: 0 kB
Active: 7562560 kB
Inactive: 933932 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32911084 kB
LowFree: 39452 kB
SwapTotal: 2097144 kB
SwapFree: 2093564 kB
Dirty: 784 kB
Writeback: 0 kB
Mapped: 4109388 kB
Slab: 552328 kB
CommitLimit: 18552684 kB
Committed_AS: 22398220 kB
PageTables: 15424 kB
VmallocTotal: 536870911 kB
VmallocUsed: 264476 kB
VmallocChunk: 536605691 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

 

[Annihilannic]

Well, it seems my HugePages guess was wrong, since you have none allocated.

However the Committed_AS does seem to be high, maybe that is where the unaccounted memory has gone:

Committed_AS: The amount of memory presently allocated on the system.
              The committed memory is a sum of all of the memory which
              has been allocated by processes, even if it has not been
              "used" by them as of yet. [b]A process which malloc()'s 1G
              of memory, but only touches 300M of it will only show up
              as using 300M of memory even if it has the address space
              allocated for the entire 1G.[/b] This 1G is memory which has
              been "committed" to by the VM and can be used at any time
              by the allocating application. With strict overcommit
              enabled on the system (mode 2 in 'vm.overcommit_memory'),
              allocations which would exceed the CommitLimit (detailed
              above) will not be permitted. This is useful if one needs
              to guarantee that processes will not fail due to lack of
              memory once that memory has been successfully allocated.

What application(s) are you running on this system?

Annihilannic.

 

[sudhakar1217]

applicaiton like following

weblogic
mq server
java
our webapplicaiton
apache http server
etc...

find the below output for more details

$ ps -e -o vsz,uid,pid,args,etime,stime,time |sort -rn
2919128 10661 29820 /opt/bea/wls/910 16-05:15:05 Oct04 04:42:17
2833916 10661 31561 /opt/bea/wls/910 16-05:12:49 Oct04 00:32:24
1318000 0 8145 ../jre/bin/java 114-11:16:58 Jun28 00:03:10
966460 10661 26271 /opt/bea/wls/910 16-10:59:52 Oct04 00:03:48
130824 10602 25860 /mqlocal/opt/IBM 102-18:53:45 Jul09 00:04:43
96396 99 26200 /opt/bea/apache/ 07:45 12:55 00:00:00
96376 99 26188 /opt/bea/apache/ 07:52 12:55 00:00:00
96372 99 23622 /opt/bea/apache/ 11:15 12:52 00:00:00
96368 99 26180 /opt/bea/apache/ 07:53 12:55 00:00:00
96368 99 23606 /opt/bea/apache/ 11:17 12:52 00:00:00
96360 99 23621 /opt/bea/apache/ 11:15 12:52 00:00:00
96352 99 31211 /opt/bea/apache/ 02:15 13:01 00:00:00
96348 99 31206 /opt/bea/apache/ 02:16 13:01 00:00:00
96340 99 27900 /opt/bea/apache/ 05:42 12:57 00:00:00
96328 99 25334 /opt/bea/apache/ 09:39 12:53 00:00:00
78152 0 8161 su candle 114-11:16:57 Jun28 00:00:00
74028 10602 24973 /mqlocal/opt/IBM 102-18:54:32 Jul09 00:00:03
72228 99 6567 LLAWP /opt/bea/a 30-09:12:28 Sep20 00:00:02
68776 10600 31291 amqzxma0 -m BT.Q 109-23:29:25 Jul02 00:00:19
57080 0 4610 crond 114-11:17:17 Jun28 00:00:00
55396 10600 31321 amqzlaa0 -mBT.QM 109-23:29:24 Jul02 00:07:50
53984 10662 544 -bash 00:00 13:03 00:00:00
53984 10662 508 -bash 00:05 13:03 00:00:00
53984 0 3150 /bin/bash /etc/r 114-11:17:43 Jun28 00:00:00
53704 0 23724 sleep 1800 10:44 12:52 00:00:00
53428 10600 31345 amqzlaa0 -mBT.QM 109-23:29:23 Jul02 00:05:50
52380 0 26990 /opt/OV/lbin/per 28-02:48:09 Sep22 00:00:03
49948 0 8159 initlog -q -c /e 114-11:16:57 Jun28 00:00:00
49696 10600 2967 amqzlaa0 -mBT.QM 8-04:58:39 Oct12 00:00:33
45256 10600 31296 /opt/mqm/bin/amq 109-23:29:25 Jul02 00:00:02
44164 10663 20927 sshd: sivarask@n 54:13 12:09 00:00:00
44164 10662 507 sshd: satulus2@p 00:06 13:03 00:00:00
43092 0 505 sshd: satulus2 [ 00:15 13:03 00:00:00
43092 0 20924 sshd: sivarask [ 54:17 12:09 00:00:00
42940 0 5532 /sbin/cpqriisd - 114-11:17:12 Jun28 00:00:00
42484 10600 31341 /opt/mqm/bin/amq 109-23:29:24 Jul02 00:08:20
34996 0 4754 /opt/OV/bin/ovcd 114-11:17:15 Jun28 00:00:01
34876 10600 31309 /opt/mqm/bin/amq 109-23:29:25 Jul02 00:00:00
34120 10600 31297 amqzmuc0 -m BT.Q 109-23:29:25 Jul02 00:02:38
32556 10600 31304 amqzmur0 -m BT.Q 109-23:29:25 Jul02 00:00:08
30808 10600 31308 /opt/mqm/bin/amq 109-23:29:25 Jul02 00:00:03
30164 10600 31323 /opt/mqm/bin/run 109-23:29:24 Jul02 00:00:03
26728 0 6563 /opt/bea/apache/ 30-09:12:28 Sep20 00:00:00
25976 0 4379 /osmf/mgmt/appmg 114-11:17:18 Jun28 06:00:20
24792 0 6051 cmanicd 114-11:17:06 Jun28 00:00:00
24076 10600 31325 /opt/mqm/bin/amq 109-23:29:24 Jul02 00:02:50
23340 38 4002 ntpd -u ntp:ntp 114-11:17:20 Jun28 00:00:00
21916 0 3106 /usr/sbin/sshd 88-15:47:01 Jul23 00:00:00
21048 0 7474 hald 114-11:16:59 Jun28 00:00:00
20772 10600 20511 /opt/mqm/bin/run 05:35:30 07:27 00:00:00
20352 10600 31312 /opt/mqm/bin/amq 109-23:29:25 Jul02 00:00:00
19792 0 4602 /osmf/mgmt/appmg 114-11:17:18 Jun28 00:17:16
18672 81 7465 dbus-daemon-1 -- 114-11:16:59 Jun28 00:00:00
18460 10663 20928 /usr/libexec/ope 54:13 12:09 00:00:00
18240 10600 31371 runmqtrm -m BT.Q 109-23:29:23 Jul02 00:00:20
18104 0 14311 /opt/quest/sbin/ 97-19:31:57 Jul14 00:00:01
17204 0 4762 /opt/OV/bin/ovbb 114-11:17:15 Jun28 00:00:00
17124 10600 31326 /opt/mqm/bin/run 109-23:29:24 Jul02 00:00:00
15380 0 3943 xinetd -stayaliv 114-11:17:27 Jun28 00:00:00
13952 2 12774 /opt/quest/sbin/ 01:44:22 11:19 00:00:00
12872 0 4018 /osmf/mgmt/appmg 114-11:17:19 Jun28 16:16:02
12592 0 5339 cmapeerd 114-11:17:13 Jun28 00:00:00
12544 0 14239 /opt/quest/sbin/ 97-19:32:01 Jul14 00:42:06
12124 0 26984 /opt/perf/bin/pe 28-02:48:09 Sep22 00:00:00
10720 0 26915 /opt/perf/bin/sc 28-02:48:09 Sep22 06:10:54
9836 0 6565 /opt/bea/apache/ 30-09:12:28 Sep20 00:00:27
9836 0 6564 /opt/bea/apache/ 30-09:12:28 Sep20 00:00:02
7484 101 7453 /osmf/mgmt/UCPS/ 114-11:17:00 Jun28 00:00:37
7484 101 7452 /osmf/mgmt/UCPS/ 114-11:17:00 Jun28 00:00:37
7484 101 21158 /osmf/mgmt/UCPS/ 114-07:09:31 Jun28 00:00:37
6324 100 25942 sleep 3600 48:05 12:15 00:00:00
4752 0 1 init [3] 114-11:18:45 Jun28 00:00:03
4508 0 7386 /osmf/mgmt/UCPS/ 114-11:17:00 Jun28 00:00:00
4428 0 4626 /opt/OV/lbin/xpl 114-11:17:17 Jun28 00:00:00
4372 10662 543 ps -e -o vsz,uid 00:00 13:03 00:00:00
4320 0 3723 /osmf/mgmt/xfer/ 114-11:17:35 Jun28 00:00:00
4300 0 8186 /osmf/mgmt/hpOVD 114-11:16:55 Jun28 01:06:08
4248 0 3754 /osmf/mgmt/xfer/ 114-11:17:32 Jun28 00:00:00
4168 0 5990 cmaidad -p 15 -s 114-11:17:06 Jun28 00:03:19
4008 0 6008 cmafcad -p 15 -s 114-11:17:06 Jun28 00:00:00
3760 0 5982 cmaeventd -p 15 114-11:17:06 Jun28 00:00:08
3632 0 3676 syslogd -m 0 114-11:17:38 Jun28 00:00:02
3608 0 2169 udevd 114-11:18:32 Jun28 00:00:00
2996 100 4788 /bin/ksh /osmf/m 114-11:17:15 Jun28 00:00:05
2896 0 8156 /osmf/mgmt/hpOVD 114-11:16:57 Jun28 00:00:05
2724 10602 8162 ksh 114-11:16:57 Jun28 00:00:00
2724 0 8160 /bin/ksh /etc/rc 114-11:16:57 Jun28 00:00:00
2724 0 4741 /bin/ksh /osmf/m 114-11:17:15 Jun28 00:00:01
2552 0 3693 irqbalance 114-11:17:38 Jun28 00:00:00
2544 0 3875 /usr/sbin/acpid 114-11:17:28 Jun28 00:00:00
2540 0 3680 klogd -x 114-11:17:38 Jun28 00:00:00
2404 0 27016 /opt/perf/bin/mi 28-02:48:07 Sep22 00:00:00
2396 0 7837 procmon -d -L -p 114-11:16:58 Jun28 00:14:02
2336 0 5327 cmathreshd -p 5 114-11:17:13 Jun28 00:00:00
2260 0 5334 cmahostd -p 15 - 114-11:17:13 Jun28 00:23:06
2244 0 5398 cmaperfd -p 30 - 114-11:17:13 Jun28 00:47:02
2004 0 5388 cmastdeqd -p 30 114-11:17:13 Jun28 00:00:02
1872 0 4687 /opt/perf/bin/tt 114-11:17:16 Jun28 00:00:00
1696 0 6004 cmaided -p 15 -s 114-11:17:06 Jun28 00:00:00
1640 0 7487 logmon -d -L /os 114-11:16:58 Jun28 00:00:07
1636 0 8110 ../lib/wrapper . 114-11:16:58 Jun28 00:00:00
VSZ UID PID COMMAND ELAPSED STIME TIME
0 0 9 [ksoftirqd/3] 114-11:18:45 Jun28 00:00:00
0 0 99 [kblockd/0] 114-11:18:44 Jun28 00:00:00
0 0 8 [migration/3] 114-11:18:45 Jun28 00:00:39
0 0 7 [ksoftirqd/2] 114-11:18:45 Jun28 00:00:00
0 0 6 [migration/2] 114-11:18:45 Jun28 00:00:33
0 0 688 [kjournald] 114-11:18:37 Jun28 00:00:54
0 0 5 [ksoftirqd/1] 114-11:18:45 Jun28 00:00:00
0 0 500 [gpp] <defunct> 00:28 13:03 00:00:00
0 0 4 [migration/1] 114-11:18:45 Jun28 00:00:31
0 0 3 [ksoftirqd/0] 114-11:18:45 Jun28 00:00:00
0 0 3045 [kjournald] 114-11:17:44 Jun28 00:00:00
0 0 3044 [kjournald] 114-11:17:44 Jun28 00:00:36
0 0 3043 [kjournald] 114-11:17:44 Jun28 00:00:00
0 0 3042 [kjournald] 114-11:17:44 Jun28 00:02:28
0 0 3041 [kjournald] 114-11:17:44 Jun28 00:01:33
0 0 3040 [kjournald] 114-11:17:44 Jun28 00:01:02
0 0 3039 [kjournald] 114-11:17:44 Jun28 00:00:26
0 0 3038 [kjournald] 114-11:17:44 Jun28 00:00:50
0 0 3037 [kjournald] 114-11:17:44 Jun28 00:01:48
0 0 3036 [kjournald] 114-11:17:44 Jun28 00:02:39
0 0 3035 [kjournald] 114-11:17:44 Jun28 00:00:26
0 0 3034 [kjournald] 114-11:17:44 Jun28 00:01:24
0 0 3033 [kjournald] 114-11:17:44 Jun28 00:00:00
0 0 3032 [kjournald] 114-11:17:44 Jun28 00:00:00
0 0 2 [migration/0] 114-11:18:45 Jun28 00:00:15
0 0 296 [kseriod] 114-11:18:44 Jun28 00:00:00
0 0 2953 [kmpathd/7] 114-11:18:29 Jun28 00:00:00
0 0 2952 [kmpathd/6] 114-11:18:29 Jun28 00:00:00
0 0 2951 [kmpathd/5] 114-11:18:29 Jun28 00:00:00
0 0 2950 [kmpathd/4] 114-11:18:29 Jun28 00:00:00
0 0 2949 [kmpathd/3] 114-11:18:29 Jun28 00:00:00
0 0 2948 [kmpathd/2] 114-11:18:29 Jun28 00:00:00
0 0 2947 [kmpathd/1] 114-11:18:29 Jun28 00:00:00
0 0 2946 [kmpathd/0] 114-11:18:29 Jun28 00:00:00
0 0 2853 [kauditd] 114-11:18:30 Jun28 00:00:00
0 0 27 [kacpid] 114-11:18:44 Jun28 00:00:00
0 0 26 [khelper] 114-11:18:44 Jun28 00:00:00
0 0 25 [events/7] 114-11:18:44 Jun28 00:00:00
0 0 24 [events/6] 114-11:18:44 Jun28 00:00:00
0 0 23 [events/5] 114-11:18:44 Jun28 00:00:00
0 0 22 [events/4] 114-11:18:44 Jun28 00:00:00
0 0 21 [events/3] 114-11:18:44 Jun28 00:00:00
0 0 20 [events/2] 114-11:18:44 Jun28 00:00:00
0 0 19 [events/1] 114-11:18:44 Jun28 00:00:00
0 0 18 [events/0] 114-11:18:44 Jun28 00:00:00
0 0 17 [ksoftirqd/7] 114-11:18:45 Jun28 00:00:00
0 0 16 [migration/7] 114-11:18:45 Jun28 00:00:20
0 0 15 [ksoftirqd/6] 114-11:18:45 Jun28 00:00:00
0 0 152 [aio/7] 114-11:18:44 Jun28 00:00:00
0 0 151 [aio/6] 114-11:18:44 Jun28 00:00:00
0 0 150 [aio/5] 114-11:18:44 Jun28 00:00:00
0 0 14 [migration/6] 114-11:18:45 Jun28 00:00:15
0 0 149 [aio/4] 114-11:18:44 Jun28 00:00:00
0 0 148 [aio/3] 114-11:18:44 Jun28 00:00:00
0 0 147 [aio/2] 114-11:18:44 Jun28 00:00:00
0 0 146 [aio/1] 114-11:18:44 Jun28 00:00:00
0 0 145 [aio/0] 114-11:18:44 Jun28 00:00:00
0 0 144 [kswapd0] 114-11:18:44 Jun28 00:00:15
0 0 143 [pdflush] 114-11:18:44 Jun28 00:00:58
0 0 142 [pdflush] 114-11:18:44 Jun28 00:00:00
0 0 13 [ksoftirqd/5] 114-11:18:45 Jun28 00:00:00
0 0 12 [migration/5] 114-11:18:45 Jun28 00:00:17
0 0 11 [ksoftirqd/4] 114-11:18:45 Jun28 00:00:00
0 0 10 [migration/4] 114-11:18:45 Jun28 00:00:19
0 0 107 [khubd] 114-11:18:44 Jun28 00:00:00
0 0 106 [kblockd/7] 114-11:18:44 Jun28 00:00:00
0 0 105 [kblockd/6] 114-11:18:44 Jun28 00:00:00
0 0 104 [kblockd/5] 114-11:18:44 Jun28 00:00:00
0 0 103 [kblockd/4] 114-11:18:44 Jun28 00:00:00
0 0 102 [kblockd/3] 114-11:18:44 Jun28 00:00:00
0 0 101 [kblockd/2] 114-11:18:44 Jun28 00:00:00
0 0 100 [kblockd/1] 114-11:18:44 Jun28 00:00:00

 

[Annihilannic]

By the way, are you aware that there is a Linux (server) forum on this site too? It sounds like your questions are more server related.

As per that documentation I quoted it seems like the memory usage you are seeing is not going to be listed in ps; I think your only option to identify the culprit will be to stop each application while monitoring the memory usage and see which one releases that committed memory.

Annihilannic.