memory issues

[harpal]

hi,

we are running a f80 aix system with 5gig of RAM and 4, this system is a running a
merchandising application we have oracle 8i also running on the system,

below is a paste from a utility called nmon6a

Memory Use Physical Virtual Paging pages/sec In Out VM parameters
% Used 100.0% 26.6% to Paging Space 0.0 1.0 numperm 74.1%
% Free 0.0% 73.4% to File System 159.9 3.0 minperm 20.0%
MB Used 5119.2MB 544.4MB Page Scans 187.3 maxperm 79.9%
MB Free 0.7MB 1503.6MB Page Cycles 0.0 minfree 120
Total(MB) 5120.0MB 2048.0MB Page Reclaim 0.0 maxfree 128

the problem we are gettin is the system at certain times of the day will not even fire up a telnet
session throwing up a "virtual memory has run out" looking at the setup there is know
way we should be runnin out of memory the SGA size of the DB is 642meg and a front
end session should take in the region of 2k to 20meg depending on user activity this noway
what therefore exceed 5gig of ram effectively half that should be more then enough. bit curious
as to why nmon report the on 0.7MB is free is there any memory management commands or smitty screens
or a snapshot that I can run to find out what process are caining all the memory.

many thanks

Harps

 

[sbix]

ps aux|awk '( $6 ~ /[0-9]/ ) { print $2" "$6 }'|sort -nr +1|head

This command will give you the 10 top memory eater process, the 1st column is the PID, the second the RSS field of ps

 

[letis]

hello harpal,

according to the nmon output, most of your physical memory is used as a cache for files (numperm=74,1%), which is not amazing as this is the default behaviour of AIX (ie optimized for a fileserver).

by default, the system will consider it can use up to 80% of the memory for caching files (this is the meaning of maxperm), this is a setting you can tune with vmtune.

if a process can not have his code and data in physical memory, the system will try to free some physical memory, or will not be able to execute the process.

before taking actions, could you send a vmstat 1 output and a svmon -G output ?

regards

 

[harpal]

As requested

root@hqax00 / >vmstat 1
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 348544 127 0 1 1 126 289 0 173 596 682 14 4 69 12
0 3 348504 183 0 0 0 43 70 0 632 2820 551 1 1 95 4
0 2 348504 169 0 0 0 0 0 0 468 950 142 0 0 99 1
0 2 348504 169 0 0 0 0 0 0 455 657 97 0 0 99 0
0 2 348544 126 0 0 0 0 0 0 423 797 71 0 0 98 1
0 2 348544 126 0 0 0 0 0 0 420 572 55 0 0 99 0
0 2 348544 126 0 0 0 0 0 0 426 564 53 0 0 99 0
0 2 348544 133 0 0 0 26 43 0 676 1779 560 0 1 96 3
0 3 348544 127 0 0 0 26 44 0 654 1401 394 1 19 69 11
0 2 348544 124 0 1 0 0 0 0 607 1501 324 1 2 96 1
0 2 348671 128 0 0 0 152 307 0 648 4747 548 17 1 80 2
1 2 348798 128 0 0 0 137 259 0 850 4519 937 14 1 85 0
0 2 348798 129 0 0 0 0 0 0 461 829 129 0 0 99 0
0 2 348805 125 0 4 0 18 29 0 452 2348 123 1 0 96 2
0 2 348823 125 0 0 0 27 67 0 486 903 179 1 0 98 1
1 2 348824 131 0 0 0 94 163 0 531 1467 285 1 0 92 7
0 3 348824 129 0 1 0 201 419 0 647 1972 529 3 1 81 15
1 2 348824 124 0 1 0 8 20 0 518 3706 5679 19 9 72 1
1 2 348784 171 0 0 0 9 17 0 498 5708 3673 19 8 72 1
0 2 348784 169 0 0 0 0 0 0 568 1300 589 2 0 96 1
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 2 348786 131 0 0 0 0 0 0 626 2572 5746 13 13 71 3
0 2 348789 127 0 0 0 0 0 0 536 1630 311 3 0 96 1
0 2 348826 124 0 0 0 34 69 0 593 1604 392 1 0 98 1
0 2 348826 124 0 0 0 0 0 0 464 752 142 0 0 99 0
0 2 348826 123 0 1 0 79 132 0 551 2482 332 4 1 90 5
0 2 348826 123 0 0 0 0 0 0 458 853 128 0 0 99 0
0 2 348826 122 0 0 0 17 17 0 530 1670 272 2 0 95 2
1 2 348826 122 0 0 0 43 64 0 601 1996 387 14 0 82 3
1 3 348829 123 0 11 0 97 200 0 768 3007 769 8 1 80 12
0 2 348836 128 0 3 0 47 87 0 613 1451 470 4 0 91 5
0 3 348836 122 0 0 0 52 117 0 654 1925 492 2 1 93 4
0 2 348836 121 0 0 0 43 59 0 578 1674 413 4 0 92 3
0 2 348836 128 0 0 0 17 25 0 550 1430 302 0 0 97 2
0 2 348836 127 0 0 0 0 0 0 566 1223 336 0 0 99 0
0 2 348836 123 0 0 0 0 0 0 512 2240 5350 9 10 81 0
0 2 348836 123 0 54 0 54 125 0 605 10322 3997 19 8 66 7
1 2 348270 675 0 92 0 140 269 0 755 23219 3256 34 6 51 9
2 2 348270 674 0 0 0 0 0 0 536 21868 5693 33 14 53 0
2 2 348270 675 0 1 0 0 0 0 549 21375 297 21 5 73 1
2 2 348310 626 0 0 0 0 0 0 472 21825 5999 34 13 52 1
kthr memory page faults cpu



root@hqax00 / >svmon -G

size inuse free pin virtual
memory 1310709 862811 377 81487 349750
pg space 524288 163482

work pers clnt
pin 81519 0 0
in use 346557 516254 0

root@hqax00 /

I have also worked out even 150 session taking 20meg each should only consume 1.5 gig and oracle itself 642meg of ram this is just over 2 gig so surely 5120meg of ram is more then suffice.

harps

 

[harpal]

root@hqax00 / >ps aux|awk '( $6 ~ /[0-9]/ ) { print $2" "$6 }'|sort -nr +1|head
51260 23348
55692 22064
80612 21640
71242 19260
53838 19256
59024 18848
66908 18252
48052 17904
83344 17860
73672 17796
root@hqax00 / >

 

[sbix]

If you wish something more "friendly" do:
ps aux|awk '( $6 ~ /[0-9]/ ) { print $0 }'|sort -nr +5|head

So you will have the description of the 10 top memory user processes

You have to run this while having problem

 

[letis]

well the vmstat output show that you have in permanence some threads blocked in memory (b column), presumably because there's no physical memory to let them running.

let me suggest make the following change :

a) if you use AIX 5.1, go in /usr/samples/kernel and from the command line type :
vmtune -p 10
(to say AIX to use at least 10% of physical memory for cache I/O)

vmtune -P 30
(to say AIX to use at MAXIMUM 30% of memory as cache I/O)

vmtune -h 1
(to say AIX to strictly obey these rules)

these thresolds are not absolute, this is to make a try, recommanded values for applications server range from 5 to 15% for minperm, from 20 to 40% for maxperm.

OR

b) if you use AIX 5.2, go in :
smitty => performance => kernel & network parameters => virtual memory manager => change/show current parameters, and change maxperm to 30, minperm to 10, and strictmaxperm to 1, ENTER.


if the system behaves OK, then add an entry from /etc/inittab to set them permenantly, or use smitty (AIX 5.2) to make these changes permanent (
smitty => performance => kernel & network parameters => virtual memory manager => change / show for next boot).

Note that the default values you use are minperm=20%, maxperm=80%, strictmaxperm=0.

tell us if it helps,

regards

 

[harpal]

letis,


hi we are currently running OSlevel 4.3.3.0 would the procedure be the same?

Harps

 

[letis]

yes the procedure is identical to AIX 5.1, but as far as i remember vmtune is not installed by default, therefore if you don't have vmtune in /usr/samples/kernel, install it from fileset bos.adt.samples


if you had to use vmtune permanently, you will have to put it to the same maintenance level thant the rest of the system, but in this situation (ie a try to see if it solves your problems), it's still not very important

regards

 

[harpal]

letis,

I have had a look at our box and vmtune is indeed located in /usr/samples/kernel

but when I try to carry out command i get following syntax

root@hqax00 /usr/samples/kernel >ls
emstat schedtune schedtune.c vmtune
root@hqax00 /usr/samples/kernel >vmtune -p 10
ksh: vmtune: not found.
root@hqax00 /usr/samples/kernel >

regards

Harps

 

[letis]

oups,

the current directory is not in the PATH for root, try
./vmtune


regards