How can I verify the I don't have enough Physical memory?

[ravantti]

Hi,
I'm suspecting that my HP-UX 11.0, 64-bit database server is lacking memory. We run SAP/Informix installation. The server is N4000 series having 4x550 processors and 4GB physical memory. I'v notices that the leds that indicate hard disk use are burning pretty much all the time (after SAP upgrade). I think it means lots of swapping or paging! Time to time the performance is a problem. How can I verify that we do not have enough physical memory? Can you tell me some rule of how much swapping or paging is healthy? I attach some swap info below:
root# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 909896 138680 87% 0 - 1 /dev/vg00/lvol2
dev 10240000 941788 9298212 9% 0 - 1 /dev/vg00/lvol9
dev 10240000 956632 9283368 9% 0 - 1 /dev/vg00/lvol10
reserve - 4060340 -4060340
memory 3213132 2204336 1008796 69%
total 24741708 9072992 15668716 37% - 0 -

 

[bi]

I'm wondering if the problem isn't so much too much swapping but misplacement on the disks of the database files.

If database files that are heavily written to (logs and rollback segments, for example, or a file with a "hot" table)are bunched up on a few disks, that would cause those disks to always be running. Have you tried running iostat during times when performance slows to see which disks are super busy? Then you could check your filesystem layout to see what is actually on those disks.

 

[jollyroger]

Do you have only one hard disk in the machine?

If so then I would seriously investigate adding more disks, looking at you swapinfo figures I can see nothing particularly wrong. However how many disks do you have in you /dev/vg00 volume group?

Please send some more details as to how the machine is set up, I agree with bi, that using another tool such as iostat or sar will give you more information to try and fix the problem.

A great link is:

http://www.docs.hp.com/hpux/onlinedocs/os/11.00/tuningwp.html

and the ITRC is also a good place to start.

However for now please let us know what your hardware set up is.

Best regards

 

[ravantti]

OK! We have a EMC-symmetrix storage with 96 physical disk and 5GB of cache memory in it. The storage shows out 12 meta volumes to hp-ux and we use 4xFW SCSI with Power Path scsi channel load balancing to access the disk. We have two internal disks that are mirrored and they build up vg00. The swaps reside on these/this disk in three parts. The informix database rlvols reiside on symmetrix disks. If I check wit sar or iostat the load it looks like below:

09:38:45 device %busy avque r+w/s blks/s avwait avserv
09:38:46 c1t6d0 43.43 1.29 83 1911 5.36 18.09
c2t6d0 30.30 1.10 66 1931 5.14 16.68
c10t0d4 1.01 0.50 2 8 5.65 2.67
c10t0d7 1.01 0.50 4 28 4.45 1.80
c10t1d1 1.01 0.50 1 4 4.38 14.21
c10t1d3 2.02 0.50 2 8 3.51 10.82
c3t0d0 1.01 0.50 4 44 5.60 2.05
c3t0d1 1.01 0.50 1 4 4.96 13.26

If I check with EMC tools the utilization of disks (where the database resides) the usage is only 2-7% or so and for scsi-channels the usage is all times around 4-12%. Should I turn off the pseudo swap with kernel parameter swapmem_on=0 ? I have run one script (for 24 hrs) that checks the free memory with vmstat command and it shows 15-35MB out of 4GB all the time! Looks bad to me.

 

[jollyroger]

Hey there

The disks from your sar report look a little over utilised, but not to bad, for example the % busy is not to high but more importantly avwait is not greater than avserv.

I would not switch off the pseudo swap for now because HP have been moving down the virtual memory road for a while.

Is the vhand daemon CPU usage very high, if you look at top, does it seem to be running a fair amount, also does the

As you say the amount of free memory with vmstat is also consistantly low so I would guess, like you, that you simply do not have enough memory on the machine.

Here is another article which will give you more information:

http://forums.itrc.hp.com/cm/components/FileAttachment/0,,0x03857d4cf554d611abdb0090277a778c,00.txt

If I were to add more memory, looking at the swapinfo I would add a minimum of 1GB, unless the management have other ideas to increase the load on the machine.

Have you adjusted any of the kernel parameters related to memory (other than the pseudo swap)?

 

[ravantti]

OK thaks for the answers. I know now that we do not have enough physical memory. I can see that the worst performing disk in our serve is internal disk that our vg00 uses where the swap also resides. I found one way to check out that the disk gives an average of 39ms response time while in just the same kind of server used as SAP application server gives constantly (from disk used for swapping in vg00) 17ms response time. I also found from statistics that the server has 15 000 -70 000 pages out per hour and the peak was 136 000 per hour. The worrying thing here is that looks like (if I'm counting it ok from swapinfo -mt) that we are using 4,0-6,0GB swap space pretty much all the time. See below the swapinfo -mt I just took.

TYPE AVAIL USED FREE USED
total 24162 7216 16946 30%

Thank You from answers!