Do I require more memory

[cichkid]

Here are my question(s)

Do I require more memory?
Do I require a larger paging space?
Is the system working OK?
Do I need to do some memory tuning?

If someone requires the output from a vmstat I can send a full days worth with the interval being 1 min.

The following are details and output from the server in question.

Machine = IBM,7026-B80
OS = AIX 4.3.3
Processors = 2
RAM = 2GB

lsps -a
PS Phys Vol Vol Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 2048MB 79 yes yes lv

VMSTAT shows that the free space drops to 0 but not often and not for long.

CPU Utilization spikes from 0% to 100% at various times of the day.

There is very little paging on this system.

Output from vmtune command:
===========================
vmtune: current values:
-p -P -r -R -f -F -N -W
mnprm mxprm mnpgahd mxpgahd minfr maxfr pd_npages mxrndwrt
104645 418580 2 8 120 128 524288 0

-M -w -k -c -b -B -u -l -d
mxpin npswrn npskill nmclust nmfsbfs hd_pbf_cnt lvm_bfcnt lrubucket defps
419400 16384 4096 1 93 128 9 131072 1

maxclient
418580

number of valid memory pages = 524249 maxperm=79.8% of real memory
maximum pinable=80.0% of real memory minperm=20.0% of real memory
number of file memory pages = 99121 numperm=18.9% of real memory

number of compressed memory pages = 0 compressed=0.0% of real memory
number of client memory pages = 0 numclient=0.0% of real memory
# of remote pgs sched-pageout = 0 maxclient=79.8% of real memory

We plan on putting some additional load on this application server in the near future.

Thanks for any help

Jim Taylor

 

[ProbablyDown]

When your server CPU is at 100% there is a good possibility of a CPU constraint. I would suggest checking in this order:

1) CPU - use the sar command

If the value is high, probably CPU constraint

2) Memory - use vmstat. If high paging then you have a memory constraint. If not:

3) Disk - use iostat. If overactive (%iowait > 25% or %tm_act > 70%)may be disk/scsi constraint.

Can be any number of issues aside from these though, but it's a start...

 

[mrn]

Get hold of a copy of nmon, it produces graphs which will help you to understand whats going on.
See

http://www-1.ibm.com/servers/esdd/articles/analyze_aix/

for more info.

As a rule for swap space (512 + mem - 256) * 1.25 so in your case (512 + 2048 - 256) * 1.25 = 2880 swap space.

What are you running on this box, I've seen strange Oracle processes eat memory on AIX, and also each time a connection is made memory is allocated, but not released on disconnection. Take a look at top 10 procs with either monitor or topas. To ease problems I also tend to reboot every couple of weeks, this seems to help.

Does CPU spike at around the same time everyday, again use nmon to take a look.



--
| Mike Nixon
| Unix Admin
|
----------------------------

 

[cichkid]

Thanks for the replies to date. This is a 7x24 server that can't be rebooted very often. If it were we would miss our SLA's with our clients.

This box is an application server which provides WebSphere/JAVA applications. The database's are Oracle but are on another server. The database server also has some difficulties (minor). The webserver Apache/HTTP server is just fine.

There does not seem to be any memory leaks with the applications. Memory is released when apps complete.

Thanks again

Jim

 

[ProbablyDown]

I'm running some Oracle jre apps on a Linux server, different I know... The funny thing I noticed though is that slowly the Oracle Java procs get orphaned slowly tying up resource unnecessarily. I eventually have to killall jre processes to free everything up. Left untouched the resources drain the server slowly dies and needs to be rebooted.

BUT WE'RE TALKING ABOUT AIX! Where uptimes of 472 days are not uncommon!

I wouldn't want to have to reboot all the time... That's why we forked out the cash for these servers, otherwise we would have went Intel ;-)

 

[aixnag]

Post top 10 processes, vmstat and i/o stat o/p ( take snap shot in peak time)



aixnag
IBM Certified Specialist - P-series AIX 5L Administration
IBM Certified Specialist - AIX V4 HACMP

 

[bjverzal]

I'd like to provide a more simple answer. If paging space is 79% used, I would say you are memory constrained. Check "pi/po" columns on vmstat.

BV

 

[theycallmetim]

bjverzal: answer need to take into effect numperm and vmtune settings.

cichkid: If the output you've shown at the beginning of this post was taken at the same time, yes. You could benefit from more memory. I'd say at LEAST another 2 GB depending on if this was at the high-point of your response issues.

You could gain a bit of breathing room by setting your minperm to 0 and your maxperm to 5. But that still won't free up all your paging. Your numperm is only 18 but your lsps shows 79. Meaning even if you did that, paging would only drop to about ~60.

CPU usage is normall to be that high when paging is occurring. It does not mean you are CPU constrained. Fix the memory problem and see what happens to your CPU.

 

[cichkid]

Perhaps I was not clear. I am NOT experiencing much paging on this server. In fact of all 50 of our servers paging is probably the lowest on this box. Usually a vmstat with the interval set at 1 min. shows that pi and po are 0. Occasionally they will jump slightly more po than pi.

This is what I typically see when the websphere apps are most active:

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 1 613027 23886 0 0 0 2 13 0 400 2341 780 26 2 72 0
2 1 613027 23885 0 0 0 0 0 0 301 5754 268 98 2 0 0
2 1 613027 23885 0 0 0 0 0 0 518 1642 827 99 0 0 0
6 1 613027 23885 0 0 0 0 0 0 1953 6909 3903 94 6 0 0
3 1 613027 23884 0 0 0 0 0 0 378 4589 907 96 4 0 0
5 1 613027 23884 0 0 0 0 0 0 656 1813 1019 96 4 0 0
4 1 613027 23882 0 0 0 0 0 0 1352 5168 2619 94 6 0 0
5 1 613027 23880 0 0 0 0 0 0 1698 7735 3658 92 8 0 0
3 1 613027 23879 0 0 0 0 0 0 821 4269 1238 96 4 0 0
3 1 613027 23877 0 0 0 0 0 0 880 2857 1483 99 1 0 0
2 1 613027 23875 0 0 0 0 0 0 1085 5811 2510 95 5 0 0
2 1 613027 23873 0 0 0 0 0 0 891 4470 2047 96 4 0 0
2 1 613027 23868 0 0 0 0 0 0 986 3466 1453 96 4 0 0
3 1 613027 23867 0 0 0 0 0 0 999 3794 1605 95 5 0 0
3 1 613027 23865 0 0 0 0 0 0 1028 3339 1715 96 4 0 0


Notice that the CPU utilization is pegged at 100%. It stays this way for about 1/2 to 1 hour then drops to about 50% for the same duration. It cycles this way during business hours. The application programmers are looking to see if they can figure out why. Most of the paging happens during backups.

Jim

 

[victorv]

hi,
seeing output of vmstat you have no paging,
memory free is low, but you have cpu almost 100%

use topas and see wich program/user fills your cpu

bye

 

[SPguru]

This question cannot be answered given the limited amount of data given. Any attempt to answer this question based on given data is the same as giving an answer based on the throw of a dart to a piece of paper on the dartborad.

Data needs to be collected for an extended period of time from many different sources. It seems that everyone wants to give their "expert" opinion, and thus, in doing so are proving their supreme ignorance in not being able to give the correct answer - not enough information!

This question seems to be showing a lot of arm-chair quarterbacking.

 

[pepperh]

I agree with Victorv, no paging problem indicated even since system start as indicated on the first line. CPU utilization is high, i/o wait does not seem tyo be a problem either. Looking for processes that are consuming excessive CPU would be the next step in determining the problem.

 

[victorv]

hi,

No I/O, No paging, No Idle,
2-4% system activity, 96-98 user Activity.

Or you have loaded a long Numeric Analisys Program or it
seems to be a typical "program in loop"; FORTRAN speaking:

10 GOTO 10

STOP

 

[aixnag]

Post top 15 processes

As others said, vmstat leaves an indication of cpu-bound, Its very difficult to conclude at this time. Why don't use 'nmon' tool to investigate further. Its very useful.

you may use this command to collect top 15 processes that have used most cpu usage
#ps gu|head -n 1;ps gu|egrep -iv "CPU|kproc"|sort +2b -3 -n -r|head -n 15



aixnag
IBM Certified Specialist - P-series AIX 5L Administration
IBM Certified Specialist - AIX V4 HACMP