%sy > 40 % (tuning question)

[bjverzal]

vmstat 3
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 1 41996 172 0 0 0 124 297 0 140 336 170 1 3 95 1
2 2 41996 125 0 0 0 3643 7593 0 942 10029 4218 12 43 39 6
2 2 41996 136 0 0 0 3673 8251 0 938 8623 4593 10 44 41 5
1 2 41996 120 0 0 0 3712 13600 0 950 8784 4439 13 42 38 7
2 1 41996 130 0 0 0 3609 5300 0 921 8506 4421 12 41 42 5
1 2 41996 131 0 0 0 3764 5313 0 961 8827 4680 16 46 33 6
1 2 41996 113 0 0 0 3596 4474 0 937 8584 4321 13 46 35 6
1 2 41996 171 0 0 0 3669 4355 0 1000 8577 4562 12 45 36 7

Got a question for the performance tuners out there...

I have a 2-way F50 with 1GB of memory. There is a small Informix database on it, but it serves as a backup server for us. We have 23 SSA disks that contains 110 filesystems, each of which mounts a striped (32K, 3 PV's) logical volume.

Right now, all the server is doing is reading from the SSA drives and dumping data off to an LTO tape drive via the IBM dual-channel Ultra2-SCSI adapter.

I am really perplexed as to why my "sy" column is over 40%. 2-weeks ago, I reconfigured Informix to free up .5GB of memory to see if the numbers would change. The numbers are the same.

Here is my vmtune output as well:

/usr/samples/kernel/vmtune
vmtune: current values:
-p -P -r -R -f -F -N -W
minperm maxperm minpgahead maxpgahead minfree maxfree pd_npages maxrandwrt
13106 39319 2 16 120 152 524288 0

-M -w -k -c -b -B -u -l -d
maxpin npswarn npskill numclust numfsbufs hd_pbuf_cnt lvm_bufcnt lrubucket defps
209707 16384 4096 1 93 528 9 131072 1

-s -n -S -h
sync_release_ilock nokillroot v_pinshm strict_maxperm
0 0 0 0

number of valid memory pages = 262133 maxperm=15.0% of real memory
maximum pinable=80.0% of real memory minperm=5.0% of real memory
number of file memory pages = 210712 numperm=80.4% of real memory

Anyone have any ideas ?

Thanks, Bill.

 

[MendotaGuy]

Here's my $.02 worth.

If all your system is doing is reading disks and writing tapes, I would expect to see more IO waits. Of course, since you have striped SSA and LTO tape you are pulling data in and pushing it out pretty fast and maybe spending more time managing memory trying to keep up.

Since this is a backup server you could safely try some things and see what happens. First, I'd change the maxpgahead to something larger, at least 64, but maybe 128 or 256. That will increase the read ahead on the disks and it may or may not help depending on how sequential your reads are.

Second, I'd change the maxfree to something higher, like 512, 1024 or 2048. AIX will use memory until there is less than minfree pages left. It will then free up pages until maxfree is available. If you are doing lots of reads, you may be spending time freeing up a small number of pages (32 in your case). By raising maxfree you will be freeing up more pages at once, but less often. This could be bad in a normal system if you start throwing out pages that you need, but if your system is just reading/writing then you're probably not going to use those file pages more than once anyway.

If you try these, be sure to keep records and statistics about what you changed and what effect they had. They may increase your cpu utilization or io wait, but if the job gets done faster then it's working.

Good luck, and let us how it worked.

 

[bjverzal]

Did some playing with your suggestions and I did not see much change either way. Tried several permutations - nothing changed for the good or bad.

If you have anything else, please advise.

TIA!

Bill.

 

[prv3116]

Are you getting any CPU errors? It might not be memory at all, but maybe your processor.
%usr + %sys > 80% = CPU bound

You don't look to be hitting 80% but you are in the 50's. When this occurs, use top or monitor to determine what process is hitting the CPU. I like monitor -alternative -s 2 .

Does it improve with a reboot? If so, your application may be having a problem with releasing or clearing registers.

Look also at your output from vmstat 1 10. Are you doing a lot of paging (pi) and (po) .
Do vmstat -s
paging space page-ins + paging space page-outs > 40% of page ins + page outs = Memory Bound
If so, trying increasing you swap space.
Remember, amateurs built the Ark, professionals built the Titanic.