vmstat - question

[zaxxon]

Hi,

I have a LPAR on a 650 with AIX 5.2 ML 5, 3x 1,45 GHz and about 12 GB RAM. There are parts of a IBM Content Manager v 8.2. running on it with several small applications and we are running a performance test, how fast this CM archive is able to retrieve documents, stored in it.

While monitoring the OS, I have following situations:

 3  0 1555073  2539   0   0   0 657 1950   0 2773 104378 11826 42 16 37  5
 3  0 1555073  2521   0   0   0 677 1791   0 2673 93265 11305 37 16 42  5
 0  1 1553171  3828   0   0   0 204  677   0 2637 93666 11983 38 13 42  7
 2  0 1553171  3708   0   0   0 568 1822   0 2786 97084 11831 43 13 38  5
 2  0 1553171  3579   0   0   0 680 1988   0 2332 89603 10743 38 14 45  2
 1  0 1553197  3390   0   3   0 649 1958   0 2427 161383 9793 63 12 20  4
184  0 1553929  2403   0  13   0 905 2652   0 1578 164907 9004 71 24  4  1
177  2 1553987  2024   0   4   0 1558 4671   0 2262 76008 13548 80 20  0  0
162  0 1554011  1938   0   2   0 540 1739   0 2777 117605 11586 82 18  0  0
86  0 1553995  1791   0   0   0 348 1158   0 4424 151702 12227 75 25  0  0
kthr     memory             page              faults        cpu
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
36  0 1553302  4116   0   1   0 1744 7565   0 3224 109229 9503 50 22 27  1
 0  0 1553260  4157   0   0   0   0    0   0 1189 6909 2843  0  2 98  0
 0  0 1553260  4157   0   0   0   0    0   0 1122 10629 4040  3  2 94  0
 0  0 1553260  4736   0   0   0   0    0   0 2045 78999 10055 35  9 50  6
 2  0 1553262  4301   0   0   0 337 1072   0 2469 95395 11143 39 13 39  9
 0  1 1553298  4128   0   2   0 629 1836   0 2806 105848 12053 49 13 36  2
 0  0 1553298  3932   0   0   0 672 2369   0 2512 80207 10686 36 11 47  6

The 1st column, the kernel threads run queue has sometimes a very high number of how kthreads waiting for a CPU. When this occurs, there is usually no idle or wait visible. I also read in an IBM Redbook about the performance toolbox, that is is not necessarily a performance problem, when this occurs.
Since we are having from time to time those high numbers in the "r" column of vmstat, and we have sometimes slow responses from the CM archive (DB2s Databases and CM software being tuned by a IBM specialist already), we want to be sure, that the OS is not having problems, that might have an impact on the CM archive.

So I would like to know if anyone of you is having such high values too ("r"-column of vmstat) and the applications running on it are busy but not having bad performance, maybe not even in non constant intervals, at all?

I searched lots of documentation, but I did not find any tool or hint, if I could check what is causing those occasionally (between every 10-30 seconds) high kthreads in the run queue...

Thanks for any hints in forward!

laters
zaxxon

 

[hfaix]

honestly, I don't watch run queues very often because I rarely have performance issues, however I've never seen one anywhere near that high. Maybe I'll take a closer look tomorrow. I think I'd start digging into the problem if it got over 10?

Interestlingly, you're not paging much but you could use more RAM (but who couldn't).

sorry.... I'm not much help.

 

[zaxxon]

Np, I am happy for any response so far
We had some paging 1-2 days ago, but I changed

maxperm -- 80 --> 50
maclient -- 80 --> 50
minfree -- 120 --> 1200
maxfree -- 128 --> 1280

Since then, I had only very minimal paging left and this only very occasionally maybe up to 10-12 pages at PGIN. Formerly I had up to 400-500 pages at PGIN which was traceable in the CM logs.
I didn't find any hint how much maxperm you should have for a heavy traffic DB2 system, but most guides suggested to start at 50% - seems to have helped. I also found no guideline for adjusting minfree/maxfree so I just tried a factor of 10

For the kthread r-queue, you can have up 128 threads waiting there per CPU, btw.

laters
zaxxon

 

[hirschaj]

The run queue can vary greatly based upon what the application is doing. I have seen some applications that dump a ton of threads all at the same time. These threads all get processed very quickly but if you are running a vmstat with a short interval of 1 or 2 seconds you will see a very large number of threads on the run queue. This can be just fine or it can be a problem. It really depends on how the application is supposed to work. With DB2 you really should not see anything like that, but I am not sure about your other application. I have seen DB2 servers that have relatively high run queue numbers because they were missing an index for a heavy hitting query. On a memeory tuning note I bet you could go lower than 50% for maxperm and maxclient but there are a lot of factors to look at with that sort of change.


Jim Hirschauer

http://www.aixexpert.com

 

[zaxxon]

Thanks for the info so far - when I have any news, I will write them in this post.

laters
zaxxon