CPU Tuning

[khalidaaa]

Hi All,

I'm having an annoying tuning problem!

We are in the stage of shifting an application that runs on an SP2 machine with AIX 4.3 to a P5 570 machine with AIX 5.2

Funny enough, we know that this application is not certified to run on 5L but our management is pushing for it. Any way, I know for sure that this application is a CPU intensive and we are running a test instance of the application for now on the 5.2 partition of the p5 570 machine. I used to have one process that eats the whole CPU dedicated to this partition and now they are two as follows:

# ps aux | more
USER       PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
oracle   48816 65.9  1.0 28072 12452      - A    22:28:50 840:41 ora_snp1_EMP2 
oracle   47960 46.1  3.0 53096 37476      - A    07:01:11 351:48 oracleEMP2 (DESCR

on topas

Name            PID  CPU%  PgSp Owner           oracle        48816  47.9   2.1 oracle
oracle        47960  47.6  26.6 oracle

# vmstat 1
System Configuration: lcpu=1 mem=5120MB
kthr     memory             page              faults        cpu     
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa 
 1  1 203615 1005614   0   0   0  93  446   0 306  307 287 94  1  4  1
 2  0 203619 1005610   0   0   0   0    0   0 266 2468 338 99  1  0  0
 2  0 203619 1005610   0   0   0   0    0   0 258 2135 294 99  0  0  0
 2  0 203619 1005610   0   0   0   0    0   0 248 2106 284 99  1  0  0
 2  0 203619 1005610   0   0   0   0    0   0 257 2334 313 99  0  0  0
 2  0 203619 1005610   0   0   0   0    0   0 250 2118 289 99  0  0  0
 2  0 203619 1005618   0   0   0   9  148   0 302 2413 421 96  4  0  0
 2  0 203619 1005618   0   0   0   0    0   0 254 2114 314 99  0  0  0
 2  0 203619 1005618   0   0   0   0    0   0 251 2427 290 99  1  0  0
 2  0 203619 1005618   0   0   0   0    0   0 248 2108 288 99  1  0  0

Is there any way on earth that i can tune them or at least find out what they are actually doing?

I know that one is used for asset management and it has to run always in the background and the other is a lengthy report that starts every day at 7 AM and it never ends until the database goes down for the backup!

any comments will be appreciated.

Regards,
Khalid

 

[khalidaaa]

Oh yeah now i have a third job that has just started similar to the reporting one i have! It seems that some of the users scheduled that one!

System Configuration: lcpu=1 mem=5120MB
kthr     memory             page              faults        cpu     
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa 
 1  1 210665 998572   0   0   0  93  447   0 306  307 287 94  1  4  1
 3  0 210669 998567   0   0   0   0    0   0 265 2374 299 96  4  0  0
 3  0 210669 998567   0   0   0   0    0   0 246 2122 276 99  0  0  0
 4  0 210669 998567   0   0   0   0    0   0 252 2326 281 99  1  0  0
 3  0 210669 998567   0   0   0   0    0   0 251 2121 303 99  0  0  0
 3  0 210669 998567   0   0   0   0    0   0 256 2323 281 99  1  0  0
 3  0 210669 998567   0   0   0   0    0   0 249 2117 283 99  1  0  0
 3  0 210669 998567   0   0   0   0    0   0 258 2333 311 99  0  0  0

Name            PID  CPU%  PgSp Owner           
oracle        48816  32.8   2.1 oracle
oracle        47960  32.0  26.6 oracle
oracle        46690  31.5  26.0 oracle

 

[DukeSSD]

You are not waiting for IO so file system buffers and IO raw speed look OK, you are not paging so no resources wasted there, looks like you have plenty of paging space, 99% CPU is used by user, so the tuning looks OK you just need more / faster processors.

ps -efk will give you an idea of what is happening, if not then you will probably need to read up on kdb.

 

[khalidaaa]

Thanks DukeSSD for your comment.

I did ps -efk but i couldn't find any thing relevent!

So i'm working my way out with kdb! Though, i never used it and i don't know what to look for in it but i will try to read more on that!

Do you have any suggestions on how to best use the kdb?

 

[khalidaaa]

Just one more thing, how about if i try to upgrade this LPAR to 5.3 and make use of the virtual processors in there!

In our old machine, there are two CPUs and one of the processes above are always binded to this CPU! so it might be the coding of that application (which i think having issues) is binding the processes to the CPUs somehow!

So if i have more virtual CPU, this might help in the problem!

What do you think?

 

[masrolyat]

The applications is using all cpu time executing its own code that's what the us column means in vmstat. Since you are not paging nor do you have I/O wait, your hands as a system admin are tied. Either the DBA can manage the workload better or seek assistance from Oracle support why their code is taking so many cpu cycles. Oracle support will most likely tell the DBA's that, that version of Oracle is NOT supported on AIX 5.2 and above.

If you did not give the test LPAR 2 processors like the SP2 has, then you could add a processor using DLPAR.

I suppose that you could ask management to hire 10-20 programmers and buy the source code for the old version of Oracle so that you can port this old version of Oracle to
AIX 5.2 --- Ok, ok sorry for the sarcasm.

 

[khalidaaa]

I really like your comment masrolyat i wish if i can say the exact words to the management but i guess i will just rephrase what you just said to them!

I did talk to our DBAs which had no clue on how to tune those two oracle instances!! They just forwarded this to the application vendor and kept quite!

Thanks for the helpful input

Regards,
Khalid

 

[DukeSSD]

What is the performance like? I suspect the machine is only running oracle and it will be a lot faster than the old machine because the p5 is faster. Can you just live with it eatng the whole cpu (s)?
Just give it enough CPU to get the performance you need and do all your other work on other LPARs. With micro-partitioning you should be able to tune this quite finely.

Could WLM put a cap on it? (I know nothing about WLM so ignore that comment if you like).

 

[khalidaaa]

As i said our problem in the old machine that we don't have enough CPU to run this system smoothly! and the reason why we went to 5.2 is that the application is not certified to run on 5L so the management decision (I didn't agree on that but i had to go with what they said though they shouldn't interfere with technical stuff) was to go to 5.2 instead of 5.3!

What you said should be applicable if using 5.3!

WLM! I don't know about this topic too!! i was supposed to go for a course on that but i haven't been there yet!

But i think it should do the work!

Thanks for your input DukeSSD I will keep you updated on what will happen to this by the end of the day.

Regards,
Khalid