Question about sar, CPUs, cores 4

[zaxxon]

Hi,
we have a LPAR which has 5 CPUs assigned. Afaik, those Power 5 CPUs have 2 cores each.
I was monitoring CPU usage with

sar -PALL 1 100

and got

AIX srdwhv05 3 5 00C6C33F4C00    06/20/08

System configuration: lcpu=10 ent=5.00 mode=Capped

11:06:57 cpu    %usr    %sys    %wio   %idle   physc   %entc
11:06:58  0        1      51       0      48    0.01     0.3
          1        4      44       0      52    0.01     0.2
          2       99       1       0       0    0.91    18.2
          3        0       1       0      99    0.09     1.8
          4        0       6       0      94    0.09     1.8
          5       98       2       0       0    0.91    18.2
          6       99       1       0       0    0.91    18.2
          7        0       1       0      99    0.09     1.8
          8       98       2       0       0    0.92    18.4
          9        0       1       0      99    0.08     1.6
          U        -       -       0      20    0.98    19.5
          -       72       1       0      27    4.02    80.5
11:06:59  0        1      68       0      31    0.01     0.2
          1        6      44       0      50    0.01     0.1
          2       99       1       0       0    0.91    18.2
          3        0       1       0      99    0.09     1.8
          4        0       8       0      92    0.09     1.8
          5       98       2       0       0    0.91    18.2
          6       99       1       0       0    0.91    18.2
          7        0       1       0      99    0.09     1.8
          8       98       2       0       0    0.92    18.4
          9        0       1       0      99    0.08     1.6
          U        -       -       0      20    0.99    19.7
          -       72       1       0      27    4.01    80.3

I don't know how those "cpu" column is assigned to CPUs, ie. if 0 & 1 = 1 CPU or if it's a random assignment. What I noticed is, that always one of those "cpu" ie. cores is very busy, while the next is idling. Can anybody explain this to me? Do I have to ignore those "idlers" because they are just displayed and don't take into account?

laters
zaxxon

 

[khalidaaa]

The other 5 CPUs are used when there is a multi-threaded process running! Multi-threading is enabled by default and can be seen using smtctl command.

Have a look at this document:

http://publib.boulder.ibm.com/infoc...pic=/com.ibm.aix.cmds/doc/aixcmds5/smtctl.htm

Regards,
Khalid

 

[spamly]

You'll see something similar in nmon when graphically represented. I was told (by a vendor) that the second core would get used more when the system really got busy.

x CPU-Utilisation-Small-View qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x 0----------25-----------50----------75----------100x
xCPU User% Sys% Wait% Idle%| | | | |x
x 0 18.4 2.0 13.9 65.7|UUUUUUUUU

http://WWWWWW

> |x
x 1 0.0 0.0 0.0 100.0|> |x
x 2 75.6 3.5 10.9 10.0|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs

http://WWWWW>

|x
x 3 0.0 0.0 0.0 100.0| > |x
x 4 28.0 1.5 31.5 39.0|UUUUUUUUUUUUUU

http://WWWWWWWWWWWWWWW>

|x
x 5 0.0 0.0 0.0 100.0|> |x
x 6 19.5 2.5 49.5 28.5|UUUUUUUUUs

http://WWWWWWWWWWWWWWWWWWWWWWWW

> |x
x 7 0.0 0.0 5.0 95.0|WW> |x
x 8 30.0 3.5 6.5 60.0|UUUUUUUUUUUUUUUsWWW > |x
x 9 0.0 0.0 0.0 100.0|> |x
x 10 23.0 3.5 2.0 71.5|UUUUUUUUUUUsW > |x
x 11 0.0 0.0 0.0 100.0| > |x
x 12 20.5 1.0 12.5 66.0|UUUUUUUUUU

http://WWWWWW

> |x
x 13 0.5 0.5 0.0 99.0| > |x
x 14 18.0 0.0 8.5 73.5|UUUUUUUUUWWWW > |x
x 15 0.5 0.0 0.0 99.5| > |x
x 16 19.0 0.5 3.0 77.5|UUUUUUUUUW > |x
x 17 0.0 0.0 0.0 100.0| > |x
x 18 9.5 1.0 4.5 85.0|UUUUWW > |x
x 19 0.0 0.0 0.0 100.0|> |x
x 20 16.1 40.7 3.0 40.2|UUUUUUUUssssssssssssssssssssW> |x
x 21 0.0 0.0 0.0 100.0| > |x
x 22 27.5 2.5 9.0 61.0|UUUUUUUUUUUUUs

http://WWWW>

|x
x 23 0.0 0.0 0.0 100.0|> |x
xPhysical Averages +-----------|------------|-----------|------------+x
xAll 29.2 1.9 6.8 62.2|>UUUUUUUUUUUUUWWW |x
x +-----------|------------|-----------|------------+x
x Network qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx

 

[zaxxon]

Thanks for the info guys. I know smtctl but there is no note on when the 2nd core is being used. It's about multithreading but no word about behaviour of the 2nd core.

laters
zaxxon

 

[khalidaaa]

I'm not sure if i got what you wrote! the 2nd core is used for multithreading! That's the behavior for it?!?!

 

[zaxxon]

You said it is used when needed, if SMT is enabled. I checked your link to the man page for smtctl on IBM's Info Center. But there stood nothing about that second core. It is about SMT, and nothing is said about the behaviour on the 2nd core. I would like to read something about the behaviour, when and if the 2nd core is used.

Meanwhile I checked if the Informix DB processes are multithreaded, but they are not. There are some applications, native AIX processes, which are multithreaded though. So I am not sure if and when they are really used or better said - I would like to read it on some IBM docu, if possible

laters
zaxxon

 

[khalidaaa]

I know that from the IBM course that i attend I can't send you the material though

Regards,
Khalid

 

[zaxxon]

Ok, thanks though

laters
zaxxon

 

[p5wizard]

There is no second core. Only the CPU registers are duplicated. SMT allows for rapid switching between the two sets of registers, hence the applications "see" two CPUs when there is really only one.

My guess is (from your sar snapshot) that you have four (test?) programs running which are CPU bound and each of them occupies one thread of one CPU. Your LPAR has the use of 5 CPUs (ent=5.00), 5 virtual CPUs (which in this case each occupy one whole physical CPU of the server) and 10 SMT Threads (lcpu=10 in sar output). SMT is smart enough not to disturb those 4 Threads because the registers needed for those threads stay where they are - in one thread of the CPU.

The choice of which thread is active and which is idle is arbitrary in this instance. If you stop the programs and relaunch them, you'll probably see a different layout.


HTH,

p5wizard

 

[ogniemi]

another view

mpstat -s 1 10

 

[zaxxon]

Ah ok - I heared about a 2nd core, now I hear it's only one with additional registers. Good to know, ty
Those programs are not test programs - it's live production with an Informix Data Warehouse DB on AIX. I checked them already with svmon -P and they are sadly not multithreaded - though I thought the "2nd core" which isn't one could take one single threaded process too... nevermind
Thank you all for the clarification.

laters
zaxxon

 

[p5wizard]

Well, how many of those programs are running?
If you're only running 4 of 'em, you can probably get around with less CPU entitlement or less virtual CPUs. But I wouldn't experiment with those settings on the production server if I were you.


HTH,

p5wizard

 

[zaxxon]

Sadly I can't tell you exactly which processes these are becaue I have no snap of that from the time performance test was going on.
It should be oninit processes but there are usually plenty of them running (ie. more than 4-5), even when there is not much traffic.
I also checked performance tuning guidelines from IBM for Informix DBs on AIX and saw there are mentioned parameters for the onconfig.cfg for virtual processors (Informix VP) relating to hardware processors. There is also an option for multiprocessors which is enabled. We can play with the parameter(s) regarding those Informix VPs so far and I advised the DB admin to tell his developers to check their code with some Explain tool etc. if they are not unnecessary or unefficient written, which will make them insulted and so it will end that an Informix consultant will be invited to check their config etc.
I cancelled checking for more options on the AIX side since everything looks very good. If they don't find anything on the Informix side, they will simply assign more CPUs, I think.
Thanks though

laters
zaxxon

 

[khalidaaa]

p5wizard,

I remember something like this been said but i just looked into the internet and found this:

SMT allows for the ability of a single physical processor to concurrently dispatch instructions from more than one hardware thread. In AIX 5L Version 5.3, a dedicated partition created with one physical processor is actually configured as a logical two-way by default. In essence, two hardware threads can actually run on one physical processor at the same time. While there are isolated situations where turning on SMT can impact performance negatively, SMT is almost always the best choice, particularly when overall throughput is more important than the throughput of an individual thread. As a result of the POWER5's unique dual-core design and support for SMT, one POWER5 chip actually appears as a four-way microprocessor to the operating system. Processors using SMT can issue multiple instructions from different code paths during one single cycle. Figure 2, an illustration of a DCM, clearly illustrates the relationship between SMT and the chip itself.

http://www.ibm.com/developerworks/aix/library/au-aixoptimization/index.html

It says that the dual-core design appears as two processors using SMT to an AIX5.3 parition!

Regards,
Khalid

 

[p5wizard]

It says that the dual-core design appears as two processors using SMT to an AIX5.3 parition!

No it doesn't, it says: a dual-core design processor (2 cores on one chip) appears as a 4-way microprocessor (2 SMT threads per core, 2 cores on one chip) to an app running on AIX5.3...


HTH,

p5wizard

 

[khalidaaa]

hmmm

I think i'm lost here

What does that mean? dual-core which is 2 core in one chip appears as a 4-way microprocessor right? so 1 core will appear to be a 2-way microprocessor on AIX5.3 using SMT right?

Regards,
Khalid

 

[khalidaaa]

Oh yeah it seems fine now!

Thanks p5wizard for the clarification!

Regards,
Khalid