Too many "wait" processes

[miteshdmanek]

I have an AIX 5.2 system with 8 CPUs. When I do a ps -ef | grep wait I see I have 32 processes out there (assuming 4 per CPU). Only 8 (in accordance with number of physical procs) have CPU time attached to them. Just wondering what causes so many to be spawned? Is this normal? Can it be detrimental to performance. Below is the output:

# ps augc | grep wait
root 516 9.1 0.0 16 16 - A Jul 13 16131:28 wait
root 1548 8.3 0.0 16 16 - A Jul 13 14754:59 wait
root 2064 8.3 0.0 16 16 - A Jul 13 14737:27 wait
root 1806 8.3 0.0 16 16 - A Jul 13 14717:13 wait
root 2322 8.2 0.0 16 16 - A Jul 13 14572:34 wait
root 1290 8.1 0.0 16 16 - A Jul 13 14482:50 wait
root 1032 7.6 0.0 16 16 - A Jul 13 13478:57 wait
root 774 5.6 0.0 16 16 - A Jul 13 9993:22 wait
root 7224 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 7482 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 4386 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 6708 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 6966 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 4128 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 7998 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 7740 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 5160 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 4644 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 4902 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 5934 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 6450 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 6192 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 5418 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 5676 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 3096 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 2580 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 3870 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 8514 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 8256 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 2838 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 3354 0.0 0.0 16 16 - A Jul 13 0:00 wait
root 3612 0.0 0.0 16 16 - A Jul 13 0:00 wait

 

[khalidaaa]

well, That's what i found in the performance redbook explaining the vmstat command

CPU idle time. This indicates the percentage of time the CPU is idle
without pending I/O. When the CPU is idle, it has nothing on the run
queue. When there is a high aggregate value for id, it means there
was nothing for the CPU to do and there were no pending I/Os. A
process called wait is bound to every CPU on the system. When the
CPU is idle, and there are no local I/Os pending, any pending I/O to a
Network File System (NFS) is charged to id.

but you are having 4 bound to each CPU i believe! so how big is the run queue for each CPU? I'm not sure but i'm assuming run queue of 4 so 4 waits for each CPU!

Regards,
Khalid

 

[miteshdmanek]

I am not sure what the run queue is by CPU. Is this something I can look up? Is it configurable? What is the appropriate run queue for a Oracle DB server?

 

[khalidaaa]

There is a run queue for each CPU. If the number of processes waiting in run queues exceeds the number of CPUs then you will have an overloaded CPU-bound system. But that doesn't apply to your case!

Refer to this document for more information:

http://www.dba-oracle.com/art_tuning2.htm

I believe that the first 8 waits in your post is occupying some percentage of the CPU but the rest are not! (0.0)

hmmm, let me dig out more on this and will come back to you

Regards,
Khalid

 

[zaxxon]

Maybe this helps:

http://www-304.ibm.com/jct09002c/isv/tech/faq/individual.jsp?oid=1:89156

laters
zaxxon

 

[khalidaaa]

yeah but why 32 processes are there? why not just 8 as the number of CPUs?

 

[zaxxon]

Just did a:

ps aux| grep -v grep| grep -c wait
32

lsdev -C| grep proc
proc0           Available 00-00         Processor
proc1           Available 00-01         Processor
proc2           Available 00-02         Processor
proc3           Available 00-03         Processor

So 32 kernel wait processes while having 4 CPUs in the box. The box is not having any problems, performance is fine and it is an AIX 5.2 ML 5.
I don't know why there are more kernel wait processes than CPUs, but it seems to be normal.

On another box we have AIX 5.2 ML 5 too; did following:

ps aux| grep -v grep| grep wait
root       24588 18.1  0.0   48   44      - A      Jun 21 63457:14 wait
root       20490 18.0  0.0   48   44      - A      Jun 21 63394:44 wait
root        8196 18.0  0.0   48   44      - A      Jun 21 63290:37 wait
root       12294 17.9  0.0   48   44      - A      Jun 21 63002:17 wait
root       16392 17.6  0.0   48   44      - A      Jun 21 62023:59 wait
root       28686  0.0  0.0   48   44      - A      Jun 21  0:00 wait

lsdev -C| grep proc
proc0           Available 00-00         Processor
proc1           Available 00-01         Processor
proc2           Available 00-02         Processor
proc6           Available 00-06         Processor
proc7           Available 00-07         Processor

Only 6 kerne wait processes with 4 CPUs.

I compared the number of kernel wait processes and saw that the number of them having non-zero exec TIME are equal to numbers of CPUs in the system.
The others having zero exec TIME are, so I guess, just maybe dummies or only used in special circumstances. Since they eat no bread and the systems run fine, I don't really care, to be honest

laters
zaxxon

 

[khalidaaa]

That's what i got on AIX 5.2!?!

# ps aux | grep -v grep | grep -c wait
64

# lsdev -C | grep proc
proc0      Available 00-00         Processor

# oslevel -s
5200-08-00

on AIX 5.3 i got this

# ps aux | grep -v grep | grep -c wait
2

# ps aux | grep -v grep | grep  wait  
root      49176  0.2  0.0   40   40      - A      Aug 03 36:42 wait
root       8196  0.1  0.0   40   40      - A      Aug 03 24:18 wait

# oslevel -s
5300-04-03

# lsdev -C | grep proc
proc0      Available 00-00         Processor

but because this server is running SMT so it is normal to have 2 wait procs?

Maybe it is a bug in AIX 5.2 or so!?! i'm not sure!

Regards
Khalid