Parallel Batch Processing

[yukonxl]

Any help would be greatly appreciated...

I need to write a unix shell script that will execute various batches of programs (some in parallel).

Example:
I have six programs and some can run in parallel. Programs sharing the same BATCH_NUMBER can run together; programs with BATCH_NUMBER i have to wait for all programs with BATCH_NUMBER i - 1 to complete before executing.

PROGRAM BATCH_NUMBER
P1.sh 1
P2.sh 1
P3.sh 1
P4.sh 2
P5.sh 2
P6.sh 3

My initial concept was to create a master control script and three batch scripts (one for each BATCH_NUMBER). The master control script would call each batch script sequentially (i.e., batch1.sh then batch2.sh then batch3.sh). Each batch script would execute each program within that batch in parallel (i.e., batch1.sh would execute P1.sh, P2.sh, and P3.sh in parallel and return to the master control script when all three programs complete.

My problem is that I don't know how to write the batch scripts in Unix (ksh). I've tried using the & command for background processing, but control is immediately returned to the master control script...it doesn't wait for the programs to complete.

Any ideas? Thanks!!!!

 

[raider2001]

The 'wait' command is what you need. It will wait for all children to complete.

 

[vgersh99]

your master will have to "spawn" batches in the background and to "wait" on every batch completion.

A batch spawned on the background will return a "jobid" - you'll "waid jobid".

man wait

vlad