submitting multiple programs at a time and random reordering of a list 1

[philipose]

Hi,
I have a list of 50 (or more) jobs to run in a text file. I need to run 1, 2 or 3 jobs at a time till they all complete. Could any of you suggest how could this be achieved ? Also is there a way to sort this job list in a random fashion ?

Thanks very much
philipose

 

[khalidaaa]

If you elaborate more on the above i might be more helpful

But i guess you can do that in a C program with multi-threading process!

 

[RodKnowlton]

If you install GNU make, you can also do it with a makefile and "-j 3" (but not the randomness part).



IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+

Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.

 

[philipose]

Thanks for the suggestions and sorry for the delay. I can do without sorting a file in a random fashion but I still to submit jobs in parallel.

For Khalidaaa's question to elaborate what I was trying:

I have shell script abc.sh which contains 10 different programs (in this case they are all sas programs but they could be any task like gzip 10 files). I want them to be able to run 2-4 such tasks at the same time.

So if abc.sh contains the below lines

i) gzip abc1.dat
ii) gzip abc2.dat
iii) gzip abc3.dat
iv) gzip abc4.dat
v) gzip abc5.dat
vi) gzip abc6.dat

I want to be able to run i) and ii) together (or more than 2 jobs), if needed.
Thanks again
philipose

 

[RodKnowlton]

If you don't mind waiting for both i) and ii)) to finish before moving on, you can use:

gzip abc1.dat & 
gzip abc2.dat & 
wait
gzip abc3.dat &
gzip abc4.dat & 
wait
gzip abc5.dat & 
gzip abc6.dat &

If you're wanting to leave abc.sh as it is and have an external process parse and launch programs in parallel, you'll need to do some sort of multithread programming. Perl and Expect are both good for this.

If you want iii) to launch as soon as either i) or ii) finish, it definitely will require multithreading.

Rather than reinvent the wheel, it would probably be simpler to install GNU make and then build a Makefile from abc.sh. The -j switch to GNU make specifies how many tasks to execute in parallel.

- Rod


IBM Certified Advanced Technical Expert pSeries and AIX 5L
CompTIA Linux+
CompTIA Security+

Wish you could view posts with a fixed font? Got Firefox & Greasemonkey? Give yourself the option.

 

[philipose]

Thanks Rod,
I was intending something similar. I grouped the programs to run into n number of groups using the modulus function and ran them in the background using the & operator and used the wait command.
thanks a lot
philipose

 

[mrn]

Just been having a play

touch a b c d e f in /home/mrn/hold

#!/bin/ksh

dir=/home/mrn/hold
cd $dir

set *

shift $(($RANDOM % $#))

echo "$dir/$1"

produces a random file each time. I'll keep on playing to see if I can include an already run exclude list & run 3 jobs at once.

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[mrn]

#!/bin/ksh

cp /home/mrn/store/* /home/mrn/hold

a=`ls /home/mrn/hold|wc -l|sed 's/[ ]//g'`
count=0

until [ $count = "$a" ] ; do

dir=/home/mrn/hold
cd $dir

set *

shift $(($RANDOM % $#))

# swap this for command
echo "$dir/$1"
 
rm $dir/$1
count="$(( $count + 1 ))"
done

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[mrn]

The following allows two jobs to be run in parallel.

#!/bin/sh

(
while [ 1 ]
do
#execute other code above.

done
)

exit $?

Need food now, hope this helps

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."