How do I write a KSH script to run programs in parallel? 1

[markmulligansr]

How do I write a KSH script to run programs in parallel?

I'd like to pass the script a variable with the number of programs I'd like to run in parallel.

If for example, I have an input file with 100 table names and if I want to run a runstats program for 10 tables at a time in parallel with no more than 10 running at the same time, how would I do this?

I would want the script to wait on the 10 runstats programs that are running and when one finishes, I'd like it to read the next record from the input file and continuously have 10 of these runstats programs running in parallel until the end of the file is reached.

If there is a way that I can check the return code from each of these programs, I'd like to do that too.

Thanks for your help.

 

[Annihilannic]

I wrote a script that does exactly that... haven't used it for ages though. It's called parallelem although you can give it whatever name you like of course!

#!/bin/ksh
#
# Ver   Date            Changes
# 1.0                   Original
# 1.1   22/12/2004      Added 'eval' so it can handle shell redirections, etc.
# 1.2   19/10/2005      Added grep -v to allow comments in command file.
# 1.3   15/09/2006      Optionally change sleeptime.
# 1.4   22/09/2006      Cosmetic change to log output.
# 1.5   10/10/2006      Why not use 'grep -c'?

SLEEPTIME=${4:-15}

if [[ $# -ne 3 ]]
then
        echo
        echo "usage: $0 <threads> <searchstring> <commandfile> [ <sleeptime> ]"
        echo
        echo "This script is designed to run a batch of like processes in parallel"
        echo "while ensuring that only a certain number are running at any one time."
        echo
        echo " <threads> is the number of processes to run in parallel."
        echo " <searchstring> is the command name to search for in 'ps -eo comm' output"
        echo " <commandfile> contains the list of commands to run.  Use '-' to read from"
        echo " standard input."
        echo " <sleeptime> is the time to sleep between counting threads.  Default is 15,"
        echo " however it may need increasing for commands that are slow to start."
        echo
        exit 1
fi

PPROCS="$1"
SEARCHSTRING="$2"
COMMANDFILE="$3"
DATECMD="date +%Z-%Y-%m-%d-%H:%M:%S"

echo "$(${DATECMD}): $0 started."
echo "$(${DATECMD}): Using ${PPROCS} threads, search string \"${SEARCHSTRING}\"."
echo "$(${DATECMD}): Reading commands from \"${COMMANDFILE}\"."

# cat required here to support '-' for reading of stdin
cat ${COMMANDFILE} | grep -v ^# | while read COMMAND
do
        until [[ $( ps -eo comm | grep -c "${SEARCHSTRING}") -lt ${PPROCS} ]]
        do
                sleep ${SLEEPTIME}
        done
        echo "$(${DATECMD}): Running: \"${COMMAND}\"."
        ( eval ${COMMAND} ; echo "$(${DATECMD}): Completed: \"${COMMAND}\"." ) &
        sleep ${SLEEPTIME}
done

wait
echo "$(${DATECMD}): All commands processed."

Annihilannic.