Running command simultaneously, please help 3

[h3nd]

Hi guys,

I really need a favour here.
Currently I have 4 CPUs in my server,

I need to load the 144 data, if I load the data sequencially. It takes about 2.5 hours which is not good.

So I need to load the data 3 data at a time assign to each CPU,

Here's roughly my script atm :

for file in `ls`
do
    [load command] $file > log$file
done

what I want is roughly like this

for file in `ls`
do
  [load command] $file > log$file & [load command] $file > log$file & [load command] $file > log$file
done

And I need that 3 loading working independently, so whenever loading finish it doesnt have to wait for other to finish but continue loading the next one.

Can you guys shed some light here ?

Any input will be much appreciated.

 

[h3nd]

Hi SamBones,

is this correct ?
while (( $(jobs -p|wc -l) >= 3 ))

Cause I tried to type manually in shell :
this parameter didnt produce the value.
$(jobs -p|wc -l)

but this one yes :
$(jobs -p) | wc -l
0

Anyway, I put my script like this :

while [[ $(jobs -p|wc -l)>=3 ]]
                do
                        sleep 10
                done

It works, but I dunno whether this is correct.

 

[h3nd]

Hi Anni,

I think I ended up using your command,

$( ps -eo comm | grep "${SEARCHSTRING}" | wc -l )

cause it waits, using "jobs -p" doesnt wait, why ?

If I have 17 files, it loads 17 files at a time ???
which is not good ?

Any explanation why ?

 

[Annihilannic]

Sam's code seems to work for me, although the jobs command seems to behave strangely for non-interactive shells (on Solaris anyway). jobs on its own says "<command unknown>" instead of the command name, and jobs -p lists the correct number of process IDs, but they are all "0". I think I noticed this was unreliable before, which is why I chose not to use it in the script.

Annihilannic.

 

[h3nd]

Hi Anni/All,

I want to change the "wait" to others but I'm not sure whether it's gonna work.

Here's my script:

Load_Trade(){
for file in `ls trade`
do
    [load command] $file > log$file 2>&1 &

    while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
    do
        sleep 5
    done

    until [[ $(ps -fu adm | grep "loadcommand" | grep "trade" | wc -l) -eq 0 ]]
    do
        sleep 5
    done
done
       
Load_Market(){
for file in `ls market`
do
    [load command] $file > log$file 2>&1 &

    while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
    do
        sleep 5
    done

    until [[ $(ps -fu adm | grep "loadcommand" | grep "market" | wc -l) -eq 0 ]]
    do
        sleep 5
    done
done

}

I have to execute in Ctrl-M Load.ksh Trade and Load.ksh Market concurrently.
Because if I use "wait" they both have to be finish at the same time.

You reckon my script is working ?

 

[Annihilannic]

You need to put your 'until' loops after your 'for' loops, not inside them.

Also, I see no terminating } for the Load_Trade function (maybe just a copy and paste error?).

Annihilannic.

 

[h3nd]

lol.. sorry, I typed wrong.. it's been along day...

Load_Trade(){
for file in `ls trade`
do
[load command] $file > log$file 2>&1 &

while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
do
sleep 5
done
done

until [[ $(ps -fu adm | grep "loadcommand" | grep "trade" | wc -l) -eq 0 ]]
do
sleep 5
done
}

Load_Market(){
for file in `ls market`
do
[load command] $file > log$file 2>&1 &

while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
do
sleep 5
done

done

until [[ $(ps -fu adm | grep "loadcommand" | grep "market" | wc -l) -eq 0 ]]
do
sleep 5
done
}
[/code]

is that gonna work same as wait ?

And I'll check in top, my cpu running around 250%-350% from max 400% because I have 4 cpus. do you think it's still ok or should I reduce it ? what's the recommended ?

Thx man

 

[Annihilannic]

That looks pretty good to me. 250-350% sounds good to me. If your server is going to be doing nothing else at the time this job is running I would aim for 400%.

Annihilannic.

 

[h3nd]

Isn't it gonna crash the server if it's full loaded ?

Dont you think so ?

 

[Annihilannic]

Of course not. Servers don't normally crash just because they're busy, only if they are completely hammered, or running certain operating systems whose names I won't mention...

Annihilannic.

 

[h3nd]

only if they are completely hammered, or running certain operating systems whose names I won't mention...

Sorry man, I dont understand this.
Can you give me any example ?

One more question, what's the best book you recommended for me to buy for korn shell ?

And also for perl ?

I used to learn perl, but now I already forgot

Thx man

 

[Annihilannic]

completely hammered - i.e. running at 10000% CPU, or trying to use 5GB of memory in a system with only 2GB of physical memory and 2GB of swap, for example.

certain operating systems whose names I won't mention - think Microsoft....

I have never bought a book for any shell, the man pages and tek-tips are enough. However, if you must, the O'Reilly books on those subjects are probably a good start.

Annihilannic.

 

[h3nd]

Hi Anni,

I found somethin interesting in my other server here, this is my big server which has link to other 4 server (20 cpus) and the other one is small one which has 1 server with 4 cpus.

I checked the top in, it says like this :

 19:52:24  up 58 days, 23:18,  2 users,  load average: 2.13, 2.45, 2.81
681 processes: 675 sleeping, 6 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total   48.2%    0.0%   22.4%   0.0%     0.5%    1.3%   27.4%
           cpu00   63.4%    0.0%   11.6%   0.0%     0.1%    1.3%   23.3%
           cpu01   45.4%    0.0%   24.3%   0.0%     0.9%    0.9%   28.2%
           cpu02   48.0%    0.0%   25.2%   0.0%     0.7%    1.5%   24.3%
           cpu03   36.1%    0.0%   28.4%   0.0%     0.1%    1.3%   33.7%

why the total is not the total from cpu00+cpu01+cpu02+cpu03, my small server showed me different which is the real total of those.

 

[Annihilannic]

What operating system(s)? What versions of top?

That top output looks normal to me, total would not normally be the other figures added up, but the average across all CPUs.

In some versions of top you can use the I (capital 'i') command to switch between Solaris and Irix views of SMP (symmetrix multi-processing, i.e. multiple CPUs). In the Irix mode the totals really are totals.

Annihilannic.

 

[h3nd]

top version :
procps version 2.0.17

OS :
LSB_VERSION="1.3"
Red Hat Enterprise Linux AS release 3 (Taroon Update 8)

There is no top I dude,
$ top I
top: Unknown argument `I'
usage: top -hvbcisqS -d delay -p pid -n iterations

hmmm, strange isnt it?

 

[PHV]

why the total is not the total from cpu00+cpu01+cpu02+cpu03
Seems that the total line is the overall average ?

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[Annihilannic]

You don't use top I. You type 'I' when top is already running. Type '?' to see all of the available options.

Are both of those servers running the same OS and version of top? If so, it would be very strange for them to be calculating the total differently...

Annihilannic.

 

[h3nd]

$ top '?'
top: Unknown argument `?'
usage: top -hvbcisqS -d delay -p pid -n iterations

same Anni, yeah they are running in the same OS and version of top, and I think it's in one server

because i typed it in server 01.

 

[Annihilannic]

You didn't read/understand what I said. ? and I are not arguments to the top command. They are keys that you press when top is running.

Annihilannic.

 

[PHV]

BTW: man top

 

[h3nd]

Hi Anni/All,

Thanks for top explanation, I got real total cpu thx.

Now, I wanna ask about my script

Load_Trade(){
for file in `ls trade`
do
    [load command] $file > log$file 2>&1 &

    while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
    do
        sleep 5
    done
done

    until [[ $(ps -fu adm | grep "loadcommand" | grep "trade" | wc -l) -eq 0 ]]
    do
        sleep 5
    done
}
       
Load_Market(){
for file in `ls market`
do
    [load command] $file > log$file 2>&1 &

    while [[ $(ps -eo comm | grep "loadcommand" |wc -l) -ge 3 ]]
    do
        sleep 5
    done

done

    until [[ $(ps -fu adm | grep "loadcommand" | grep "market" | wc -l) -eq 0 ]]
    do
        sleep 5
    done
}

This is not working properly, which means the script finish before the loading actually finish.

I can tell that when I see the script finish but something seemed still working at the background and give me the different result.

is there any way to change "wait" command but can distinguish between 2 module ?