Parallel process, not linear

[ryanc2]

I have an ftp script that reads a control file with a list of file types (based on naming convention) to transfer. It will search for all files named according to the first type in the control file, then the second, etc. This runs throughout the day and moves files to/from an ftp server. However, while one set of files is being moved, it will not go to the next file type, causing slower results. I would like to make it so it will ftp files in parallel instead of waiting for the previous process to finish. The main FTP loop looks something like this:

exec < ${control_file}
IFS=:
while read mode type file ldir rdir
do
check for new files to send
while f=$(line)
do
FTP the files
check for completion
done < ${ftp_list}
done

Any help would be appreciated.

 

[stefanwagner]

I'm sceptical that this would improve the speed, as long as you don't ftp from different machines.
In a LAN, the hdd-speed could be the limitation, and in the web the networkspeed.

seeking a job as java-programmer in Berlin:

http://home.arcor.de/hirnstrom/bewerbung

 

[Chacalinc]

make 2 scripts, 1 to transfer only one file, and another one to call the transfer scripts, example:

::main.sh::
exec < ${control_file}
IFS=:
while read mode type file ldir rdir
do
#check for new files to send
while f=$(line)
do
# FTP the files
# NOTE THIS:
transfer.sh $f > log_all.log &
done < ${ftp_list}
done


::transfer.sh::

ftp server <<< EOF
user
password
cd target_folder
bin
put $f
bye
EOF

check for completion

 

[Chacalinc]

stefanwagner: software like prozilla, getitright, download accelerator, etc. make the same thing... this is because the FTP protocol has lot of overhead, so real transfer is worst than the current bandwidth.

Cheers.

 

[ryanc2]

Thanks for the input. I didn't necessarily mean to improve speed (mis-spoke), but rather more productive. If one file takes 10 minutes to transfer, I don't want to hold up small files that take just seconds. Currently, the smaller files are building up waiting for the large file to complete.

Chacalinc - I may try your suggestion of a seperate FTP script for accomplishing the transfer.

Thanks