Need help with automated FTP

jadelester · Mar 12, 2002

I need help with a script I am working on...

First, let me say that my scripting knowledge is limited, and what I have gotten so far has been either from what I do know, snipits from this site, or help from friends and books.

What I am trying to do is create a script where depending on what files are different between the local and remote servers, get the new files from the remote server, and depending on where they come from, store them in an appropiate directory, and change the first character in the filename to correspond with the location it was pulled from (i.e change file named G0000V00 to D0000V00 for dallas files, or K0000V00 for kansas files, etc (note that the "0's" will change, but will always be numeric).

Currently I have a script that will FTP to the server and log the output of "ls" to a local file, I then pass the file thru sed and diff (on a local and remote file list created by ls output) a few times to clean it up to get just the remote files that are not present on the local server.

Here is what I have so far:

This part creates a cleaned file with the remote files that are different from the local files in a specified directory.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
#!/usr/bin/ksh

# set variables

export IP_ADDRESS=209.51.15.30
export REMOTE_BASE_DIR=/home/jade/ftptest/
export UID=********
export PASWD=*******

export LOCAL_BASE_DIR=/export/home/jlester/ftptest/
export REMOTE_FILE_LIST=$LOCAL_BASE_DIR/remote_ls.output
export LOCAL_FILE_LIST=$LOCAL_BASE_DIR/local_ls.output

export TMP_FILE=$LOCAL_BASE_DIR/tmp

#temp variables
export FOO=$LOCAL_BASE_DIR/foo
export FOO2=$LOCAL_BASE_DIR/foo2
export FOO3=$LOCAL_BASE_DIR/foo3

# generate list of files on remote server.

ftp -inv <<-EOF >$TMP_FILE
open $IP_ADDRESS
user $UID $PASWD
cd $REMOTE_BASE_DIR
ls
quit
EOF

NUM_LINES=`wc -l <$TMP_FILE |sed 's/ //g'`
let LAST_GOOD_LINE=$NUM_LINES-6
sed -n "8,${LAST_GOOD_LINE}p" $TMP_FILE | sort > $REMOTE_FILE_LIST

# generate list of files on local server, and populate the temporary file with "get " before each filename.

ls $LOCAL_BASE_DIR | sort > $LOCAL_FILE_LIST
diff $LOCAL_FILE_LIST $REMOTE_FILE_LIST >$FOO
grep '>' $FOO | sed 's/^..//' > $FOO2
cat $FOO2 |sed 's/^/get /g' >$FOO3

exit 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Next comes the part that I am having the most trouble with. What I am trying to do is read in a control file with all the information for the servers I need to check. Note that there are multiple entries to this file. The control file format is:

IP:UID

ASS:RemoteDir:LocalDir

ex: 127.0.0.1:UID:/path/to/remotefile/:/path/to/local/file

The main problem that I am having is that I am dealing with several remote servers, and several different files per server. I need to be able to read the file into the script, and loop thru each line, AND somehow read the file "foo3" to determine which files are the different files, and initiate an FTP session to get the remote files (again storing them in the correct directory, with the first character changed).

This is what I have so far...

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
#!/bin/ksh

CfgFile=/export/home/jlester/scripts/tempftplist.conf
LogFile=/export/home/jlester/ftptest/ftplog

sed 's/:/ /g' $CfgFile | while read IPAddr UserName Password RDir LDir
do
#cat <<-FOO
ftp -inv <<-BAR >$LogFile
open $IPAddr
user $UserName $Password
cd $RDir
*** This is where I need help ********
bye
BAR
#FOO

echo
done

exit 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Of course I want to combine all of this into one script and log the output, and put some error checking in, but right now it is one step at a time, I just want to make it work. If there are any ways you can suggest to accomplish what I am needing, or see something I may have forgot, please feel free to speak up!

Sorry for being long winded, but I am trying to be as concise as possible. Thanks in advance for any help help you can provide.

Thanks

Jade Lester

pcunix · Mar 13, 2002

It's time for you to move to Perl.

Don't panic: you will like it, and it's going to make things like this SO much easier that the small amount of pain you endure to get past the initial learning will be repaid over and over and over.

Take a look at

http://pcunix.com/Unixart/loveperl.html

for a painless look at why you will love this.

That link will also point you at some good books, and I have lots of other sample perl stuff on my site too.

I know- you don't want to learn something new- please trust me; it's not that hard, a lot of it is very much like what you already know about sed and awk etc. but it's just so much BETTER. Easier, faster, better.

I put this off for years- wish I hadn't..

Tony Lawrence
SCO Unix/Linux Resources

http://pcunix.com

tony@pcunix.com

RobJordan · Mar 13, 2002

#!/bin/ksh

CfgFile=/export/home/jlester/scripts/tempftplist.conf
LogFile=/export/home/jlester/ftptest/ftplog

# Not sure how you are corelating your file list to the server they came from
# but I imagine you will need another loop surrounding the one below
# to read the files you will have to get - I will use the variable FILE below

# doesn't look like you have the LOCAL prefix name in your config file
# i.e. D for DALLAS, etc.
# I have to assume that you can add this to your CfgFile
# meaning each entry should also have a LOCAL

cat $SOME_FILE_LIST|while read FILE
do

sed 's/:/ /g' $CfgFile | while read IPAddr UserName Password RDir LDir LOCAL
do
OLD_PREFIX=`echo FILE|awk -F"[0-9]" '{print $1}' # i am assuming the 2nd pos is always numeric
POSTFIX=`echo $FILE|awk -F"$OLD_PREFIX" '{print $2}' #there's problaby a better way to chop the 1st character off than this - maybe using sed?
NEW_FILE="$LOCAL$POSTFIX"
#cat <<-FOO
ftp -inv <<-BAR >$LogFile
open $IPAddr
user $UserName $Password
*** This is where I need help ********
get $RDir/$LogFile LDir/$NEW_FILE
bye
BAR
#FOO

echo
done

done

exit 0

this is by no means a solution, but might give you some more ideas.
I was thinking that maybe you should dynamicly build the ftp section with a shell function
so that you could if needed get mutiple files from one server
rather than reconnecting for each file you need from a particular server

if i think of something else, I will post it later.

I agree with Tony that perl will be easier to do this task
if you have perl availabe to use.

Robert Robert G. Jordan

Robert@JORDAN2000.com

http://www.JORDAN2000.com/scripts

Unix Sys Admin
Chicago, Illinois U.S.A.
[lightsaber]

jadelester · Mar 13, 2002

Thanks guys!!

I will play with this and see what I can do with it. You are not the first to suggest perl, and I have built Perl5 on my system, but don't really have the time to spend learning it for this solution (under somewhat of a time crunch...). From what I have seen and heard it is a better language to script in, and I agree that it is something I need to add to my skills.

Again Thanks for the replys, and I will see what I can do!

Jade Lester

RobJordan · Mar 13, 2002

here's a better way to rename your files
and it will work with any file name up to 100 characters

FILE=G0000V00
LOCAL=D

POSTFIX=`echo $FILE|cut -c2-100` # this chops off the first charcater
NEW_FILE="$LOCAL$POSTFIX" #this appends the LOCAL to the new file name

echo $NEW_FILE
D0000V00

Robert
Robert G. Jordan

Robert@JORDAN2000.com

http://www.JORDAN2000.com/scripts

Unix Sys Admin
Chicago, Illinois U.S.A.
[lightsaber]

RobJordan · Mar 13, 2002

I wrote this in theory, but here's the premise.
1. look at each line of your master list reading the ips
2. look for the appearance of such ips in your transmit file list (you will need to add the ips in the 2nd field)
3. create a list of ips with new files and corresponding
files to be transmitted for that ip
4. dynamicly build the ftp statement and echo into a file
5. execute that file
6. repeat the process for every ip that new files

Robert

#!/usr/bin/ksh
TMP_LIST=/tmp/TMP_LIST.$$

# lets assume your file list XMT_FILE_LIST has the filename:ip address
# and CONFIG_FILE is your master config file

IPAddr=`echo $LINE|awk -F":" '{print $1)'`
UserName=`echo $LINE|awk -F":" '{print $2)'`
Passord=`echo $LINE|awk -F":" '{print $3)'`
RDir=`echo $LINE|awk -F":" '{print $4)'`
LDir=`echo $LINE|awk -F":" '{print $5)'`
LOCAL=`echo $LINE|awk -F":" '{print $6)'`

Build_FTP_Code ()
{
UserName=`grep $IPAddr CONFIG_FILE|awk -F":" '{print $2)'`
Password=`grep $IPAddr CONFIG_FILE|awk -F":" '{print $3)'`
RDir=`grep $IPAddr CONFIG_FILE|awk -F":" '{print $4)'`
LDir=`grep $IPAddr CONFIG_FILE|awk -F":" '{print $5)'`
LOCAL=`grep $IPAddr CONFIG_FILE|awk -F":" '{print $6)'`

echo "ftp -n $IPAddr<<END_SCRIPT" >> $TMP_LIST.ftp.IPAddr
echo "quote USER $UserName" >> TMP_LIST.ftp.IPAddr
echo "quote PASS $Password" >> TMP_LIST.ftp.IPAddr
cat $TMP_LIST.files.$IPAddr|while read LINE
do
FILE=`echo $LINE|awk -F":" '{print $1}'
POSTFIX=`echo $FILE|cut -c2-100`
NEW_FILE="$LOCAL$POSTFIX"
echo "get $RDir/$FILE $LDir/$NEW_FILE" TMP_LIST.ftp.IPAddr
done
echo "quit" >> TMP_LIST.ftp.IPAddr
echo "END_SCRIPT" >> TMP_LIST.ftp.IPAddrA

$TMP_LIST.ftp.IPAddr # this file executes with all the ftp info for currernt host
}

# i am not so great with sed - you may be able to subsitute the awk satements below
# your master config file with IPAddr UserName Password RDir LDir LOCAL
cat $CONFIG_FILE|while read LINE #find out which host we need to get files from
do
IPAddr=`echo $LINE|awk -F":" '{print $1)'`
FOUND=`grep $IPAddr $XMT_FILE_LIST`
if test -n "$FOUND"
then
echo $IPAddr >> $TMP_LIST.hosts
grep $IPAddr $XTM_FILE_LIST >> $TMP_LIST.files.$IPAddr
fi
done

HOST_COUNT=`cat $$TMP_LIST.hosts|wc -l`
if [ $HOST_COUNT -gt 0 ]
then
echo "$HOST_COUNT hosts have new files"
else
exit 1
fi

COUNT=0
until [ $COUNT -eq $HOST_COUNT ] # need to use until because you can't have nested while read statments
do
COUNT=`expr $COUNT + 1`
IPAddr=`sed -n "${COUNT}p" TMP_LIST.hosts`
Build_FTP_Code
done Robert G. Jordan

Robert@JORDAN2000.com

http://www.JORDAN2000.com/scripts

Unix Sys Admin
Chicago, Illinois U.S.A.
[lightsaber]

RobJordan · Mar 13, 2002

ps

you can remove these at the top of the script
I left them in by accident.

IPAddr=`echo $LINE|awk -F":" '{print $1)'`
UserName=`echo $LINE|awk -F":" '{print $2)'`
Passord=`echo $LINE|awk -F":" '{print $3)'`
RDir=`echo $LINE|awk -F":" '{print $4)'`
LDir=`echo $LINE|awk -F":" '{print $5)'`
LOCAL=`echo $LINE|awk -F":" '{print $6)'`

Robert G. Jordan

Robert@JORDAN2000.com

http://www.JORDAN2000.com/scripts

Unix Sys Admin
Chicago, Illinois U.S.A.
[lightsaber]

RobJordan · Mar 13, 2002

my bad - you can have nested while read statements
as long as they use different variable names.

so you can change this section

COUNT=0
until [ $COUNT -eq $HOST_COUNT ] # need to use until because you can't have nested while read statments
do
COUNT=`expr $COUNT + 1`
IPAddr=`sed -n "${COUNT}p" TMP_LIST.hosts`
Build_FTP_Code
done

to:
cat $TMP_LIST.hosts|while read HOST
do
IPAddr=$HOST
Build_FTP_Code
done
Robert G. Jordan

Robert@JORDAN2000.com

http://www.JORDAN2000.com/scripts

Unix Sys Admin
Chicago, Illinois U.S.A.
[lightsaber]

jadelester · Mar 15, 2002

Tony / Robert:

Thanks a bunch for your suggestions, they helped alot.

I was able to get it to work, using peices of your suggestions with the code I had written, plus some help from my local guru.

Here is the code that works, just in case you are curious.

It may not come out as clean as it should be due to the way the site displays indented text, but the jist is there..

Anyway, thanks again for your assistance!

Jade Lester

#!/usr/bin/ksh

set -a

# set variables
#
Date=`date +%y%m%d.%H%M%S`
BaseDir=/export/home/jlester/scripts/cdr_ftp

VarDir=$BaseDir/var
LogDir=$BaseDir/log
CfgDir=$BaseDir/cfg
FileListsDir=$BaseDir/filelists

BaseName=`basename $0`

LogFile=$LogDir/$BaseName.log.$Date

# send stdout, stderr to the log.
#
exec 1>$LogFile 2>&1

CfgFile=$CfgDir/CONFIG_FILE

TmpFile1=$VarDir/$$.tmp.1
TmpFile2=$VarDir/$$.tmp.2

Logit() {
LogDate=`date "+%D %T"`

case "$1" in
"&quot

echo ;;
"timestamp&quot

echo "$LogDate ==> (timestamp)" ;;
*) echo "$LogDate ==> $1" ;;
esac

if echo "$1" |/usr/xpg4/bin/grep -q ERROR
then
let ErrorCount=$ErrorCount+1
fi
}

Logit "$BaseName started"
Logit

# loop over all lines in config file.
#
grep -v '^#' $CfgFile |sed 's/:/ /g' |while read FTPServer UserName Password RemoteDir LocalDir
do
Logit "working on server \"$FTPServer\", remdir \"$RemoteDir\" localdir \"$LocalDir\""
Logit

DataSource=$FTPServer:`echo $RemoteDir | sed 's/\//_/g'`

AllFilesWeHaveEverFetched=$FileListsDir/$DataSource.history

CurrentRemoteFileList=$FileListsDir/$$.$DataSource.remote_files
FileListToFetch=$FileListsDir/$$.$DataSource.files_to_get
FTPInputFile=$VarDir/$$.$DataSource.ftp_input

# in case we are running for the first time.
#
touch $AllFilesWeHaveEverFetched

# generate list of files on remote server.
#
Logit "getting list of files from remote server"
Logit

rm -f $TmpFile1

ftp -inv <<-EOF
open $FTPServer
user $UserName $Password
cd $RemoteDir
ls -- $TmpFile1
bye
EOF

Logit

sort $TmpFile1 >$CurrentRemoteFileList

# list of files we have ever fetched should already be sorted from last run,
# so just use diff to see what is new on the remote server.
#
Logit "figuring out what files we need to get"
Logit

diff $AllFilesWeHaveEverFetched $CurrentRemoteFileList | grep '>' | sed 's/^..//' > $FileListT
oFetch

if [ -s $FileListToFetch ]
then
# build ftp input file to actually get files from remote server.
#
Logit "building ftp input file"
Logit

cat <<-EOF >$FTPInputFile
open $FTPServer
user $UserName $Password
cd $RemoteDir
lcd $LocalDir
EOF

sed 's/^/get /' $FileListToFetch >>$FTPInputFile

cat <<-EOF >>$FTPInputFile
bye
EOF

# now actually get the files.
#
Logit "getting files"
Logit

ftp -inv <$FTPInputFile

Logit

# update list of files we have ever fetched.
#
Logit "updating list of files we have ever fetched"
Logit

cat $AllFilesWeHaveEverFetched $FileListToFetch | sort >$TmpFile1
mv $TmpFile1 $AllFilesWeHaveEverFetched
else
Logit "no new files on remote server"
Logit
fi

Logit "==============================================================================="
Logit
done

Logit "cleaning up"
Logit

rm -f $VarDir/$$* $FileListsDir/$$*

Logit "$BaseName stopped"

exit 0

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Need help with automated FTP

jadelester

Technical User

pcunix

MIS

RobJordan

MIS

jadelester

Technical User

RobJordan

MIS

RobJordan

MIS

RobJordan

MIS

RobJordan

MIS

jadelester

Technical User

Similar threads

Part and Inventory Search

Sponsor