Create a .SH script to download from external ftp to internal ftp and transfer to windows share... 1

[ghath]

Hi there, I hope you're good...

I am new to linux commands and shell scripting and would like to create a .sh script on a linux server to download data from an external FTP site to our internal ftp site and then transfer the data to a share on a windows fileserver. i would also like to create a notification email when the process is complete. Any help would be greatly appreciated?

Here is a script we use to upload data to the external FTP but I now need to create one for downloading the data:

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='example'
TMPUPABC='/ftp/FTPSITE/tmpupload'
MNTDIRABC='/mnt/server_mastershare/FTPSITE/To_example_ABC'
MNTARCABC='/mnt/server_mastershare/FTPSITE/Archive_Upload_ABC'
# Email settings
SUBJECT="FTP upload for ABC"
EMAIL="Joe.Bloggs@example.co.uk"
EMAILMESSAGEABC="/ftp/FTPSITE/FTPSITEemailmessageABC.txt"
FTPRESULT="/ftp/FTPSITE/ftpresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`
# Start of main script
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIRABC)" ]; then
mkdir /mnt/server_mastershare/FTPSITE/Archive_upload_ABC/$DATE
cp -r $MNTDIRABC/* /mnt/server_mastershare/FTPSITE/Archive_upload_ABC/$DATE
mv -f $MNTDIRABC/* $TMPUPABC
cd $TMPUPABC
echo "FTP to FTPSITE for ABC : found the following files and will now attempt to upload them"> $EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
tree $TMPUPABC >>$EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
echo "Listing of remote server">> $EMAILMESSAGEABC
echo "**********************************************************************">> $EMAILMESSAGEABC
# Connect to remote server using lftp and sync 'up'
lftp -u user,example sftp://example.co.uk <<EOD
cd from_gbo
cd ABC
mirror -R --Remove-source-files
ls >> $EMAILMESSAGEABC
quit 0
EOD
/usr/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGEABC
# Tidy files & folders
rm -rf $TMPUPABC/*
rm -rf $EMAILMESSAGEABC
else
echo "$TMPDUPABC is Empty"
fi

 

[ghath]

Help!

Please can anybody offer any direction, or if you need more information just ask?

Thanks in advance

 

[Annihilannic]

Well, it appears having that upload script already gives you a lot of clues; why don't you just try studying the code a little, rearranging it and changing the appropriate FTP commands to do what you require? Besides, it seems to use a special FTP client and commands ("lftp" and "mirror") that I personally haven't encountered, so you may have better luck anyway.

If you get stuck with the scripting, post here with specific questions.

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[ghath]

Thanks for your post above, I appreciate that the script already gives alot of clues but I'm a complete novice when it comes to creating a script from the begining and manipulating the above upload script to a downloads script is a daunting task and I have no idea where to start? I have been studying the code but I'm having problems with structuring the script.

I assume I need to add the following variables to my downloads script in the same way as on the uploads script?

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='example'
TMPDOABC='/ftp/FTPSITE/tmpdownload'
MNTDIRABC='/mnt/server_mastershare/FTPSITE/From_example_ABC'
MNTARCABC='/mnt/server_mastershare/FTPSITE/Archive_Download_ABC'
# Email settings
SUBJECT="FTP upload for ABC"
EMAIL="Joe.Bloggs@example.co.uk"
EMAILMESSAGEABC="/ftp/FTPSITE/FTPSITEemailmessageABC.txt"
FTPRESULT="/ftp/FTPSITE/ftpresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


If yes, I assume I then need to add the code directly below it for the following tasks:

1. connect to external ftp sites directory to be downloaded
2. download data to internal ftp
3. move data to internal windows share
4. create an archive copy on windows share
5. Add code to email out if data has been recieved.

If anybody can help me structure this script then it would be a real help, much appreciated in advance.

 

[ghath]

Please don't laugh. Here is what I have so far, it needs a lot more work. I'm just finding out from one of the senior helpdesk guys how the email EMAILMESSAGEEVF="/ftp/ABCEVF/ABCupemailmessageEVF.txt" file works as i haven't been able to find it on the server at that path! I'm hoping this text file can be copied and amended to be used as a template to be added to the script to send out an email when the transfer is complete.

Can a member on here have a look and see if the code below is even close to being correct for this process? Also how would I go about debugging the script on a windows machine?

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='password'
MNTDIREVF='/mnt/sc-la-fs-01_mastershare/ABC/From_ABC_EVF'
MNTARCEVF='/mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF'
# Email settings
SUBJECT="ABC FTP Download for EVF"
EMAIL="joe.bloggs@example.co.uk"
EMAILMESSAGEEVF="/ftp/ABCEVF/ABCdownemailmessageEVF.txt"
FTPRESULT="/ftp/ABCEVF/ftpdownloadresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


# Start of main script
# Connect to remote server using lftp and sync 'down'
lftp -u gbo,password sftp://example.co.uk <<EOD

mirror /to_gbo/DMEVF /ftp://sc-la-app-01/ABC_SFTP/For_system_EVF/
quit 0
EOD
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIREVF)" ]; then
mkdir /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cp -r $MNTDIREVF/* /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE


cd ?
echo "FTP from ABC for EVF : found the following files and will now attempt to download them"> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
tree $TMPUPEVF >>$EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
echo "Listing of remote server">> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
fi

Thanks in advance, I'm finding unix scripting a slow learning process

 

[Annihilannic]

The email message is created by the script. The first echo "blahblah" >$EMAILMESSAGEEVF creates the file, and the subsequent echo "****" >>$EMAILMESSAGEEVF lines append to that file. So there is no separate template.

You could debug the script on a Windows system by using Cygwin which allows you to run a bash shell and a bunch of GNU utilities under Windows; however I would test it on the target server just using test files and directory locations until you're ready to switch them to the real locations. That way you know that the commands and utilities you are depending on are definitely available (such as, for example, that lftp client you are using, which may not be available for Cygwin).

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[ghath]

Thanks Annihilannic, your posts are much appreciated. Here is the script as it stands, can you see any glaring mistakes before I start testing it on the unix server? Thanks

#!/bin/bash
# FTP settings
HOST='example.co.uk'
USER='user'
PASSWD='password'
TMPDOEVF='/ftp/ABCEVF/tmpdownload'
MNTDIREVF='/mnt/sc-la-fs-01_mastershare/ABC/From_ABC_EVF'
MNTARCEVF='/mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF'
# Email settings
SUBJECT="ABC FTP Download for EVF"
EMAIL="joe.bloggs@example.co.uk"
EMAILMESSAGEEVF="/ftp/ABCEVF/ABCdownemailmessageEVF.txt"
FTPRESULT="/ftp/ABCEVF/ftpdownloadresult.txt"
# Other variables
DATE=`date +%Y-%m-%d_%H%M`


# Start of main script
# Connect to remote server using lftp and sync 'down'
lftp -u gbo,password sftp://example.co.uk <<EOD

mirror /to_gbo/DMEVF /ftp://sc-la-app-01/ABC_SFTP/For_system_EVF/
quit 0
EOD
# look for empty dir and generate email if files found
if [ "$(ls -A $MNTDIREVF)" ]; then
mkdir /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cp -r $MNTDIREVF/* /mnt/sc-la-fs-01_mastershare/ABC/Archive_Download_EVF/$DATE
cd $TMPDOEVF
echo "FTP from ABC for EVF : found the following files and will now attempt to download them"> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
tree $TMPDOEVF >>$EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF
echo "Listing of remote server">> $EMAILMESSAGEEVF
echo "**********************************************************************">> $EMAILMESSAGEEVF

/usr/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGEEVF
# Tidy files & folders
rm -rf $TMPDOEVF/*
rm -rf $EMAILMESSAGEEVF
else
echo "$TMPDOPEVF is Empty"
fi

 

[Annihilannic]

Since you've bothered to define the MNTARCEVF variable, you may as well use it here:

mkdir $MNTARCEVF/$DATE
cp -r $MNTDIREVF/* $MNTARCEVF/$DATE

Also the code seems to be slighly confused between MNTDIREVF and TMPDOPEVF and TMPDOEVF... e.g. your conditional statement checks for the presence of MNTDIREVF, but then the error message reports that TMPDOPEVF (which is not defined anywhere... extra "P" perhaps) is empty.

When the email message says "will now attempt"... is it referring to a future action (to be implemented) or the lftp mirror step that was just completed?





Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[ghath]

Thanks for the last post, I have made the changes as you suggested and think I'm ready for testing.

I need help to debug the shell script in an effective way but have no idea how, I've heard that using the SH command is the most effective way but am finding this difficult to read up on. I have been looking for a dummys guide online but nothing is simple enough to follow.

Do I just copy the script onto the linux server and set the scripts permissions to read and execute first and then use SH (inside script or when running script?) and then add flags to show the relevant errors?

Any instructions will be much appreciated on how to debug this effectively, the simpler the better! Thanks

 

[PHV]

At the command prompt on the linux server:
sh -x /path/to/script

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[ghath]

PHV, thanks for the post.

I'm using Putty SSH to connect to the server and the above command isn't working, I assume this needs to be run on the local server? Is there a way of doing this through putty?

Thanks

 

[Annihilannic]

The exact error message you are getting would be helpful.

Try bash -x /path/to/your/script perhaps?

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]