AWK improvement / LOGIC suggestions 1

[AnotherAlan]

Morning All,

I have a requirement to copy files from a source dir to a destination dir based on the following criteria;

The source files cannot be changed / modified or moved and will hold 7 days worth of data. These files are application generated and can be created at any time of day.

The destination directory should only contain files that have not yet been copied from the source directory.
Files in here will be picked up and then deleted by another application.

My logic is to create the above scenario by using a logfile to monitor which files have been copied and to compare all new files in the source dir against this log prior to copying to the destination directory. Hope this makes sense.

Anyhow, I have written the below ksh script that does the job. It's not pretty but it works. However, as I have been trying to learn AWK for a number of years now (with limited success obviously) I am keen to know of a way to do this.
I'm also open to any suggestions on the logic I have used.
I'm not a programmer, just a lowly Admin, hence the more than likely poor choice.

#!/bin/ksh

# Declare Variables

COPY_LOG=TEST/copy_log.`date +%b-%e-%T`
SOURCE_DIR=TEST/test_source
SOURCE_DIR_LOG=TEST/test_source.log
DEST_DIR=TEST/test_dest
CHECK_LOG=TEST/check_log
FILES_2_COPY=TEST/files_2_copy.log

exec > $COPY_LOG 2>&1

# Grab list of cpty files from Source

ls -1 $SOURCE_DIR | grep cpty* > $SOURCE_DIR_LOG

# Compare source files against check log - create log of files to copy

[ -f $FILES_2_COPY ] && rm $FILES_2_COPY
fgrep -v -f $CHECK_LOG $SOURCE_DIR_LOG > $FILES_2_COPY

# Copy files in FILES_2_COPY to Destination directory

echo "Files copied to TEST_DEST directory"
while read filename; do
echo $filename
cp -p $SOURCE_DIR/$filename $DEST_DIR && echo $filename >> $CHECK_LOG

done < $FILES_2_COPY


Many thanks all

 

[LKBrwnDBA]

Take a look at the rsync command.



----------------------------------------------------------------------------
The person who says it can't be done should not interrupt the person doing it. -- Chinese proverb

 

[AnotherAlan]

Cheers LKBrwnDBA,

To my knowledge rsync will attempt to keep both directories in sync and as the app on the destination side will delete all files it captures I will be forever publishing uneccesary files. It wont keep a log of what has already been copied over and subsequently picked up by the app.

Thanks

 

[Annihilannic]

First some comments about the existing Korn shell script.

ls -1 $SOURCE_DIR | grep cpty* > $SOURCE_DIR_LOG

You can just uset ls $SOURCE_DIR | grep cpty > $SOURCE_DIR_LOG (or perhaps (cd $SOURCE_DIR ; ls cpty*) > $SOURCE_DIR_LOG). -1 is unnecessary because it is the default when the destination is not a tty. Also cpty* looks like a filespec, not a true grep regular expression, so probably won't have exactly the desired effect (although that's just being pedantic, the difference is slight).

[ -f $FILES_2_COPY ] && rm $FILES_2_COPY

You can omit this line, the file will be overwritten by
> $FILES_2_COPY in the next step anyway. In fact, you may as well eliminate the $FILES_2_COPY completely and just pipe your fgrep output directly into the while read loop, unless you want to keep the file for debugging or something.

Consider adding -x to your fgrep - this will prevent files that coincidentally contain the same substrings from matching, e.g. if cpty12 had previously been copied, it could prevent a new file called cpty123 from being copied later.

In my opinion adding some awk code to this script would be superfluous, it will only be replacing the job that fgrep is doing equally well. It's not really a task for a pattern matching and processing language.

Annihilannic.

 

[AnotherAlan]

Thanks Anni,

Great call on the -x to fgrep, that could have been nasty otherwise.
Same for the files_2_copy log and the ls -1.

I think maybe I should improve my ksh scripting before moving on to awk...!

Always appreciated
Cheers

Alan