script question

[svar]

A hopefully quick question on the a problem that should be
common enough:

I have a script on a server. In this server files
arrive let's say randomly in bursts.
These files have a size (whan gzipped) of about 150K maximum. They arrive in
a proprietary zipped format. The task I am doing is:
1) use the proprietary program to unzip them,
2) use gzip on them (gzip -q9 file)
3) ftp them to another server
4) remove them.

The problem is that some files come corrupted,
I am not sure why this happens. COuld it be that
I am trying to unzip and then gzip a file that is still
"coming"?
If so, is there a standard practice in shell scripts
to prevent this. I suppose you cannot 'lock' the file, so
the best solution I can think of is
to put a delay or to go to sleep if the number of files
found is less than 2 (meaning the file may be still "in transit&quotor something like that.
How does one implement in sh
if(number of files in the current directory <2){....}

?

Any other ideas/suggestions/things to check are greatly
appeciated.

P.S. Ordinarily I would use something like perl for the
jo except it is not installed on this machine and
I am not allowed to install it

svar

 

[MrTom]

easiest method is to get the sender of files to upload them in to an upload folder and then move the file to a completed directory when finished. you can then just work on the completed dir.

 

[hvn]

Also, depending on the proprietary zip/unzip utility you're using, in theory you shouldn't be able to unzip the files if they're not complete, or else you'll receive some or other error when trying to unzip the files.

May guess would be that the file doesn't unzip since it's a partial file and your script then gzips the partial file.

If the proprietary zip/unzip utility has a &quot;test&quot; option, eg: gunzip -t filename, you could modify your script to first do a test on the compressed file, and if the test fails, wait for &quot;x&quot; seconds/minutes and do the test again.
Once the test exits without any errors you could then have the scriptfile procede with the unzipping, gzipping etc.

 

[svar]

Thanks MrTom and hvn,
I guess I should have been clearer. When I say proprietary format,
you should forget anything about zip, xompress and gzip. Period.
As a matter of fact the name of the proprietary zipper/unzipper is called
avViewer.

Ordinarily the solution would be to have the site that sends the
files in the first place to also send a file with checksums. Good idea, but forget
it, as it is not under my control. The grand scheme I was using is
files arrive, propietary unzip, then gzip, then send to another
machine via ftp. That other machine is under my complete command.
There, the files are gunzipped and parsed. If gunzip fails, then
the following process is executed:
telnet to the remote
go the the directory where the files are stored
grab that file and send it to my directory on the remote
the demon running there will make sure it is ftpd correctly
and then I should be able to gunzip filne.

This worked fine for a long time. However, it is not robust enough.
The reason is that I had a filesystem corruption(similarly if we had a power failure on my machine) , probably due to a bad SCSI disk sector
and as a result I am behind schedule. This means that when gunzip
fails and telnets to the remote to find that file stored, the file is no longer
there because it has been erased by various maintenance/disk cleaner
scripts

This means I must be more robust and make sure the proprietary unzip/gzip
works the first time.
This in turn implies some trick like:
if I only have a few files in the directory on the remote,
they may have not arrived completely. So what the script should do is sleep
and wake up a bit later, i.e. NEVER go to work on the last few files
(in terms of arrival time, which is also the alphabetical order)

So, this brings me back to the technical aspect: HOW do I tell
a script:
check number of files in current directory
if < 3 sleep

?

Thanks, svar

 

[RhythmAce]

Could you have your script run at particular time intervals? Then do an &quot;ls&quot; or something and put the directory list into a file called, let's say, &quot;queue&quot;. Then your script can pass the list to your zip/unzip program. That way anything coming in while this is in progress will not be in the list. There is a very slim chance that a file will be in the process of being transfered while &quot;queue is being written, but you could put a delay loop in the script of a few seconds before zipping if this is a problem.

 

[timsr]

You could write a script that checks the size of the file then sleep for a period, then check again. If the file sizes match, the compress uncompress should work unless the file is corrupted. I use a similar process to backup files to a cd burner.