Check bytes of the File transferred.

[ranjank]

Hi,

Is there any way where in i can check the bytes of the file that i have tranferred from the remote location.I want to write a shell script for the same.

--Ranjan.

 

[feherke]

Hi

To check the bytes ? Strange request. You probably want to verify a checksum of the file. For example an MD5, SHA1 or CRC32 with [tt]md5sum[/tt], [tt]sha1sum[/tt], or other tools.

Feherke.

http://rootshell.be/~feherke/

 

[Annihilannic]

I think he means file size?

Annihilannic.

 

[ranjank]

Hi,

bytes means i want to check the file size.ow i can use MD5 tool in shell script?

--Ranjan.

 

[feherke]

Hi

Of course you can.

Feherke.

http://rootshell.be/~feherke/

 

[ranjank]

I have not used MD5 tool till date.Can someone give me pointers on how to use it in the script.

 

[feherke]

Hi

Just like from command line.

But what does your script more exactly ? Copies one or more files and with what tool ?

Feherke.

http://rootshell.be/~feherke/

 

[ranjank]

My script copies one tar file say abc.tar from one location to other.The file size is 1.6 GB approx.Everytime i have check manually the file size properly transferred.I want to automate this process so that i don't have to do it manually.

--Ranjan.

 

[feherke]

Hi

With [tt]scp[/tt] and [tt]ssh[/tt] manually looks like this :

[blue]master #[/blue] md5sum lasagna.txt > checksum.md5

[blue]master #[/blue] scp lasagna.txt checksum.md5 slave:/tmp
lasagna.txt          100% |*****************************|   793       00:00
checksum.md5         100% |*****************************|    46       00:00

[blue]master #[/blue] ssh slave

[green]slave #[/green] cd /tmp

[green]slave #[/green] md5sum -c checksum.md5
lasagna.txt: OK

As you can see, the verification has to be done on the destination machine, so the copying method is important. Again : what tool you use to copy ?

Feherke.

http://rootshell.be/~feherke/

 

[Annihilannic]

Something like:

#!/bin/ksh

LOCALSIZE=$(ls -l localfilename | awk '{print $5}')
REMOTESIZE=$(
  ftp -n remotehost << HERE | awk '$NF == "remotefilename" {print $5}'
  user username password
  dir remotefilename
HERE)

if [[ "$LOCALSIZE" -ne "$REMOTESIZE" ]]
then
   echo "Remote file size $REMOTESIZE differs from local file size $LOCALSIZE"
fi

Annihilannic.

 

[Annihilannic]

feherke's method is better since it validates the file contents as well, although it will take a significant amount of time for a 1.6GB file.

Annihilannic.

 

[ranjank]

Thanks a lot Feherke and Annihilannic.

I will try this.

I use scp and Rsync on some files.

--Ranjan.

 

[feherke]

Hi

May I have a question, Annihilannic ?

I see that you use [tt]ls -l[/tt] to get the file size. [tt]du -b[/tt] would be much simple. I also use [tt]ls[/tt] because this :

[gray]# SuSE :[/gray]

[blue]master #[/blue] uname -a
Linux master 2.4.19-4GB #1 Wed Sep 25 18:58:10 UTC 2002 i686 unknown

[blue]master #[/blue] ls -l lasagna.txt
-rw-r--r--    1 master  users         793 2005-04-26 11:49 lasagna.txt

[blue]master #[/blue] du -b lasagna.txt
[red]4096[/red]    lasagna.txt

[gray]# Sorcerer :[/gray]

[green]slave #[/green] uname -a
Linux slave 2.6.11.12 #1 SMP Mon Sep 26 15:45:52 EEST 2005 i686 GNU/Linux

[green]slave #[/green] ls -l lasagna.txt
-rw-r--r-- 1 slave users 793 Aug  2 13:29 lasagna.txt

[green]slave #[/green] du -b lasagna.txt
793     lasagna.txt

Did you experienced similar problems too ?

Feherke.

http://rootshell.be/~feherke/

 

[Annihilannic]

What is Sorcerer, a Linux distro?

You may find that the filesystem type is different. It seems that du on ext2/3 reports the amount of space allocated to the file, including unused space in the last block, wherease on whatever filesystem you are using on Sorcerer it does not.

Remember du means "disk usage" so it makes sense for it to report the amount of space that is using on disk, rather than the real size of the file.


Annihilannic.

 

[Annihilannic]

Another example of this, now that I think of it, is when you have 'sparse' files... i.e. a file that some software has created and written, say, 3KB of data to offset 103438276. If you use du it will say that the disk usage is about 3KB, but if you use ls it will show a file size of around 100MB.

Annihilannic.

 

[feherke]

Hi

Yes, Sorcerer is a Linux distribution.

No, the filesystems does not differ. Both are ReiserFS.

Yes, I know what is [tt]du[/tt] for, and I know that is not a reliable measurement tool. But on same filesystem type I expected to behave identical.

Thanks for the answers.

Feherke.

http://rootshell.be/~feherke/

 

[PHV]

Anyway, I don't think that -b is a POSIX legal option for du ...

 

[feherke]

Hi

Hmm... Quite weak. I do not know how much are scripts used in Unix, but for example in Sorcerer the full package management ( downloadig, dependency checking, compiling, installing, upgrading ) is done by shell scripts. For such complex tasks I think the POSIX compatibility is passed.

Feherke.

http://rootshell.be/~feherke/

 

[PHV]

I think the POSIX compatibility is passed
Sorry to disagree as we are in a *general* unix scripting forum.
They are lot of dedicated *nix fora here for dialect's variations.

 

[Annihilannic]

No, the filesystems does not differ. Both are ReiserFS.

That's odd alright. Maybe it's due to different options being used when creating the filesystems? Or different versions of the kernel module or kernel code used to drive it (quite likely since one is a 2.4 kernel and the other is 2.6)... or different versions of du...?

Annihilannic.