file transfers 1

[screwloose]

Any suggestions on transferring 100's of gig of data over a network?

Options:
1) tar | compress files then ftp and untar (tar 2GB limit)
2) ftp individual files
3) rcp (though the speed would be the same as ftp pretty much if not a little slower)
4) cpio (but 2GB limit)
5) ???? (suggestions welcomed)

and the tar/compress methods takes up too much time, so I am trying to get around that.

Another thing to consider: many, many files that are 50+ GB in size, which tar and cpio don't allow.

Thanks.

 

[raztaboule]

Did you have "timing" constraint ? are the files frequently updated ? how many 100s of Giga on what kind of network ?

I don't have something better than ftp / rcp but ...
if i had that kind of thing to do i will consider "cuting" the files in order to send smaller files. In case of failure you have a point to restart, you don't have to redo the send for the entire file.
take the 10.000 first line. send . check it. take the 10.000 next line . send . check .append to first piece. and so on.
there will be overhead because of the read/create the piece to send/send it/check( with a checksum ?)/append processing but it will be much safe.

 

[adamsd]

Try "rsync". If you don't have it installed do a search in the web and down load it and look up the man pages. Its the BEST.. if something stops the data transfer, when restarted it will sync and start from where it stopped. I have used it and do large data transfers. You will also have to set your environment for it to work.
I don't know if this is the site but try it.

http://rsync.samba.org/

GOOD LUCK!!

 

[Boria]

Hi,
I do this using two tar's
# tar -cvf - DataToTransfer|rsh RemoteHost "cd Location;tar -xvf -"

and never have a problem.
Regards Boris.

 

[raztaboule]

thank you adamsd for the rsync link.
looks nice !

 

[screwloose]

Thanks for the suggestions.

rsync is a great tool. I implemented it at my last job because they were tar'ing up a directory, then doing an nfs mount, then a copy, unmount nfs, then a untar, then change perms, etc.........

Needless to say they really liked rsync, but they were not sure at first because they didn't want to use open source, but they finally caved in.