Best way to copy many small files (millions)?

[Yrrk]

I'm looking for suggestions/tools to copy 20M ~60KB files from one system to another. In particular if you have any real world experience doing something similar i'd love to hear about it.

Ideally solution would be:

1. restartable in case of failure
2. fast.. 1Gbps speed if I/O subsystem can support it
3. preserves symlinks and metadata

 

[Annihilannic]

I would use cpio over ssh:

[tt]find . | cpio -o | ssh desthost 'cd /destdir && cpio -idm'[/tt]

You could use rsh instead for higher speed (no encryption required), however I would only set that up temporarily for security reasons.

It's not restartable, you could consider rsync if you needed to resume a transfer, although it can be very slow to check that number of files for the required updates, probably just as quick to start again. cpio -i without the -u option would not bother trying to write the existing files again, so might save a little time in a 'resume' scenario.

Annihilannic.

 

[stevexff]

If it's a one-off, zip them all into one file and FTP it over, then unzip it at the other end.

Steve

[small]"Every program can be reduced by one instruction, and every program has at least one bug. Therefore, any program can be reduced to one instruction which doesn't work." (Object:erlDesignPatterns)[/small]

 

[msimko]

You might look at rsync. Otherwise, tar and ftp, with or without gzip or bzip.

 

[elgrandeperro]

I concur with rsync. First, if the files don't change much it does the minimal amount. You might have to use the --delete option to remove files that have recently been removed on the source end for multiple runs.

If the files paths are moved around or the files rewritten lots, I would not use rsync.