Copying a large file tree using multiple connections?

[forrie]

We have a remote mount point that we are copying files to. Presently, with a single instance of rsync. The file tree has 100s of files -- what I'm trying to figure out is how to split that entire job up and use, for example, three connections to maximize our bandwidth and perhaps get the files copied over more quickly.

I'm not sure this could be done with rsync by itself, without a lot of tinkering.

Anyone have ideas about how to do this?


Thanks.

 

[fredericofonseca]

a search for "parallel rsync" shows up a few things such as

http://blog.ciberterminal.net/2012/10/16/parallel-rsyncing-a-huge-directory-tree/

look at the above and at others that are found

Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

FAQ219-2884
FAQ181-2886

 

[SamBones]

You say that you are writing it to a "remote mount point". If that's a filesystem that's mounted via NFS, you need to keep in mind that you're hitting the network whenever you read or write anything to or from it. This adds a lot of latency for all disk operations and slows everything down. If you can do the updating of the directory tree on the system that actually "owns" the filesystem (exports it), not the one mounting it, the updates can go a lot faster.

Not sure if I explained it well, or if I'm interpreting your configuration correctly. A little more information on your configuration might help.