Robocopy summary time section explain

[qaz123]

Hi everyone,
We are using Robocopy as a backup tool.
Trying to estimate the amount of time a backup operation will take,
I used the Robocopy's summary time section to help me.

Getting samples from previous backups, I came up with this:

===================================================================
Day 1:

Files : *.*

Exc Dirs : D:\data\Transfer

Options : *.* /TS /S /E /COPYALL /SECFIX /PURGE /MIR /NP /R:1 /W:2

Total Copied Skipped Mismatch FAILED Extras
Dirs : 95086 311 94775 0 0 162
Files : 398801 313211 85590 0 0 592
Bytes : 372.600 g 317.693 g 54.907 g 0 0 67.75 m
Times : 8:50:46 7:33:41 0:00:00 1:17:05


Speed : 12531318 Bytes/sec.
Speed : 717.047 MegaBytes/min.

===================================================================
Day 2:

Files : *.*

Exc Dirs : D:\data\Transfer

Options : *.* /TS /S /E /COPYALL /SECFIX /PURGE /MIR /NP /R:1 /W:2

Total Copied Skipped Mismatch FAILED Extras
Dirs : 95198 309 94889 0 0 197
Files : 399596 86825 312771 0 0 470
Bytes : 372.986 g 55.464 g 317.522 g 0 0 95.34 m
Times : 6:18:54 5:02:35 0:00:00 1:16:18


Speed : 3280242 Bytes/sec.
Speed : 187.696 MegaBytes/min.


===================================================================
Day 3:

Files : *.*

Exc Dirs : D:\data\Transfer

Options : *.* /TS /S /E /COPYALL /SECFIX /PURGE /MIR /NP /R:1 /W:2

Total Copied Skipped Mismatch FAILED Extras
Dirs : 95242 199 95043 0 0 155
Files : 399647 313064 86583 0 0 343
Bytes : 373.751 g 318.463 g 55.287 g 0 0 36.52 m
Times : 7:57:51 6:46:41 0:00:00 1:11:09

Speed : 14013321 Bytes/sec.
Speed : 801.848 MegaBytes/min.

===================================================================


Copied Time
Day 1 : 317.693 g 7:33:41
Day 2: 55.464 g 5:02:35
Day 3: 318.463 g 6:46:41

Question: Can someone explain to me how 317G take 7:33 Hrs and 55G take 5:02 Hrs.

Thank you
Simon

 

[58sniper]

bandwidth, disk speed (read and write), compression, encryption, server load....

Do you have your Tek-Tips.com Swag? I've got mine!

Stop by the new Tek-Tips group at LinkedIn.

 

[qaz123]

Thank you 58sniper, I already look into those possible causes.
Unfortunately, I have the same results on different servers at different time.
What is bugging me is the amount of files transferred / copied.

We have a pattern like: 313211 86825 313064.

I am searching the reason for this.
Any suggestions will be welcome.
Thank you
Simon

 

[strongm]

Robocopy does not just copy the files. It has to check whether it is supposed to copy the files, and part of that check involves examining the files on the destination. In other words whether you copy a file or not Robocopy basically has to touch all the files on the destination to check their timestamp and size. Particularly if you have a lot of small files this can add some unexpected overhead to the elapsed time (which can be exacerbated by any bandwidth issues)

 

[goombawaho]

If you wanted to test that idea by strongm, which I think is a good one, turn off the /MIR and just let it do a straight copy. Then you'll know that Robocopy having to think about WHAT to copy (by using /MIR) is the cause and not JUST the actual copying speed.

 

[strongm]

/MIR is just shorthand for /PURGE /E

Turning it off won't stop Robocopy from querying destination files, I'm afraid. It always checks the timestamp and filesize of any file in the destination that exists in the source. I'd only expect to see a fairly minor speed impact because Robocopy no longer needs to do the work to support /PURGE

 

[goombawaho]

I believe that it WILL prevent robocopy from querying destination files. Without /MIR, it should just be a dumb copy of everything - no thinking, from my understanding. Now we have to just see about that or find some definition.

In this string, Options: *.* /TS /S /E /COPYALL /SECFIX /PURGE /MIR /NP /R:1 /W:2
the /E /Purge is redundant if you're going to use the /MIR

But for testing, I'm saying to chop off the /Purge and /Mir and see what happens.

 

[strongm]

>no thinking, from my understanding

Robocopy never does a dumb copy, I'm afraid. The default action is to only copy a file if the source and destination have different time stamps or different file sizes. And it can only achieve that if it reads the timestamp and file size of the files it finds in the destination (and which it normally does through running a GetDirectory API call against the destination folder which populates an array of FILE_DIRECTORY_INFORMATION structures)

Of course, if the destination is empty then it'll do what appears to be a dumb copy (but then so will /MIR). But we are not looking at that scenario, given the vast number of files being skipped each night.

 

[qaz123]

Thank you everyone for your time on this matter.
- Goal: Estimating the average time for the copy operation
How many files have changed / copied / moved / etc

I did more tests and my conclusions are :
- robocopy is a good program for copying and transfer information.
- More flags should be added to produce a more meaningful log
- Listing only file that are copied
- Listing only file that has changed
- Listing only file that are newer
- etc.

- Only the " Total " column has a real value

- Dot relied on the summary for knowing how what has been modified in the files
- The "Copied" column does not show the real work done

- Using the " /L " options

Options : *.* /TS /NDL /L /S /E /DCOPYA /COPYAT /PURGE /MIR /NP /R:1 /W:2

Total Copied Skipped Mismatch FAILED Extras
Dirs : 190 189 1 0 0 0
Files : 2603 2603 0 0 0 0
Bytes : 154.71 m 154.71 m 0 0 0 0
Times : 0:00:00 0:00:00 0:00:00 0:00:00

- The "Copied" column show the same data as the "Total" column

If you know a way to correct this issue, I will be glad to run some more tests.
Again Thank you
Simon

 

[strongm]

Not sure what you think you've lirned from this

You've

a) 'copied' to an empty destination - so no testing of destination folders will technically tale place (well, it does, but GetDirectory returns nothing)
b) you are deliberately copying no data

so the results of 0 0 0 0 are pretty much expected here

Also, if you are running this locally rather than over the network we won't see much; the overheads would be too low.

And 2603 files is nowhere near enough to see any real effect.