Advice on reading/copying very, very large files

[grittyminder]

I have a script that is atempting to read all the data from a very, very large file on a remote server. For whatever reason, the script fails, randomly, typically with either of the following two errors: "The semaphore timeout period has expired" or "The specified network name is no longer available." When the script does complete successfully it takes anywhere from 5 minutes to to two hours (depending on how long the file is). I'd attributed the script failure to 1) tumultuous network traffic and 2) the old-ness of the box that the script is running. But now I'm wondering if I'm doing something wrong? Actually, I'm not really sure what I'm asking... I need help but I'm not really sure what. I thought that reading directly from the remote file was probably bad, so I tried copying the file locally THEN reading it. But the problem then is that the copy function takes FOREVER to complete when the file is very, very large, (and, actually, now that I think of it usually doesn't copy over at all; it just fails). Oh yeah, the vbscript copyFile function didn't even work for me (the function call just hangs) so I was calling xcopy from inside the script instead. The script in its current form attempts to xcopy a file locally 3 times then quits if unsuccessful. I would say that 90% of the time the script is unsuccessful and quits. There has to be a better way to do this. Maybe there is a better copy function? Maybe I'm approaching the problem wrong? Any ideas?

 

[PHV]

A more robust copy utility is robocopy

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[grittyminder]

Ah, but I tried robocopy. Maybe I don't understand how it works or I'm using it incorrectly, but after I invoke robocopy the vbscript immediately continues execution of the remainder of the script (the remainder of the script following the call to robocopy). (note: vbscript waits for the copy function to complete before continuing execution w/ xcopy). That wan't a big deal for me though--I just put in a check after the robocopy call to make sure that execution didn't continue until the file was copied over completely (i.e. while file isn't in the destination folder loop). However, all the times I tried robocopy it never actually successfully copied the file over (then again, I would get impatient and halt execution of the script after waiting a certain period of time). So you're claiming, hands down, that robo copy is a superior copy function to xcopy? Why is that?

 

[PHV]

ROBOCOPY :: Robust File Copy for Windows :: Version XP010

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[FISKO]

Hi I use Syncback from

http://www.2brightsparks.com/

I don't know if it will help in your situation, but I use it to transfer large files between servers.

 

[grittyminder]

Cool. Thanks for the syncback tip (I'm downloading the trial right now).

 

[tsuji]

You cannot blame copyfile method when you script a wrong copyfile line.

 

[Jerz]

I use copyfile daily to copy about 50 gigs of files from about 100 servers to a central server, some files larger than a gig each. It works fine, and is typically 2 to 3 times faster than any command line utility (copy, xcopy, robocopy) I've tried. What does your code look like? You have posted none. I have a diferent script that forms a robocopy command and executes it. Robocopy has useful switches and also works well if speed is not your main concern.