Directory remediation 1

[goliath]

Greets!

I would like to write a perl script that spawns x(variable) simultaneous tasks to work through a directory structure…

In this particular case, I want to use Robocopy.

I’ve found that a single Robocopy of a large directory structure is very slow compared to 4 simultaneous Robocopies breaking up the structure.

However I’d like to automate this process and let Perl kick off the robocopies keeping 4 going at a time as it works through the directory tree.

Any thoughts or idea’s would be appreciated, I’m still a young Perl programmer =)

Thanks!

 

[Grant]

Hi goliath,

May I ask what Robocopy is?

Grant.

 

[goliath]

Robocopy is a Win32 application not unlike Xcopy - just more functionality =)

 

[goliath]

I'm going to try to help explain my situation since I get the feeling I did a poor job in my first attempt.

The big picture goal is to copy an entire directory structure from one drive to another using RoboCopy (a Win32 tool very similar to xcopy or copy).

I have found that doing a straight robocopy of large directory structures takes a very long time. I can reduce this time significantly by breaking up the structure to be copied into 3-6 different copy statments. Since I have to do this for several directory structures on several servers I'm trying to write a Perl script to help automate this.

My goal is to have Pearl read in the directory tree then break it up into 4 parts and queue up the robocopies. Could I have it be perfect it would use size but I'm thinking that's a bit much to ask for at this time =)

Thanks in advance for any idea's. I'm working on a draft script today and will post what I come up with.

 

[Grant]

Hi goliath,

I'm guessing that you need to tell people more about RoboCopy itself. And how you use it. You say it is a Win32 tool. What do you mean by that? Is it a perl module? Are you expecting that others are familiar with it?

Grant.

 

[justice41]

Here a thread (thread219-349156) with an example of queueing up a set number of processes. You can change the TaskN() sub to do what you want or replace it with whatever you are using to call robocopy (system call, etc.).

jaa

 

[goliath]

Robocopy and Xcopy are just file system copy commands for Windows (NT, XP, 2000).

example: Robocopy C: D:

Copies everything from the C drive to the D drive

So in my case I want to read in the directory structure from a given point such as C:\Apps and then have it start copying to the target such as D:\Apps

So, if C:\Apps has folders A, B, C, D, E, F, G, H, I this script will queue up the first four (A, B, C, D) and do a Robocopy C:\Apps\A (to) D:\Apps\A for the first four folders and as it finishes up a folder it will then process the next one.

For now I'm assuming only one level of depth for the folders as far as break down. So if there are folders deeper in the directory structure they do not get broken up, just the first level from the source as its defined and subfolders are included in the robocopy.

My apologies for my shortcomings on explaining this - it's a bit trickier than I thought it would be.

Basically I'm looking to make a parallel tasking file copy Perl script based on the first level of folders =)

 

[goliath]

Justice41,
This works on Windows?

I've been fighting with this for about a week and I'm making no progress...

I'll give it a fresh start in the morning.

 

[goliath]

Okies,

Here's my draft trying to pick stuff from your link - but I'm still quite confused about how to get that working...

My program:

# Qeueing Robocopy
# Operating system: Windows XP
# Enter a directory (eg. D:\Users)
# Read in directories beneath and queue them up (eg. D:\Users\a, D:\Users\b, D:\Users\c)
# Process queue
system(cls)
use POSIX ':sys_wait_h';

my @queued = ();
my $maxqueue = 2; # number of concurrent procs allowed in queue

# Get tree for processing
print "> Enter source root directory to start Robocopies from: ";
$tree = uc(<STDIN>); # input search directory; must put a / at the end of the directory
# Get destination for copy
print &quot;> Enter destination directory: &quot;;
$to = uc(<STDIN>); # input destination

chomp $tree;
opendir (TREE, $tree); # open the directory entered
@alf=readdir TREE; # read the directory into a list
# print &quot;DirTree: @alf\n\n&quot;; # display contents of directory
undef @dir;
foreach $item (@alf)
{
if (-d &quot;$tree$item&quot; &&!(-l &quot;$tree$item&quot)
{
push (@dir,$item.&quot;,&quot;
}
}

print &quot;Directory start: $tree\n&quot;;
print &quot;Directories: @dir\n&quot;; #print the directorys only

my @queued = @dir;

while (@queued) {
my $lasthost = pop(@host) if @host;
if ( defined($lasthost) and my $pid = fork ) { # parent process

push @queued, $pid;

} elsif (defined $pid) { # child process

print &quot; TaskN on: $lasthost \n&quot;;
TaskN ($lasthost);
exit;

}
next if @host and @queued < $maxqueue;
while (1) {

my $nextpid = shift @queued;
last if waitpid($nextpid, WNOHANG);
push @queued, $nextpid;
sleep 1;

}
}


sub TaskN {
print &quot;* Robocopying $from to $to \n&quot;;
# system (&quot;start RoboCopy $from $to /E /NP&quot;
}

 

[justice41]

In your case you would assign @dir to @host (instead of @queued) and then enter the while loop as it was in the original code, with both @queued and @host as the test. The loop takes care of the rest (queuing up the directories into @queued, etc).

my @host = @dir;

       while (@host or @queued) {
       ...

The while loop executes while there are still 'hosts' (directories in your case) to be queued up (still values in the @hosts array), or there are still hosts left in the queue (values in the @queued array). So it terminates when all hosts have been queued up, processed, and then removed from the queue.

jaa

 

[justice41]

Also, you are passing the current 'from' directory to TaskN (contained in the $lasthost variable) in each child process. So your TaskN would look something like

sub TaskN {
    my $from = shift; # Get directory passed as argument
    print "* Robocopying $from to $to \n";
    # system ("start RoboCopy $from $to /E /NP");
}

Since I'm not sure the details of Robocopy, I don't know that exact form that $from and $to should be in. You might have to tweak them.

jaa

 

[goliath]

Whoot!

Thanks a million for all your help! It's working, and here's the code! I've got a few minor issues to figure out with quotes because of long file names, but otherwise it's great!


# Program: Q-Robo.pl
# Purpose: Qeue and run multiple Robocopies at a time
# Operating system: Windows XP
# Enter a directory (eg. D:\Users\)
# Read in directories beneath and queue them up (eg. D:\Users\a, D:\Users\b, D:\Users\c)
# Process queue
system(cls); # clear the screen
use POSIX ':sys_wait_h'; # load posix module

my @queued = (); # initialize @queued to null
my $maxqueue = 3; # number of concurrent processes allowed in queue

print &quot;> Enter soucre and destination, include final backslash\n&quot;;
# Get source tree for processing
print &quot;> Enter source root directory to start Robocopies from: &quot;;
$tree = uc(<STDIN>); # input search directory; must put a / at the end of the directory

# Get destination for copy
print &quot;> Enter destination directory: &quot;;
$to = uc(<STDIN>); # input destination
chomp($to);

chomp $tree;
opendir (TREE, $tree) || die &quot;ERROR! Unable to open $tree :$!&quot;; # open the directory entered
@alf=readdir TREE; # read the directory into a list
# print &quot;DirTree: @alf\n\n&quot;; # display contents of directory
undef @dir;
foreach $item (@alf)
{
if (-d &quot;$tree$item&quot; &&!(-l &quot;$tree$item&quot)
{
if ($item eq &quot;..&quot { next} # Keep directory markers out of array
if ($item eq &quot;.&quot { next}
push (@dir,$item);# add directories to @dir array with comma seperation
}
}
print &quot;Directory start: $tree\n&quot;;
print &quot;Directories: @dir\n\n&quot;; #print the directories only

my @todo = @dir;

while (@todo or @queued) {
# print &quot;ToDo: @todo\n&quot;;
my $lasttodo = pop(@todo) if @todo;
if ( defined($lasttodo) and my $pid = fork ) { # parent process

push @queued, $pid;

} elsif (defined $pid) { # child process

print LOG &quot;TaskN on $lasttodo \n&quot;;
TaskN ($lasttodo);
exit;
}
next if @todo and @queued < $maxqueue;
while (1) {

my $nextpid = shift @queued;
last if waitpid($nextpid, WNOHANG);
push @queued, $nextpid;
sleep 1;

}
}


sub TaskN {
my $from = shift; # Get directory passed as argument
print &quot;* Copying $tree$from to $to$from \n&quot;;
$syscom = &quot;Robocopy $tree$from $to$from /E /NP /Z /V /R:5 /w:3 /LOG+:$from.log&quot;;
print &quot;$syscom\n&quot;; # Generate command line to pass to system
system ($syscom); # execute command
}