Copy stuff in increments of 10 2

[Stiddy]

I have a little routine where I read in a text file full of workstations or servers to perform some kinda of action on. Maybe copy a file, create a directory, etc. So I started thinking about trying to improve on it a little. For example: If I have five thousand workstations that I have to copy a file to I know I can use the File::Copy module and read in my list of wks's and copy the file to each machine one by one. But what if I could copy say in increments of 10 or 15 wks's at a time. That would increase the speed of this routine. I don't really know where to start. Could someone point me in the right direction. Here is my normal routine.

use File::Copy;
use Net:ing;
my $host = Net:ing->new("icmp");


##### Open logfiles for input and Read in machines to be tested #####
open (COPIED_FILE, ">>Copied_FIle.log");
open (UNREACHABLE, ">>UnReachable.log");
open (WKS, "<d:All_Workstations.log") || die ("Cannot open All_Workstations.log: $!");
while(<WKS>) {
chomp $_;
$_ =~ tr/a-z/A-Z/;
if ($host->ping($_, 5)){&COPY_DATA}
else {
print "$_ unreachable\n";
print UNREACHABLE ("$_\n")
}
}
close COPIED_FILE;
close UNREACHABLE;
close WKS;
$host->close();

sub COPY_DATA {
copy("blah.txt","\\\\$_\\c\$\\DIRECTORY");
}
###################################################
ALL_Workstations.log

workstation1
workstation2
workstation3
workstation4
workstation5
workstation6
workstation7
workstation8
.....

HTH - Stiddy

 

[TrojanWarBlade]

You're right.
Sorry I was rushing. Been very busy.
Try this:

#!/usr/bin/perl -w
use strict;

my $batchsize = 4;
my $count     = $batchsize;
my @items     = ();
while(<DATA>) {
  chomp;
  if($count--) {
    push @items, $_;
    next;
  }
  process_items(\@items);
  @items = ($_);
  $count = $batchsize-1;
}
process_items(\@items) if(@items);

sub process_items {
  my $arrayref = shift;
  print "<", join("><", @$arrayref), ">\n";
}

__DATA__
red
yellow
green
purple
fruit
salad
down
up
left
right
tall
short
fat
skinny
back
forward

Trojan.

 

[Stiddy]

Trojan and Mike,

That works great. Now, one last question and then I will open up the PERL book and learn what everything in the code does line for line. How will I refer to the data coming into the sub process_items? If I just do

print "$arrayref\n";

I get:

ARRAY(0x1831c08)
ARRAY(0x1831c08)
ARRAY(0x1831c08)
ARRAY(0x1831c08)

HTH - Stiddy

 

[TrojanWarBlade]

If you remove the backslashes on the parameters then you can just use the array @_ directly in the function.


Trojan.

 

[Stiddy]

Trojan,

Thanks for your time and effort. I learned from this little experience. I am still however a little confused how to remove the backslashes. Thanks again.

HTH - Stiddy

 

[KevinADC]

since the last question seems to have gone unanswered:

I am still however a little confused how to remove the backslashes

here it is without the backslashes:

my $batchsize = 4;
my $count     = $batchsize;
my @items     = ();
while(<DATA>) {
  chomp;
  if($count--) {
    push @items, $_;
    next;
  }
  process_items(@items);
  @items = ($_);
  $count = $batchsize-1;
}
process_items(@items) if(@items);

sub process_items {
  my @array = @_;
  print '<', join('><', @array), ">\n";
}

__DATA__
red
yellow
green
purple
fruit
salad
down
up
left
right
tall
short
fat
skinny
back
forward

 

[TrojanWarBlade]

Sorry guys, I should have answered that post.
Thanks for the reminder Kevin and removing the backslashes for me.
I would just like to point out though that in "process_items", there is no need to copy the data from "@_" to "@array".
"process_items" could be as simple as:

sub process_items {
  print '<', join('><', @_), ">\n";
}

Trojan.

 

[Stiddy]

Well removing the backslashes worked but I dont neccessarily want the < and > around the data which is ok i just remove that peice of the code. I am still however stumped as to how to do what I was orginally trying to do which is copy a file to multiple servers at the same time. When those 10 or so servers have gotten the file copied to them then do the next 10 in the list. If I do something like

foreach $_ (@array) {
copy (mike.txt,\\\\@array\\c\$\\temp);
}

I am still only copying that file one at a time to those 10 servers passed to the subroutine. Any ideas?

Thanks again in advance...

HTH - Stiddy

 

[TrojanWarBlade]

What on earth are all those backslashes for!
:-O

I guess there are a few ways of doing this.
You could use threads, running 10 parallel threads and then rejoining them before returning for the next batch of ten.
You could "fork" child processes to do the copying but then you'd need to watch the children to see when they finished.
I guess I'd go with the threads since that's the eaisest way to track all 10 copies at once.
Anyone want to have a go at the code whilst I grab a bite to eat?


Trojan.

 

[Stiddy]

Trojan,

I dont know of any other way to connect to a remote computer without the slashes. As far as I am aware you can use double \\ slashes in a windows environment or a / slash also. I guess the forward / is nicer looking though. I am just so used to using backaslashes. You made me laugh. haha....thanks

HTH - Stiddy

 

[Stiddy]

Where can I learn more about threads. Lookin in the 3 books I have I see nothing about it. I do however see some docs on fork. Can you point me in the right direction or help? Thanks...

HTH - Stiddy

 

[TrojanWarBlade]

Yep, try here


Trojan.

 

[Stiddy]

Thanks, it doesnt look to bad. I will give it a try. Will follow up tommorrow.

HTH - Stiddy

 

[Stiddy]

use strict;
use threads;
open (DATA, "text.log") || die ("Cannot open All_Workstations.log: $!");
my $batchsize = 3;
my $count = $batchsize;
my @items = ();
while(<DATA>) {
chomp;
if($count--) {
push @items, $_;
next;
}
process_items(@items);
@items = ($_);
$count = $batchsize-1;
}
process_items(@items) if(@items);

sub process_items {
my @array = @_;
my $thr = threads->new(\&threads);
}


sub threads {
print "$_";
`sleep 3`;
}


FAILS with:
purpleA thread exited while 7 threads were running, <DATA> line 16.

Any ideas?

HTH - Stiddy

 

[TrojanWarBlade]

You didn't listen to my suggestion.
Your code kicks off all processing in threads in one hit and then exits immediately.
Of course you're quitting with threads still running.
I suggested starting all but the last one of the batch (of say 10) in threads and let the last one execute normally.
That would give you some chance for the other (9) threads to complete whilst waiting for the last one.
You could, instead (and probably preferrably) use a join at the end of "process_items" to ensure that the threads have all completed before returning.


Trojan.

 

[Stiddy]

Then I would assume you mean to

use strict;
use threads;
open (DATA, "text.log") || die ("Cannot open All_Workstations.log: $!");

my $batchsize = 3;
my $count = $batchsize;
my @items = ();
while(<DATA>) {
chomp;
if($count--) {
push @items, $_;
next;
}
process_items(@items);
@items = ($_);
$count = $batchsize-1;
}
process_items(@items) if(@items);

sub process_items {
my @array = @_;
my $thr = threads->new(\&threads);
$thr->join;
#print " ", join(" ", @array), "\n";
}


sub threads {
print "$_";
`sleep 3`;
}

I get:

purpledownrightfatforwardUse of uninitialized value in string at D:\scripts\Test\Stuff.pl line 33, <DATA> line 16.

HTH - Stiddy

 

[TrojanWarBlade]

No, in "process_items", create an array of thread objects and then "join" them all back after the loop.
Sorry, I don't have time to comment further at the mo, I might get a little time later.
Good luck!


Trojan.