learingperl01
MIS
Hello Perl experts,
I am pretty new to Perl and having a memory problem with my code. I have narrowed it down to the following section of my code which always seems to generated the out of memory error when running.
Now a little background. I want to process all files in a directory which is held by $processedDirPath variable and only work with files that end with "rtsd001".
What I am wondering what is the MOST memory efficient/best way to read files in a directory then modify each file found (Meaning I have to rewrite the file to remove all data before the STRUC20 line. )
The directory which I am running readdir against contains about 6-10 thousand files each about <1k - 40k> in size.
So, working with this large number of files what would be the best way to do this?
Thanks for the help in advance.
I am pretty new to Perl and having a memory problem with my code. I have narrowed it down to the following section of my code which always seems to generated the out of memory error when running.
Code:
opendir (TXT, "$processedDirPath") or die "Cannot open $processedDirPath directory $!\n";
while ( (my $matchingFiles = readdir(TXT)) ) {
if ( $matchingFiles =~ /\.rtsd001$/ ) {
my @fileContents;
my $UNMODIFIED;
my $MODIFIED;
open(UNMODIFIED, "<$processedDirPath/$matchingFiles") or die "Cannot open $UNMODIFIED for reading $!\n";
open(MODIFIED, ">$processedDirPath/$matchingFiles.old") or die "Cannot open $MODIFIED for writing $!\n";
while (<UNMODIFIED>) {
/STRUC20/ and @fileContents=(), next or push @fileContents, $_;
}
print MODIFIED @fileContents;
close(UNMODIFIED);
close(MODIFIED);
system( "/bin/mv", "$processedDirPath/$matchingFiles.old", "$processedDirPath/$matchingFiles" ) == 0 or warn "Move command filed $!\n";;
}
}
closedir TXT;
Now a little background. I want to process all files in a directory which is held by $processedDirPath variable and only work with files that end with "rtsd001".
What I am wondering what is the MOST memory efficient/best way to read files in a directory then modify each file found (Meaning I have to rewrite the file to remove all data before the STRUC20 line. )
The directory which I am running readdir against contains about 6-10 thousand files each about <1k - 40k> in size.
So, working with this large number of files what would be the best way to do this?
Thanks for the help in advance.