Read multiple files, sum up the data and write out to new files

[mgarcia22]

Hello all,
I am writing a program to read in multiple CSV files that have been exported from Ethereal, sum up the bytes for each second, calculate bits per second, and then average each of the values (bytes and bits) and write all of this data out to a file.

Below is the code I have so far - it works fine with one file, but when it get to subsequent files, I find that the byte and bit totals from the previous file have been added to the second output file. Right now, each input file will generate its own output file; however, the ultimate goal would be to have each input file write out to its own tab in an XLS file.

First things first...making sure the numbers are correct. Here is the aforementioned code:

<code>
#!/bin/perl

$filedir = $ARGV[0];

opendir (DP, $filedir) || die "Cannot open $filedir.\n";
while ($file = readdir (DP) )
{
if($file !~ /\.csv$/)
{
print "Skipping $file\n";
next;
}

open(IN,"$filedir/$file");
$file =~ s/\.csv$//;
$ofile = "$filedir$file".".bps";
print "$ofile\n";
while ($line = <IN>)
{
# ($secs,$value) = split(m#,#,$line);
($j1,$secs,$j2,$j3,$j4,$value) = split(m#,#,$line);
$secs =~ s#"##g;
$value =~ s#"##g;
#print "[$secs] [$value] [$line]\n";
($whole,$part) = split(m#\.#,$secs);
$tot{$whole} = $tot{$whole} + $value;
$cnt{$whole}++;
}
close(IN);



$max = 0;
open(OUT,">$ofile");
print OUT "Time\tBytes per Second\tBits per Second\n";
$count = 0;
$bytetotal = 0;
$bittotal = 0;
$bits = 0;
$bytetotal = 0;
$key = 0;
foreach $key (sort keys %tot)
{
$bytetotal += $tot{$key};
$avg = ($tot{$key} / $cnt{$key});
$bits = ($tot{$key} * 8);
$bittotal += $bits;

print OUT "$key\t$tot{$key}\t$bits\n";

if ($tot{$key} > $max)
{
$max = $tot{$key};
$maxbit = $bits;
$maxtime = $key;
}
$count++;

}
print "Count is $count\n";
print "Bit total is $bittotal\n";
$byteavg = ($bytetotal / $count);
$bitavg = ($bittotal / $count);
#print OUT "\nnum: $max\ttime: $maxtime\n";
print OUT "\nAverage\n\t$byteavg\t$bitavg\nMax\n$maxtime\t$max\t$maxbit\n";
close(OUT);
}

</code>

Any help is greatly appreciated

 

[travs69]

You should use strict!
You might want to convert to using

http://search.cpan.org/search?module=Text::CSV

it will save you a lot of time in the long run.

Use the code tags.. it makes things a lot easier to read.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Travis - Those who say it cannot be done are usually interrupted by someone else doing it; Give the wrong symptoms, get the wrong solutions;

 

[travs69]

You definitely need to put
my $tot;
my $cnt;
right below where you open the file.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Travis - Those who say it cannot be done are usually interrupted by someone else doing it; Give the wrong symptoms, get the wrong solutions;

 

[blackhawkdoc64]

I agree with Travs, you need to reset your variables that hold those values prior to each iteration of your loop.

 

[mgarcia22]

Thanks for the info. I will certainly look at the module you suggest.

I have found a short-term work-around:

Added
$tot{$key};
$cnt{$key};
just before count++ in the code below

foreach $key (sort keys %tot)
     {
    $bytetotal += $tot{$key};
    $avg = ($tot{$key} / $cnt{$key});
     $bits = ($tot{$key} * 8);
     $bittotal += $bits; 

    print OUT "$key\t$tot{$key}\t$bits\n";
    
    if ($tot{$key} > $max)
       {
       $max = $tot{$key};
           $maxbit = $bits;
           $maxtime = $key;
       }
# Added the two new lines here
    $tot{$key};
    $cnt{$key}; 
    $count++;

 

[stevexff]

Travis is right. There's a lot more to parsing CSV files than just

my @stuff = split /,/;

. Text::CSV_XS is just one example where you can save time and aggro by standing on the shoulders of giants. CPAN is your friend...

Steve

[small]"Every program can be reduced by one instruction, and every program has at least one bug. Therefore, any program can be reduced to one instruction which doesn't work." (Object:erlDesignPatterns)[/small]