What would cause a script to exit prematurely?

[sloppyhack]

I have a script that analyzes contents of a file and compares it's contents to another file. This script has been working fine...but when we throw a large amount of data at it....it stops prematurely. The file has 6.6 million rows but there seems to be available memory on the machine when it stops.

Anyone know what could cause a Perl script to close prematurely? Memory leaks? We have processed up to 2 million records before without a problem...6.6 million is causing it to stop before it finishes.

Cheers,

Sloppyhack

 

[TrojanWarBlade]

Without seenig any code it could be a little difficult to track.
It's very easy to write code that uses more memory than it should per record so that's a possibility but there could be other issues like unusual record formats.
Can you post anything for us to examine?


Trojan.

 

[sloppyhack]

Thanks. Here's one spot where premature exit is happening. The script reads 6.6 million lines from one file then comes over to read this file which it will compare to the 6.6 million lines it has already processed. Gets to line 300 of the second file and just stops. No error is thrown by Perl or by the error handling I put in the script. Just stops. Unusual record formats are highly likely too as this data is coming from someone else's system. We are running this on a huge box too so the system never says it has run out of memory. Seems to have 1.5 gigs of memory when it stops. The memory does seem to take a big drop just before it exits. Here is the read snippet where I beleive the exit is occuring.

This snippet is reading the file and comparing the keys from the items it's reading to the keys from the items from a previous file it read in. Never had a script just die on my like this?

while (<IM>) {
chomp;
print "$.\n";
my @fields = split /\t/; #splits the line into values
if ($. == 1) { #find header row
@prodkeys = @fields; #assign header info to the hash keys
next;
}
unless (/[\t]*$/) { #skip any blank lines
++$imtotal;
my %prodhash = (); #this stuff defines the array of hashes
$dtlkey = 0;
@prodhash{@prodkeys} = @fields;

$prodhash{'VENCODE'} =~ s/^[ ]*|[ ]*$//g; #clean any leading or trailing spaces from the keys
$prodhash{'VENCATNUM'} =~ s/^[ ]*|[ ]*$//g;
$prodhash{'FACID'} =~ s/^[ ]*|[ ]*$//g;

$itemkey = $prodhash{'VENCODE'} . $prodhash{'VENCATNUM'};

if ($prodhash{'UOM'}) {

$prodhash{'UOM'} =~ s/^[ ]*|[ ]*$//g;

$dtlkey = $prodhash{'VENCODE'} . $prodhash{'VENCATNUM'} . $prodhash{'UOM'} . $prodhash{'FACID'};

}

if ($unqitems{$itemkey}) {

$unqitems{$itemkey}{'INIM'} = "Y"; #flag unique PO history items that are in the IM

} else {

push @imnotpo, \%prodhash; #store all im items that are not found in the PO history
}


if ($dtlitems{$dtlkey}) { #check in IM for the item for detail report

$dtlitems{$dtlkey}{'INIM'} = 'Y'; #flag unique PO history items that are in the IM

} else {

push @imnotpodtl, \%prodhash; #store all im items that are not found in the PO history

}

}
}




Cheers,

Sloppyhack

 

[TrojanWarBlade]

Are you running with "use warnings" and "use diagnostics"?
If no, do so and see if you get anything interesting.

Trojan.

 

[TrojanWarBlade]

Your skip blank lines "unless" test looks pointless.
Your regex whitespace class ought really to be \s+.
You should generally not use "*" where you want to match something cos it can match nothing.
Personally I don't like the 2 parameter form of "split". I usually suggest that you always add a third param of -1 to ensure that you don't miss empty fields.

Sorry, this isn't meant to be nit picking, just checking the code out bit by bit.

Other than that, I can't really see anything obvious.


Trojan.

 

[sloppyhack]

Yeah. That's all old stuff that I'm sure any real programmer could improve. I did run it with "use warnings"...and got nothing unusual. The usual undeclared variable warning. I'll try turning on diagnostics to see what happens.


Cheers,

Sloppyhack

 

[TrojanWarBlade]

If you're running unix or linux you could try "echo $?" after it stops to see if there is any error code.


Trojan.