memory question (apparent leak)

[efoss]

I'm working with really big files - ~10 giga each. I'm programming on a Mac. Mac has an "Activity Monitor" that lets you look at, among other things, how much memory you're using. The memory is divided into 4 categories - free, wired, active and inactive - and the total amount adds up to the amount of RAM that you have (i.e. it's not counting virtual memory). When I run a program to parse these huge files, my "free" memory starts getting eaten up and turning into "inactive" memory until there is essentially no "free" memory left. The total RAM is 4 giga. However, this parsing program isn't doing anything that requires a lot of memory. It's going through the file and remembering 4 different things, which it sticks into scalar variables, and then periodically printing out all four scalar variables. I'm not making any mistake like pushing stuff onto an array and never clearing it off. But still, the farther I get into the file, the less "free" memory I have. Does this make sense to anyone?

 

[KevinADC]

You may think the program is doing nothing to eat up all that memory, but evidently it is. Without seeing code, no way to know.

------------------------------------------
- Kevin, perl coder unexceptional!

 

[efoss]

Here's the complete code:

#!/usr/bin/perl -w
use strict;

#put this into a directory in which all of the files that end in .txt are .RAW to .txt files that you want converted to the perl processed 4 column format

my $line;
my @array;
my $scan;
my $acquisition_mode;
my $MS_mode;
my $mz;
my $intensity;
my $time;
my $path;
my $file;
my @files;
my $output_file;


opendir (DIR, "\.");
@files = readdir DIR;
closedir DIR;


foreach $file (@files) {
if ($file =~ /\.txt$/ and $file !~ /^perl_processed/) {
open (INPUT, "<$file") or die "can't find $file\n";
$output_file = $file;
$output_file = "perl_processed_profile_$output_file";
open (OUTPUT1, ">$output_file");

while ($line = <INPUT>) {
chomp $line;
$line =~ s/\r$//;
$line =~ s/^\s+//;
$line =~ s/\s+$//;
if ($line =~ /\S/) {
if ($line =~ /^ScanHeader/) {
$scan = "";
$time = "";
$mz = "";
$intensity = "";
$MS_mode = "";
$acquisition_mode = "";
if ($line =~ /^ScanHeader # (\d+)$/) {
$scan = $1;
}
else {
die "unexpected scan header line: $line\n";
}
}
if ($line =~ /^start_time/) {
if ($line =~ /^start_time = (\d+\.\d+), end_time = 0\.000000, packet_type = 0$/) {
$time = $1;
}
else {
die "didn't expect this time line:\n$line\n";
}
}
if ($line =~ /^Polarity positive/) {
if ($line eq "Polarity positive, Profile Data, Full Scan Type, MS Scan") {
$MS_mode = 1;
$acquisition_mode = "profile";
}
elsif ($line eq "Polarity positive, Cenrtoid Data, Full Scan Type, MS Scan") {
$MS_mode = 1;
$acquisition_mode = "centroid";
}
elsif ($line =~ /^Polarity positive, \w+ Data, Full Scan Type, MS2 Scan$/) {
$MS_mode = 2;
$acquisition_mode = "MS2";
}
else {
die "this line was unexpected:\n$line\n";
}
}
if ($line =~ /^Packet \#/) {
if ($line =~ /^Packet \# (\d+), intensity = (\d+\.\d+), mass\/position = (\d+\.\d+)$/) {
if ($MS_mode == 1) {
$intensity = $2;
$mz = $3;
if ($acquisition_mode eq "profile") {
die "this one was supposed to have been collected in centroid mode\n";
}
elsif ($acquisition_mode eq "centroid") {
print OUTPUT1 "$scan\t$time\t$mz\t$intensity\n";
}
elsif ($acquisition_mode ne "MS2") {
die "unexpected acquisition mode:\n$acquisition_mode\n";
}
}
}
else {
die "an unexpected line here:\n$line\n";
}
}
}
}

close OUTPUT1;
close INPUT;
}
}

print "done!\n";

 

[wardy66]

I have a feeling I read somewhere about "free" vs "inactive".

I think inactive is effectively free. See if you can google something along those lines - it might not be the script's problem.

 

[MikeLacey]

From

http://docs.info.apple.com/article.html?artnum=107918

Inactive memory is no longer being used and has been cached to disk, it will remain in RAM until another application needs the space. Leaving this information in RAM is to your advantage if you (or a client of your computer) come back to it later.

I read that as meaning - to you - don't worry about it; the system will re-use inactive memory as it needs to.

Mike

http://www.myspace.com/micahhowzat

 

[efoss]

Thanks. That's reasonable. I still find it a bit disturbing, though, that more and more memory would have been used for something, when it's not clear why that would be. Let's say you have 4 gigs of RAM and an enormous file consisting of a bizillion lines, each of which says "hello world". You run this program on the file:

open (INPUT, "<file.txt");

while (my $line = <INPUT>) {
$line = $line;
}

close INPUT

You start out and see that essentially all of your RAM is "free". Then you run this program and see all your memory go from "free" to "inactive". On the one hand, you might think "who cares, inactive is just as good as free". On the other hand, you might think "wow, if I'm filling up 4 gigs of RAM in storing the words "hello world" in a single variable, what other screwy things are going on here?"

 

[KevinADC]

True, but it appears to be an operating system issue, not a perl issue?

------------------------------------------
- Kevin, perl coder unexceptional!

 

[efoss]

I really don't know what it is - it could very well be an operating system issue, though Perl normally doesn't elicit this weird response from this computer. What I was kind of hoping for was an answer like "yes, this happens once a file gets big enough" and then some explanation about some obscure feature of Perl that you almost never run into except in situations like this, or something along those lines. But maybe it is an operating system issue - I don't know. I'll keep an eye on this to see if I can make sense of it as I continue to work with big files.

 

[KevinADC]

Maybe ask on perlmonks.com if you haven't already.

------------------------------------------
- Kevin, perl coder unexceptional!