Out of memory error

[GoalieB]

I use this script to process file of 275,000 type '4' records.
Sometimes it fails with an error:
"Out of memory error"

If I comment out some of the total lines and rerun it will then finish.
I had a print statement in the Type 2 code section. There are about 2700 of these records.

Any suggestions? Is it simply file size or could the output file be running out of disk space?

Thanks,

BEGIN{ print "Counting claims ...\n"}
/^0/ { print "======> Header: ",$0; }
/^2/ { ++pharm_cnt;}  -- Type 2 record
/^4/ { ++claim_total; -- Type 4 record 
      # unpack() converts from fixed width # to signed #
      ing_cost_amt   += unpack(substr($0,122,9));     
      disp_fee_amt   += unpack(substr($0,131,9));    
      copay_amt      += unpack(substr($0,140,9));
      sales_tax_amt  += unpack(substr($0,149,9));      
      claim_bill_amt += unpack(substr($0,158,9));
      ing_cost_clm_amt  += unpack(substr($0,495,9)); 
      retail_price_amt  += unpack(substr($0,504,9));  
      admin_fee_amt  += unpack(substr($0,257,9));  
      pat_paid_amt   += unpack(substr($0,569,9));        
     }
/^6/ { ++d; batch_billed_amt += unpack(substr($0,22,11));}
/^8/ { ++e; print "======> Tape Control: ",$0 }
END {
#     print all the totals;
}

 

[PHV]

Where is the unpack function defined ?

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[futurelet]

[tt]
/^2/ { ++pharm_cnt;} -- Type 2 record
/^4/ { ++claim_total; -- Type 4 record[/tt]
[tt]
/^2/ { ++pharm_cnt } # Type 2 record
/^4/ { ++claim_total # Type 4 record[/tt]

 

[GoalieB]

Here is the unpack() function and a stranslate() function it calls.

##
## unpack function - takes a number in packed format and unpacks it. 
##                 - returns a signed number 
function unpack( num_packed )
{
    num_length = length( num_packed );
    num_strip = substr( num_packed , 1 , num_length-1 );
    lastchar_packed = substr( num_packed , num_length , 1)
#
#   Convert packed last character to a sign or zero.
#
    sign_char = stranslate("123456789{ABCDEFGHI}JKLMNOPQR","000000000++++++++++----------", lastchar_packed );
#
#   Convert packed last character to numeric last character.
#
    lastchar_numeric = stranslate("{ABCDEFGHI}JKLMNOPQR","01234567890123456789", lastchar_packed );
#
#   Concatenate the signed number back together.
#
    num_unpacked = sign_char num_strip lastchar_numeric;
    return num_unpacked
}

##
## translate function - like tr UNIX command.
##
function stranslate(from, to, target, lf, lt, t_ar, i, c)
{
    lf = length(from)
    lt = length(to)
    for (i = 1; i <= lt; i++)
        t_ar[substr(from, i, 1)] = substr(to, i, 1)
    if (lt < lf)
        for (; i <= lf; i++)
            t_ar[substr(from, i, 1)] = substr(to, lt, 1)
    for (i = 1; i <= lf; i++) {
        c = substr(from, i, 1)
        if (index(target, c) > 0)
            gsub(c, t_ar[c], target)
    }
    return target
}

 

[futurelet]

GoalieB,
A question about the code.
[tt]
/^2/ { ++pharm_cnt;} -- Type 2 record
[/tt]
If "Type 2 record" is a comment, why is it not preceded by a comment character?

 

[GoalieB]

Futurelet, that was added just for the post. It wasn't in the actual script.

GB

 

[futurelet]

The stranslate() function should be thrown out. It uses gsub(), but it doesn't escape regular expression metacharacters. In addition, its logic is wrong. Example:
[tt]
print stranslate( "_#", "#_", "foo#bar_baz" )
[/tt]
produces
[tt]
foo_bar_baz
[/tt]
Try something like this:
[tt]
function stranslate(from,to,target ,i,len,c,p,accum)
{ len = length(from); c = substr(to,length(to))
while ( length(to) < len )
to = to c
len = length(target)
for (i=1; i<=len; i++)
{ c = substr(target,i,1)
if ( p = index(from,c) )
c = substr(to,p,1)
accum = accum c
}
return accum
}
[/tt]

 

[GoalieB]

Thanks futurelet. I will try your version of the translate function and see if I still get the error.

-GB

 

[GoalieB]

Thanks to futurelet. I plugged in the new translate function and I haven't gotten the out of memory error since.

Thanks everyone!

-GB