Max size of array

[lfoata]

I am using nawk on a solaris 2.6 server.
the script has been running for a a few month without any problems.
For the past two days it has been looping and sucking CPU like crazy.
The only change is the input that is larger than before.

The script builds an array with the input.

The question is : Does anyone know if there a maximum number of occurences in an array with nawk ? and if so what is the maximum size ?

 

[vgersh99]

What I've encountered with Sun's nawk's arrays is that it'll keep allocating entries til it runs out of "swap". Your CPU will "peg up" and nothing will be done.

The way out is to reexamine your implementation logic.
If somehow your associative arrays are indexed by "integers" maybe should be converted to "floats" which will be stored/indexed as "double".

vlad

 

[lfoata]

Thanks for your reply vgersh99 .
I checked the swap, only 100 Megs are used and 1 Gig is free.

Here is some more details:
The array has about 200.000 rows with about 6 colums. It comes from an input file of about 200000 lines.

Any other ideas ?

 

[Krunek]

lfoata,

Here are some limits of "one true awk:"

100 fields
3000 chars per input record
3000 chars per output record
1024 chars per field
3000 chars per printf string
400 chars max literal string
400 chars in char class
15 open files
1 pipe
double-precision floating point


Maybe it helps.

Bye!

KP.

 

[Krunek]

Hi again!

I'm not *nix expert. I use DOS, Win, and Linux
versions of awk. But I have some associations:

(1) Maybe your nawk has some RAM-memory
limitations.

If take to much data, DOS version of gawk
displays message "can't allocate memory" or
"memory exhausted."

(2) In fact, all arrays in awk are associative;
numeric subscripts are converted to strings
before using them as array subscripts.

Maybe is string that supscripts an element of the
array to big.

Bye!

KP.

 

[pcdas]

Hi Ifoata

Krunek listed the limits available in the books. But your question of if there is a maximum number of occurences in an array with nawk ? and if so what is the maximum size remained unanswered. I too have faced the array limit problem many times but could not figure out the real reason, till date. The array limit has been always a big problem . As per my experience the size of the array subscript and the total data holding in an array decides the limit and there is no fixed formulae for it.
I had created a second array whenever I faced similar problem. You can count the no. of elements you are storing in your 1st array and start storing in the 2nd array when the limit exceeds. Best of Luck.
... P C Das (India)

 

[lfoata]

Thanks to all the answers. Since I could not find out what the limet for an array is, I decided to build a workaraound. I created a database with the data and query the database now. It took some more time, but is more flexible and easyer to manage.

Thank you again