Why it takes up so much memory?

[Tilen]

Hi

I have a problem, I load into an array from textfile. Textfile is 70MB. It's a simple load into an array, line by line and value by value.. no additional processing.

But when loading is done, my memory usage jumps up for 200MB!

This is how I do it:

...
assignfile(text_file,HandlerFlowsFN);
settextbuf(text_file,BufferSize);
reset(text_file);
i:=0;
SetLength(Handler_Flows,MasterHandlerFlowIndex);
while not eof(text_file) do
begin
Readln(text_file,loadtext);
Line.QuoteChar:='¨';
Line.DelimitedText:=loadtext;
Handler_Flows.A:=StrToIntDef(Line[0],0);
Handler_Flows.B:=StrToIntDef(Line[1],0);
Handler_Flows.C:=StrToIntDef(Line[2],0);
Handler_Flows.D:=Line[3];
Handler_Flows.E:=StrToIntDef(Line[4],0);
Handler_Flows.F:=Line[5];
Handler_Flows.G:=Line[6];
inc(i);
end;
FreeAndNil(Line);
CloseFile(text_file);
...

MasterHandlerFlowIndex is the number of records in file. It's 2.602.020. I also tried with incremental SetLength, first to set it to 10000 and then resetting it until it finishes, but same thing.

Anyone has any idea, why is it taking so much memory?
Is there anything I can do?

this is array :

THandler_Flow = record
A: integer;
B: integer;
C: integer;
D: string;
E: integer;
F: string;
G: string;
End;

Thanx a lot
Tilen

 

[sggaunt]

Well your handler_flow record structure isnt 'packed' and dosnt use short data types, so Delphi will allocate the default sizes for each item, these will mostly be larger or much larger than your actual file data.
You don't specify Buffersize for the text buffer.



Steve: Delphi a feersum engin indeed.

 

[Tilen]

Ok, I just found it in help :

THandler_Flow = Packed record

Does Packed have any implications on data, speed?
Would it be better if I use smallint ot shortint instead of integer?


I have BufferSize : array [1..8192] of Char;

Thanx
Tilen

 

[sggaunt]

You will certainly save memory by using shorter data types
Use sizeof() to determine the space that your full record structure is taking up.

showmessage(IntToStr(SizeOf(Handler_flows));



Steve: Delphi a feersum engin indeed.

 

[Tilen]

I tried with Packed and without it, same thing.

I changed all integer except first one to smallint with packed and without it... no change.

SizeOf is 4... I guess 4 bytes.. and with 2.6M records = 9.9MB...

I have Optimization in Complier options.

Any other suggestions?

 

[bledazemi]

You are making an error in calculations. One of your THandler_Flow record takes 28 bytes rather than 4.

2.6MB*28=72MB

Yet this is still less your reports of 200MB memory usage


Anyway you need to be aware that each record field requires its space.

THandler_Flow = record
A: integer; //4 bytes
B: integer; //4 bytes
C: integer; //4 bytes
D: string; //4 bytes (yes 4 even for empty string)
E: integer; //4 bytes
F: string; //4 bytes
G: string; //4 bytes
End;

It is true that sizeof(Handler_Flows) is 4. Handler_Flows itself is not a THandler_flow record its an array of such records and an array reference such as Handler_flows is basically a pointer and takes only 4 bytes.

You should check the size of your records using sizeof(THandler_flow)

By the way making your record packed will have no effect in this case because all your record fields take 4 bytes each (you would win some space if some record fields took a space which is not a multiple of 4).

For more in relation to packed records have a look at:

http://www.delphibasics.co.uk/RTL.asp?Name=Packed

"It is in our collective behaviour that we are most mysterious" Lewis Thomas

 

[bledazemi]

It just struck me that 72MB is the space required for 2.6 M records if the strings of your records are kept empty.

You mentioned that your text file is 70MB big , so if you load your all of it you will be using at least 72MB+70M=142MB


Still 58MB are missing somewhere




"It is in our collective behaviour that we are most mysterious" Lewis Thomas

 

[Tilen]

You're right, it should be SizeOf(THandler_Flow) ...

Well, I'll just have to accept the fact it takes so much memory and try to fidn a workaround or doing it differently.


Thanx to all of you guys for trying to help me.

 

[Tilen]

I think I got it.

It's the 'Line' thing:

while not eof(text_file) do
begin
Readln(text_file,loadtext);
Line.QuoteChar:='¨'; //<-- it looks like this is causing the problem
Line.DelimitedText:=loadtext; //<--
Handler_Flows.A:=StrToIntDef(Line[0],0);
Handler_Flows.B:=StrToIntDef(Line[1],0);
Handler_Flows.C:=StrToIntDef(Line[2],0);
Handler_Flows.D:=Line[3];
Handler_Flows.E:=StrToIntDef(Line[4],0);
Handler_Flows.F:=Line[5];
Handler_Flows.G:=Line[6];
inc(i);
end;
FreeAndNil(Line);


I tried it without Line usage, and it didn't use that much memory at all.

So, it looks like it doesn't use the same memory space for each line from file that it process into Line object. So the memory usage goes up and up.



Can I do it with pointers, would it work? How?

 

[bledazemi]

Have you tried clearing the line before proceeding:
...
Line.Clear;
Line.QuoteChar:='¨';
Line.DelimitedText:=loadtext;
...

I dont know if this will make a difference but its worth a try

 

[Tilen]

I tried

Line.Clear

Line.CleanupInstance

No change.

I also tried with

Line:=TSTringList.Create;// start of while
and
Line:=nil; or FreeAndNil(Line); // end of while

But this is actullay much worse, memory usage jumps up to 500MB with creating and freeing each time.

 

[Taltoss]

Out of curiosity why would you try to load a 70meg text file into an array? Sounds like it is almost time to think about using some sort of database interface.

 

[Tilen]

I just got it.

I was wrong the whole time. It wasn't the Line object, but strings...

I changed the record

THandler_Flow = record
A: integer;
B: integer;
C: integer;
D: string[1];
E: integer;
F: string[5];
G: string[5];
End;

Now it only uses a bit less than 100Mb, which is much better.

But, I don't understand why, because most of the values for D,F,G are one character long. So Delphi should've managed it properly.

Strange. Well, at least I got it down half size.

 

[Glenn9999]

But, I don't understand why, because most of the values for D,F,G are one character long. So Delphi should've managed it properly.

Pascal strings hold their maximum value in bytes. A short string would hold 256 bytes of memory (255 positions, plus a length byte).

It's always good to qualify your strings if you know they will always be less than a certain size.

 

[Tilen]

I thought that string is always used exactly how much text is in there. So if you store 'A' is only gonna use 1B, if you store 'ABCDE' is gonna use 5B, plus a length byte.

So A:String[5] actually takes 5B(+1) even if you put just 'A' into it.

 

[Glenn9999]

So A:String[5] actually takes 5B(+1) even if you put just 'A' into it.

Yes. For that matter, you might change "D: string[1];" to "char", because having the dynamic length byte is a wasted byte if you're going to have a max of 1 character.

 

[Tilen]

OK. Thanx.