lowering memory fragmentation

[Cosmin3]

Hi.

In one of my applications I use an array of records (hundreds of items) with 4 memory streams (a few MB each) in each item.
Most of the times the data from one of the streams is backed up in one of the streams (for Undo) and then is modified and the result is copied into the other 2 and into the source stream (in different forms, not exactly the same) - for each item.
There is no way to know the exact size of each stream before the modification (I'm using 3rd party components to compress the data). So I set the size of the stream to the uncompressed data size, compress and then set the size to the value returned by the 3rd party component.
Sometimes items are inserted and/or removed from the array.
The application is 32 bit and using ~ 1.5 GB RAM on 32 bit OSs and ~ 3 GB RAM on 64 bit OSs (large address aware).
But sometimes is not enough, especially when the user is performing a lot of operations on data, fragmenting the memory.

I use now Delphi XE8 with the default memory manager (FastMM4).
For very good reasons I can't make a 64 bit application so I'm stuck with 32 bit.
My current OS is Windows 8.1 x86.

I searched before asking but couldn't find anything useful.

Once I tried to make a custom TMemoryStream, didn't work.

Any tip on how to lower the memory fragmentation?

Thank you in advance.

 

[whosrdaddy]

Fragmentation comes from reallocating blocks of memory.
Try to work with fixed size streams (or try to lower the number of reallocations).

/Daddy

-----------------------------------------------------
Helping people is my job...

 

[Cosmin3]

Thank you.

That's a good advice, but unfortunately I can't work with fixed size streams since every item from the records array is unique (different size, different content). And I already lowered the number of reallocations to minimum.

Meantime I tried another approach: to make again a custom memory stream. But this time I didn't use any ancestor and I divided its memory in chunks (usually 8 KB in size).

This is the current code

http://files.engineering.com/getfil...d31-0202cd6867bd&file=VirtualMemoryStream.pas

I used this code to test it:

function CurrentProcessMemory: Cardinal;
var
   MemCounters: TProcessMemoryCounters;
begin
   MemCounters.cb := SizeOf(MemCounters);
   if GetProcessMemoryInfo(GetCurrentProcess,
      @MemCounters,
      SizeOf(MemCounters)) then
      Result := MemCounters.WorkingSetSize
   else
      Result := 0;
end;

const
   Iterations = 500;
   InitialSize = 1048576;
var
   i, j: Integer;
   arrTest1: array of TVirtualMemoryStream;
   pmem: Cardinal;
begin
     pmem := CurrentProcessMemory;
     SetLength(arrTest1, Iterations);
     for i := 0 to Iterations - 1 do
     begin
        arrTest1[i] := TVirtualMemoryStream.Create(InitialSize);
        arrTest1[i].FillMemory(0, InitialSize, 3);
     end;
     i := 0;
     try
        while True do
        begin
           Inc(i);
           for j := 0 to Iterations - 1 do
           begin
              arrTest1[j].Size := 11 * arrTest1[j].Size div 10;
              arrTest1[j].FillMemory(0, arrTest1[j].Size, 3);
           end;
        end;
     except
        pmem := CurrentProcessMemory - pmem;
        for j := 0 to Iterations - 1 do
           arrTest1[j].Free;
        ShowMessage(IntToStr(i) + #13 + FloatToStr(pmem / 1048576));
     end;     
end;

It gives OutOfMemory error when tries to increase from 1738 MB to 1912 MB (14th operation).
I tested with a default TMemoryStream and the error appears when tries to increase from 1389 MB to 1528 MB (11th operation).

So it appears the memory is less fragmented with TVirtualMemoryStream.
But I'm opened to critics and suggestions...

 

[whosrdaddy]

I wouldn't do any reallocs, in your case.
Use one memorystream (allocated at largest size possible) and write a manager that handles your objects.

/Daddy

-----------------------------------------------------
Helping people is my job...

 

[Glenn9999]

It sounds from your program requirements like you need to just allocate one fixed block of memory when your program starts and then leave it until your program exits. Then manage what you do within the program with that memory and record things yourself.

I don't know if I can offer any further assistance, but if it helps, I did write a generic file sorting program once upon a time which handled a lot of memory at one time. One discovery I did make in the process which you will need to think about is that memory you use isn't necessarily all "memory". Depending on how the OS memory manager handles it (I'm pretty sure Microsoft hasn't touched it since XP for the 32 bit side), you might be thrashing the disk a lot for memory being paged in and out, and not end up with what you might think.

In any event, a general rule I got hit with hard is that you can't necessarily guarantee any amount of memory will ever be available on a system at any time your program runs. So you need to be very flexible about how you go about it as well as your expectations.

Hope some of that helps...

 

[Cosmin3]

Thank you both for the advice.
But my app has to work with any stream size (not just a few MB) and it should use memory as it needs, not take all the memory.
Yes, I see the advantages of using a fixed size memory area, but unfortunately I can't use this approach

For now I'm working on improving the virtual memory stream.
One of the streams from the record array is compressed with LZ4 (C++ dll). The problem is it requires, for compression or decompression, a contiguous memory area; but the virtual memory stream has its memory in noncontiguous 8 KB chunks.
Options:
1. I can use a contiguous memory area as buffer but it will increase memory usage and fragmentation and it will decrease the speed.
2. I could modify the LZ4 sources to work with chunks. But my C++ programing skills are not so great

 

[Glenn9999]

>But my app has to work with any stream size (not just a few MB) and it should use memory as it needs, not take all the memory.

Sounds like that generic file sorting program I mentioned. Like I mentioned above, be flexible. Based on the job at hand, I have X byte records, and have Y amount of memory to work with. So I allocate enough memory (once) for Z records when I sort, which won't starve the system and thrash the disk (there are API calls to find this out). If the number of records in the file is greater than Z records, then I have to process the file in sections, put them to temporary files, and then merge the results. Of course if the file records are < Z, then I can just allocate the needed memory, do the sort, write straight to output and be done. More or less, the design takes into account that it may or may not have a certain amount of memory to play with, so it adjusts itself to get success no matter what the memory conditions are and no matter what size of file is presented it.

So, you can definitely work it out to do something similar with whatever your process is. Of course, most programmers don't have to think this way with their programs, since memory is not a factor of consideration 99.999% of the time. So it seems weird. But it's quite workable, when you think it through.

 

[Cosmin3]

I'll try to explain more what the app is doing.

The streams are in fact animation frames.
In the main stream each frame is stored using lossless compression (lz4). When the user is changing the frames using the interface, each frame is decompressed, modified and recompressed. Also, in the 2nd and the 3rd stream the image and its transparency are stored using a lossy compression (they are needed for preview and for saving the animation in a file).
The 4th stream is for Undo, it stores the data from the main stream before making the modifications.

This is the app (installer):

https://drive.google.com/open?id=0ByKxAD_t9uvLZ19ScGhIN1E3LXc

If you want I can show you the source code too.