ADDING AND RELEASING LIBRARY VFPCOMPRESSION.FLL 2

[Gerrit Broekhuis]

Hi,

In two applications I use zipping/unzipping files quite a few times. I'm using vfpcompression.fll to do this and this works very fast and reliable as far as I can see.
Every time I zip or unzip now I use the code like this:

SET LIBRARY TO vfpcompression ADDITIVE
&& do something with files, zip or unzip
RELEASE LIBRARY vfpcompression

This is working fine, but adding and releasing the library each time looks a bit weird. Why not just call the library in the start.prg and release it in the quitting procedure?
What are the pro's and con's?

Regards, Gerrit

 

[Mike Lewis]

I have never used RELEASE LIBRARY. I never even knew it existed. I simply do a SET LIBRARY ... ADDITIVE at the start of the session, and then forget about it. As far as I know, there is no reason to release a library - certainly no performance issues that I have come across.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

I agree, but indeed for a process like zipping in the range of MBs the load/unlod of a DLL is a small hit. It has more time cost if you'd connect and disconnect from a backend database between each query.

You can of course check allocated memory by SYS(1016). If you make tests at the command window notice in that window of the IDE VFP causes regular garbage collection, whereas you'd have to call SYS(1104) in your own code to trigger that. See Memory Management in

http://www.foxpert.com/docs/howfoxproworks.en.htm.

There are several documented and undocumented SYS functions about memory usage in the range SYS(1001) to SYS(1016).

All of these functions return varying values when executed in the Command Window, because the Command Window permanently enters the idle loop and therefore runs a garbage collection. Some functions filter on internally used handles, others don't do that. Nonetheless, these functions are good indicators for memory usage.

Which also means it's all a bit fuzzy and you don't get the immediate effect of RELEASE LIBRARY, so you get a blurry picture of memory usage with a lag that only becomes apparent after garbage collection, for eample. I don't find whether DLLs and FLLs fall into that VFP managed memory pool, but it would be strange, if not. There surely is more to get from Windows API functions, too.

One thing is obviously, though: The C/C++ code within an FLL can use VFP functions defined in winapims.lib that's shipped with VFP and linked when you build FLLs, but Craig Boyd might not have taken zlib source code and changed it to using VFP memory management, so that FLL gets memory more directly from Windows OS by malloc calls, reservicng process stack memory. That's not governed by VFP. Same is true for any API function, it's not so special, VFP simply has its internal memory management based on handles instead of addresses to support memory defragmentation with its own garbage collector.

I guess it can help to unload FLLs or DLLs therefore, not because of the memory they use up as themselves as code, that's surely still under VFP management, but because of the memory they allocate for themselves, ie for compression/decompression. That's where the remark comes from:

Usually, you notice the stack in error conditions such as "insufficient stack space" or "Mismatched PUSHJMP/POPJMP call". Only C/C++ developers writing an FLL have to deal with the stack.

And when a DLL unloads (or an FLL, which mainly still is a DLL with some extras specifically to ease the usage from VFP in comparison to DECLARE, but loads and unloads with the same mechanisms), it's own memory allocations should be deallocated and thus you could say you only get doubtless memory deallocations by unloading an FLL. Well, if an FLL is written with the VFP API C++ functions it also really puts its own memory usage under the roof of VFPs own memory management. So you only can be sure if you know the FLL source code and should assume stack memory allocation from FLL and DLL functions that are underthe radar of VFP, although not under the radar of Windows. The C/C++ code would need to be messy to cause memory leaks especially when unloading.

I have used exactly that compression FLL without unloading in a system that zips often without issues, so the internal stack memory usage should be okay, just because it's not on VFPs own radar it isn't bad. The base project we talk of here is zlib:

https://zlib.net/

And that's well maintained. While there are no FLL updates, the version Craig has used when building his FLL is still good to go, I don't suspect Win11 changed something in that respect, for example, you'd see it break immediately, if it breaks, as the main thing used of Windows is memory allocation, anything else is the compression/decompresion algorithms that are done within that memory, nothing Windows itself influences. So if memory allocation a la zlib 1.1.x or whichever version is within vfpcompression.fll is changed, it would stop working instead of degrading over time.

There's much caching done on files with hard and software and on the OS level, so to unload an FLL file put side to side with your EXE in the same appdir and then load it again should go fast. Not as fast as compressing a string in memory, but whenever you create new ZIP files that's surely the bottleneck as the initial writing of a file is always uncached file IO in comparisn to loading a DLL you already had loaded before.

I often added to an existing archive, so that would surely suffer from permanent unloading/reloading the FLL as it keeps forgetting what it had cached internally. If you do new files in every ZIP process it won't suffer from refreshing the FLL, but you also likely have no real problem you solve with this, as memory management still is okay not done with the VFP memory management. The only thing you gain is the memory of the FLL and what it cached for periods where you don't use the zipping functionality.

Observe and see which effect it has on performance and stability. I see Mike has written a short answer from experience and in the end that's what I know from this FLL, too. It cn be different with different FLLs. but in the majority of cases you don't have to deal with low RAM of a client, unless you apply a low limit to your own process with SYS(3050), which would be stepping on your own feet.

Chriss

 

[Chris Miller]

One more addendum: Process memory is mainly divided into a processes heap(s) and a stack. And you can use both with Windows API calls also available more directly in the C language than in VFP. It's more likely the compression/decompression happens in memory allocated on the or a heap, nevertheless it's not within VFPs memory management but it would hardly be successful if it would have memory leaks in its heap or stack usage, so you can use vfpcompression.fll on the basis of loading once and keeping it loaded. Besides the dependence on how ou use it, as given with my example of adding to a zip archive over thee span of multiple interactions that contribute to it until it reaches a threshold and a new zip is started.

Chriss

 

[Gerrit Broekhuis]

Hi Mike and Chris,

Thank hou very much for your comments. I will do more testing next week and keep an eye on stability and memory usage. I do have some specific memory mamangement settings.

So far I’m very hgappy with this compression method.

Regards, Gerrit

 

[mJindrova]

Note...
VFPCOMPRESSION.FLL is compiled with static zlib version 1.1.4



mJindrova

 

[Chris Miller]

Thanks Martina,

it's in fact a bit of a weak point of the FLL that it sticks to some version of the zlib library.
Indeed I find a bug report from this year March:

https://www.openwall.com/lists/oss-security/2022/03/24/1

mentioned in an article in TheRegister

https://www.theregister.com/2022/03/30/zlib_data_bug/

It would be nice to have source code of FLLs to update included components, too, that are under continued development, whereas the FLL isn't.

No panic. As always, it is most important to have up-to-date antivirus software that finds malicious code, also within zips and also malicious zips themselves, but you don't need to be afraid of vfpcompression producing malware by random chance, even though Travis Ormandy says:

it really was a bug in
zlib-1.2.11 when compressing (not decompressing!) certain inputs.

Well, in short the data or files you compress would need to be such "certain inputs" by pure chance.

Chriss

 

[Chris Miller]

Just two more very general thoughts on DLLs:

Dynamic
Just because they are called dynamic link libraries, does mainly mean they are separate files not compiled into an EXE. It does not obligate you to load and unload them. The possibility to do so can be used for example, when you have a start routine using a DLL to verify signatures, which you only do once at startup, then of course, release those dlls after that's done. But even if you only use a function sparsely and not regularly, I see no reason to unload and load things. In short dynamic offers the ability to load and unload, but does not obligate to do that.

Runtime DLLs
Think about the VFP9 runtime itself, it actually runs your EXE after the EXE loads and points the runtime to your main start file within the EXE, you don't unload the runtime dll, it would end your EXE. What that means is the DLL doesn't have the nature to bloat memory usage or the nature to need unloading when it's not used all the time. The memory including code loaded from DLLs isn't more vulnerable than your core EXE code and also doesn't expire or degrade. If you're afraid of program errors causing memory leak, then why use a programming language that's built with C/C++, which still has this issue? VFP isn't a compiler or language that bootstrapped itself from its C/C++ root and will also always load the MS C/C++ runtime alongside the VFP runtime DLLs. So, if you're mainly concerned with memory management errors that cause bloat or leaks or buffer overruns and exploits, then you would need to program in a language generation that does hide away memory management from the developer, as VFP does, but also is in itself programmed in such a language, which VFP isn't.

Chriss

 

[Chris Miller]

Oh, and obviously you can instead make use of OS ZIP file handling for zipping and unzipping, too, see faq184-5113

Of course vfpcompression offers more than these two simple operations, you can get callbacks for showing progress of zip/unzip, you can get zip contents, add or extract single files, etc.
I think the latter is also possible with Shell.application, as o.NameSpace(cFileZip).items is the collection of files of which you can also decide to only zip or unzip some of them, or instead of using copyhere just determine filenameand other properties.

But further features like encryption have to be done in other ways, if you'd change to using the OS shell.application.

Chriss

 

[Steve-vfp9user]

I use this and had the advice sometime ago on this forum:

SET LIBRARY TO LOCFILE("vfpcompression.fll")

?ZipOpen('\myfile.zip', '\myfolder', .T.)

?ZipFile('\myfolder\table01.dbf', .T.)
?ZipFile('\myfolder\table02.dbf', .T.)

....more of the same here

?ZipClose()

SET LIBRARY TO

Not sure if that is one of the same, but my purpose was to zip and back up tables to a USB memory stick

Hope that helps

Thank you

Steve Williams
VFP9, SP2, Windows 10

 

[Chris Miller]

The only difference of the end using SET LIBRARY TO vs RELEASE LIBRARY vfpcompression is, that you release all FLLs your way, Steve. It makes a difference, if you use multiple FLLs, but no differene, if you only use one FLL.

Chriss

 

[Steve-vfp9user]

Hey Chriss

Yes, absolutely. That is the only FLL I have in that project but right to point it out.

Thank you

Steve Williams
VFP9, SP2, Windows 10

 

[Mike Lewis]

Steve,

May I make a small suggestion. If you only intend to run your code yourself - as a developer tool - it should work well. It is nice and simple, and will do its job. But if it is likely to be run by end users, I would argue strongly against using LOCFILE(), for two reasons.

First, if LOCFILE() can't find the target file (in this case, the FLL) in the default directory or search path, it displays a File Open dialogue. The trouble is that the end user probably won't know what to use this dialogue for: they won't know what file they are supposed to be looking for. They probably don't know what an FLL is, nor should they need to. Locating files for internal use this way should be a matter for the program, not the user.

Secondly, if the user cancels out of the dialogue (which they will do if they don't why it is there), VFP invokes your error handler. This is not what you want.

I hope you don't mind me pointing this out.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads