VFPCompression - 'Forgetting' ZipOpen()/ZipClose()

[GriffMG]

I use Craig's VFPCompression a lot, it is brilliant.

Once in a while though, after adding a file to a zip, VFP 'forgets' the ZipOpen() and ZipClose() procedures.
So I get an error and my zip file is lost. The error looks like this: Nature: File 'zipclose.prg' does not exist.

I'm trying this just now - and it seems to work, but is there a better way?

	** just after adding a file
	REDUCEMEMORY()
	SET LIBRARY TO (m.WHEREAMI+"VFPCOMPRESSION.FLL")
	ZIPCLOSE()

The REDUCEMEMORY() function is a simple call (and may not even be necessary).

FUNCTION REDUCEMEMORY
	PRIVATE NPROC,BB
	DECLARE INTEGER SetProcessWorkingSetSize IN kernel32 AS SetProcessWorkingSetSize  ;
		INTEGER hProcess , INTEGER dwMinimumWorkingSetSize , ;
		INTEGER dwMaximumWorkingSetSize
	DECLARE INTEGER GetCurrentProcess IN kernel32 AS GetCurrentProcess
	NPROC = GETCURRENTPROCESS()
	BB = SETPROCESSWORKINGSETSIZE(NPROC,-1,-1)
	CLEAR DLLS SETPROCESSWORKINGSETSIZE
	CLEAR DLLS GETCURRENTPROCESS
	RETURN(.T.)

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

To my knowledge unloading an FLL is done with RELEASE LIBRARY only. As you need to SET LIBRARY again after calling your REDUCEMEMORY function, I guess SetProcessWorkingSetSize causes the FLL to unload, even though VFP should "protect" this resource. I remember having read about SetProcessWorkingSetSize from Christof Wollenahaupt, as a more stringent way than controlling the memory usage via SYS(3050).

When this is the case, there are two logical things to make this procedure failsafe again: Only run it after you're finished with any FLL function usage or also REAEASE LIBRARY and then SET LIBRARY again.

I maintain an application that does CLEAR DLLs after each and every DLL function usage, I don't think of this as a good way. In todays computers, but even at times we had far less then 2GB for a foxpro process, loaded DLL functions are not a resource problem, it should be sufficient to manually trigger garbage collection via SYS(1104).

I didn't try to replicate the problem, but I also wonder, why you would call REDUCEMEMORY before ZIPCLOSE. The logical point in time to free resources is after you finished some process with external tools like OCXes or DLLs or FLLs, so I'd only REDUCEMEMORY after ZIPCLOSE.

If you're from the camp of releasing everyhintg as soon as possible and not profit from things resident you'll use more often anyway, then perhaps the best option is to add RELEASE LIBRYR calls for all loaded libraries into the REDUCEMEMORY function and then only call it after usage of something is completed, for example put that into a zip class Destroy() event. And also make it a habit to SET LIBRARY repeatedly at the Init() of some class.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[GriffMG]

Hi Olaf

I think you have slightly misunderstood, the reducememory() and set library calls I have added and seem to resolve an issue whereby VFP is 'losing' the previous SET LIBRARY to the VFPCompression.fll after a number of files are added to a zip. Like a memory leak or something, I put the reducememory in - in case the zip file had taken over some memory that could be released - and have to do the new set library because otherwise it crashes at the ZIPCLOSE().

This happens on one machine, doing a zip process that is repeated everyday on about 30 machines, once in a while (twice a week, maybe), we are talking fairly large files and significant number of files, but the process usually completes successfully when restarted.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

So you say this effect also happens when you don't call SetProcessWorkingSetSize via REDUCEMEMORY at all? Just the sole use of the FLL?
Perhaps even more so?

Craig Boyd might be able to say more on the inner workings, I know ZIP is based on zlib.dll, and it has ways to specify working memory for compression. I guess Craig made the FLL based on the open source code of this DLL with its lib, so statically linked, instead of using the DLL, anyway the FLL inherits the memory management of the zlib code.

There's not much you can do from outside, and if purging memory even causes VFPCompression functions to be unloaded I doubt it leads to healthy result zip files.

The FLL has a weakness Craig never addressed, no matter how you set the compression level the compression remains medium. It's a pity the projects Craig used are not open source, as to fix this problem I think you'd need finer control about the compression buffer used and be able to trigger garbage collection for the library.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[GriffMG]

I haven't retried it without the ReduceMemory() call yet, the code as is does seem to work *more* reliably, I just wondered why VFP was forgetting the SET LIBRARY and whether simply putting another call to it was the best approach - it feels like a road roller to crack a nut.

I had the idea that Craig had pretty much retired, don't know why I had that idea, so was unlikely to answer a query like this, but I will try and email him - perhaps he will humour me.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

>I just wondered why VFP was forgetting the SET LIBRARY

It seems SetProcessWorkingSetSize is very rigorous of freeing memory, even objects still in use. An FLL is nothing but a specific DLL, so it resides in memory with the main process and its memory also is purgeed from any memory management function. SetProcessWorkingSetSize is meant ot be used onece initially, but I even find myself recommending it, it's also mentioned in a FAQ here. I didn't made that bad exxperience with it, maybe it's also a fauit of the FLL or zlib library and not a general problem of how VFP loads FLLs and keeps its code in memory.

Bye, Olaf.



Olaf Doschke Software Engineering

https://www.doschke.name

 

[Steve-vfp9user]

GriffMG

I'm still learning VFP9 but use the following to compress a file (which is later emailed as a backup). I have had no issues with this:

WAIT "Adding compressed back up file, please wait...." WINDOW NOWAIT

SET LIBRARY TO LOCFILE("vfpcompression.fll")

?ZipOpen('\myzip.zip', '\backupfolder', .T.)
?ZipFile('\backupdata\myfile.dbf', .T.)
?ZipFile('\backupdata\myfile.cdx', .T.)
?ZipFile('\backupdata\myfile.fpt', .T.)

* More files here....

?ZipClose()

SET LIBRARY TO

Not sure if that would help to stop you losing the process. I got the code also from Craig's VFPCompression.

Steve Williams
VFP9, SP2, Windows 10

 

[GriffMG]

Hi Steve,

Thank you for that, it is practically the same code as I use, except I have it dressed up in a loop to add the files one by one and stop from time to time to find out how big the zip file is and what's left to go in there (it can get bigger than 2GB, so I watch out, stop at 1.5GB and start a new file a little later).

This losing the access to the code is fairly random, only seems to affect one out of about 30 processors and usually succeeds on a restart, of the process - not the server.

I seem to have created a solution, redoing the SET LIBRARY regardless of whether or not I need it, but I don't like 'sometimes'... I like certainty.


Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Steve-vfp9user]

Glad you got it sorted

Thank you

Steve Williams
VFP9, SP2, Windows 10

 

[Olaf Doschke]

I don't know if you get certainty repeating SET LIBRARY evertime before using a function. The ZipOpen/FipFile/ZipFolder/ZipClose family of functions will all work on a file handle you create with ZipOpen and don't get passed out, this is stored in some internal place in the memory allocated by the FLL. By the way ZipFolder obviously is no solution, as you then can't stop and create a next ZIP archive, if you reach 2GB.

The only real safe way including usage of REDUCEMEMORY I think would be to ZipOpen/ZipFile/ZipClose/REDUCEMEMOERY and do RELEASE LIBRARY/SET LIBRARY within REDUCEMEMORY to get rid of any memory leaks leading to errors in long run usage of the FLL. It would be a serious Windows issue if RELEASE LIBRARY, which unloads the FLL (which technically also is a DLL) would fail to deallocate memory fully. But you need ZIPCLOSE first, as unloaduing the FLL also means unloading the internal file handle.

Or you use something completely different when you identified this FLL is unstable for your usage.

Obviously, a mode automatically spanning an archive would be nice. 7zip does that with command line

#DEFINE HIDEWINDOW 0

DECLARE INTEGER ShellExecute IN shell32.dll ; 
  INTEGER hndWin, ; 
  STRING cAction, ; 
  STRING cFileName, ; 
  STRING cParams, ;  
  STRING cDir, ; 
  INTEGER nShowWin
  
cHome = Addbs(Home())
cAction = "open" 
cFileName = "C:\Program Files\7-Zip\7z.exe"
cParams = [a "D:\temp\archive.zip" "&cHome*.dbf" -v1k]
lnResult = ShellExecute(0,cAction,cFileName,cParams,"",HIDEWINDOW)
?lnResult

This creates 1KB sized archives (-v1k) of all dbf files in HOME() if you want to cap this to 2gb use -v2g, I'd perhaps cap this below this threshold, either to 2000m or simply 1g, 1gigabyte files.

You find 7zip at

https://www.7-zip.org/

and the documentation of the 7z.exe commands and switches at

https://sevenzip.osdn.jp/chm/cmdline/commands/index.htm

and

https://sevenzip.osdn.jp/chm/cmdline/switches/index.htm

Olaf Doschke Software Engineering

https://www.doschke.name

 

[GriffMG]

Well Olaf,

It does seem to be working thus far, I have had three errorless creations, all just below 1GB.

I'll come back at the end of the week and let you know.


Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

I can imagine reloading the library works as I already guessed by looking into loaded libraries and seeing it's unnecessary to load again or if it loads again the file handle there must be somewhere is just 1 as is most often the connection handle to SQL Server you get from SQLCONNECT/SQLSTRINGCONNECT.

We'll see, if you get errors sometimes I suspect not because 'File 'zipclose.prg' does not exist" (or zipfile.prg or whatever), but because when SET LIBRARY really reloads the FLL the file handle it stores internally is not initialized and therefore invalid. This would just be logical.

Bye, Olaf.





Olaf Doschke Software Engineering

https://www.doschke.name

 

[GriffMG]

I'm thinking you may be right, let's see if it makes a week.

I notice the other 29 have not failed, like this, in more than a decade of continuous use...

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

The other 29 customers? Might be they just don't have that much data. If you're sure files don't make up more than 2GB I'd also recommend using ZIPFOLDER, simplest way of using VFPCompression, but that goes against the idea to be able to stop after each file.

Maybe they just had a bad drive sector, bad combination of files - what speaks against the latter is the zipping works when you restart it.

Anyway, there are tons of ways to zip files, the FAQ also has several solutions.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[GriffMG]

It is the biggest quantity of data by far.

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you. ​

 

[Olaf Doschke]

The maximum database size I was responsible for was 18GB, but my application wasn't backing that up, the backups were done by the file servers hosting that database besides others. Maybe that would also be an idea for this customer, simply use usual backup systems on the server side.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name