Performance problem 4

[jamesrav]

I have about 90,000 dbf files in a single folder. They contain from 0 to 100 records, but generally 5-20. I have a .prg file that cycles thru all the tables and does some processing based on contents. The problem/puzzle is that sometimes this .prg file can go thru all 90,000 in about 1 minute, and other times it can take 15 minutes. No change to the .prg at all. I've checked the task manager and it's not the case that the CPU is occupied by other processes when the .prg is functioning slowly. It just seems that VFP in some situations 'knows' about the tables and can move thru them quickly, and in other cases it is seeing the data for the first time (and I should add that I cycle thru the 90,000 multiple times with different criteria, and when it's operating well, the 1 minute time is consistent , and when slow, it's 15 minutes every time thru - never improves). So is this a VFP issue or Windows in general? I've tried adding some Sys() commands at the start (based on the Help file) but it does not improve things.

 

[Rick C. Hodgin]

Mike, the argument Chris made was a teaching argument. He pointed out that Christian would still be able to accept or reject my code even if I refactored his existing function. I hadn't thought of that before. So, he taught me something.

You may have said the same thing, but you didn't do it in a teaching way, and it didn't make me rethink my former way of thinking.

This is the first I'm hearing of an error. I'll try it on my system and see if I can workaround it.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I have Windows 2000 Professional, Windows Server 2003, and Windows 7 Professional. I can't replicate the error in any of those versions because they don't have a c:\windows\csc\ folder. However, using adirex() recursively on c:\windows\ completes in several seconds with no errors and a fully populated array.

--
Rick C. Hodgin

 

[Chris Miller]

CSC is a client side caching folder and I guess it's not always present, but strictly a system folder. It may only exist once you activate Offline Files.

The situation is even more dire than for hidden folders, for which Adir has flags to include or exclude them. When AdirEx tries to read in the CSC folder it throws error 1098 and stops there. When you let ADirEx create or populate a folder, you have all data up to that point, but there's no continuation. The properties of the CSC directoy say it has no owner and to get access you would need to take ownership. Well, if you turn Offline Files off, you can remove that directory.

I'm sure you could replicate this by simply creating a folder under ownership of another user account and rejecting permission to your normal account, then the same should happen. But no worries, you didn't introduce the error. It's simply not foreseen to handle such exceptions.

Adir() and Sys(2000) treat the case, as if "C:\Windows\CSC\ is an empty directory. I think there is a chance to improve that by adding the info that directoy can't be read with your permissions, stored with the record (or array item) of the folder or in an extra cursor or array, like TABLEUPDATE() creates with the right parameters. The usual exception handling isn't helpful in the middle of recursing a large directory structure like c:\Windows\ with lotsof special and system folders. Well, that's something Christian didn't foresee as he didn't introduce recursion.

Chriss

 

[Rick C. Hodgin]

I think I know in the code where that error's being generated. Will look at it later this week.

--
Rick C. Hodgin

 

[Chris Miller]

Rick, congrats to the MVP of the week.

Chriss

 

[Rick C. Hodgin]

Thank you, Chris.

 

[Rick C. Hodgin]

Mike, the algorithm used by Christian's original code is the exact same algorithm used by my enhancement.

My change has these differences:
1. It goes into folders rather than just reporting on them
2. If using my code it creates the FULLPATH() name, not just the filename

Everything else is the same. The algorithm is literally Christian's algorithm, save those enhancements.

--
Rick C. Hodgin

 

[Mike Lewis]

I've nothing to add to this discussion. I just wanted to be the first person to add Post Number 100 to a Tek Tips thread.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

Mike Yearwood,

you're overlooking the other fact: I tried both FLL versions and theyboth fail on the CSC folder. Even if you use a recursive function that doesn't recurse, Christians original FLL will be used on c:\Windows\CSC at some point and throw the error. So I wonder if you really used that FLL. The error was not introduced by Rick, it is already thrown by the original FLL.

Chriss

 

[Rick C. Hodgin]

In the versions of Windows I have I'm not able to reproduce the error you're seeing, Mike. I'm not sure where / why it's happening in the code.

Since the source code is available and you are able to reproduce the error, if you're interested in solving it install Visual Studio, get the source code, compile into debug mode, and I'll go online with Zoom or Teams and we can go through and figure out where the error's happening. I'll correct it and when it's all working I'll submit a new upstream patch to the original library on GitHub.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Mike, my offer remains open if you'd like to help fix the bug.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I'd prefer to work together. Teamwork is important. And solving bugs together brings comraderie amongst people. We could use some comraderie with you, Mike.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Mike, you're an amazing person. Truly stunning. I mean that sincerely.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I'll research the bug and post the correction when found.

--
Rick C. Hodgin

 

[Chris Miller]

Mike Yearwood,

while 344906 is very different from 187399 it's still just the inherited error you ignored using the original FLL. Ignoring that in the middle of recursion means not determining all files past that folder and if you do so you get the rest of the files. You still just ignore the facts that already were shown to you.

But thanks for pointing out that you ignored the error, it wouldn't have worked to get through all of Windows folders otherwise, but initially you acted as if you have proven Rick has introduced the problem. It' s recursion that introduces the problem. it's recursion that requires to not stop in the middle of its work, ignore or collect error information and continue. Or similar to Tableupdate() offer the option to stop or continue with the rest. And you don't even realize that that's the only problem, that this leads to that result and no further error has been introduced.

Chriss

 

[Rick C. Hodgin]

Mike, everything you post is negative. I'm reminded of a line by Col. Thomas McKean in 1776: "That man would depress a hyena."

Have you no love? No joy? No peace? Is everything a battle to you?

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I do not consider anyone a friend in the Fox community anymore.

Noted.

How sad.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Good luck, Mike. And good-bye.

--
Rick C. Hodgin

 

[jamesrav]

this thread certainly grew a lot I have not needed to test any of the methods supplied as of yet (thanks to all for their contributions) since running the little 'pre-check' routine (which takes about 9 minutes) followed by an immediate 2nd run of it (which takes as little as 30 seconds), has always then led to fast iteration thru the 90,000 files. In one case it took multiple tries of the pre-check, and none were faster than 3 or 4 minutes. I expected this to result in slow processing for the duration, but as time went on the processing gradually improved and eventually reached 'max speed'. And that was only a single occurrence. I'm quite amazed that even with a slowish old Dual Core computer it can run thru 90,000 files and do some filtering, processing, writing to a table, etc in 1 minute.

 

[Rick C. Hodgin]

I'll research the bug and post the correction when found.

Attached is the version which corrects the exception issue when attempting to access something the user doesn't have access to while recursing.

Here's the source code change on GitHub.
Here's the Pull request on Christian's original vfp2c32 project.

--
Rick C. Hodgin