Performance problem 4

[jamesrav]

I have about 90,000 dbf files in a single folder. They contain from 0 to 100 records, but generally 5-20. I have a .prg file that cycles thru all the tables and does some processing based on contents. The problem/puzzle is that sometimes this .prg file can go thru all 90,000 in about 1 minute, and other times it can take 15 minutes. No change to the .prg at all. I've checked the task manager and it's not the case that the CPU is occupied by other processes when the .prg is functioning slowly. It just seems that VFP in some situations 'knows' about the tables and can move thru them quickly, and in other cases it is seeing the data for the first time (and I should add that I cycle thru the 90,000 multiple times with different criteria, and when it's operating well, the 1 minute time is consistent , and when slow, it's 15 minutes every time thru - never improves). So is this a VFP issue or Windows in general? I've tried adding some Sys() commands at the start (based on the Help file) but it does not improve things.

 

[Rick C. Hodgin]

Can someone else test it? It works on my machine.

--
Rick C. Hodgin

 

[Chris Miller]

The version of the download FLL is 2.0.0.19, which is Christians original version number, maybe you uploaded the wrong FLL file.
Edit: It acts exactly the same as Christians original FLL, Error "Access Denied" when trying to get files with file skeleton 'C:\Windows\CSC\*.*'.

Chriss

 

[Rick C. Hodgin]

Is it not working?

--
Rick C. Hodgin

 

[Chris Miller]

As said, it's still stopping and erroring.

If your Window folder has no CSC subdirectory or you can raead it, here's a way to reproduce the problem:

lcDir= GETENV("TEMP")+"\accessdenied"
MKDIR (lcDir)
RUN cacls "&lcDir" /e /c /d %username%
? adirex("c_Files",lcDir+'\*.*')

That results in error 1098 'Access denied'.

I see a binary difference comparing filetostr() of both flls (original and latest download) with ==, so there is a change, but not as needed.

For sake of later deleting the directory from temp:

lcDir= GETENV("TEMP")+"\accessdenied"
RUN cacls "&lcDir" /e /c /g %username%:f

Chriss

 

[Rick C. Hodgin]

That's the correct version. It works on my computer using Windows Server 2003 logged in as a regular user and trying to access folder created by the administrator. In the old FLL version I get Access Denied, and in the new one it just logs the folder's existence and then keeps going to other folders.

I'll need someone else's help to see what's happening on other people's computers where it's still getting Access Denied.

--
Rick C. Hodgin

 

[Chris Miller]

Rick, when I take your current GitHub repository and compile it I get a version that does ignore errors.

So your change works, but the FLL you uploaded doesn't contain it.

Chriss

 

[Rick C. Hodgin]

That version I uploaded is the same version that is on my PC and does work. I did a WinMerge compare on the two, and they are binary identical.

The vfp2c32.fll I uploaded yesterday was built at 10:03pm on November 10, which was just after the 10:02pm build of my vfp2c32d.fll debug version I was testing beforehand. It is the correct version.

When I do a full Rebuild on the source code just now, it generates an ABI which shows only two three byte sequences that are different in the entire file:

Difference at    312, tek-tip's fll = 11, new vfp2c32.fll = 21
Difference at    313, tek-tip's fll = 88, new vfp2c32.fll = da
Difference at    314, tek-tip's fll = 8c, new vfp2c32.fll = 8d
Difference at 257444, tek-tip's fll = 0e, new vfp2c32.fll = 1d
Difference at 257445, tek-tip's fll = 88, new vfp2c32.fll = da
Difference at 257446, tek-tip's fll = 8c, new vfp2c32.fll = 8d

And each time I rebuild, those same three byte sequences change, which I discovered through dumpbin /ALL that those are the bytes for the date code:

[61][8D][DA][21] time date stamp Thu Nov 11 22:06:09 2021
[61][8D][DA][1D] time date stamp Thu Nov 11 22:06:05 2021

In any event, the version that I uploaded yesterday is the correct version. If it's not working for you guys I don't know what to say. Chriss, you could recompile it and upload your version if it's different.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I downloaded the vfp2c32.fll version I posted above on 11/10 and used it on a current Windows 10 Professional 64-bit machine and VFP9 with SP2 + a couple other hot fixes. From the command window:

* I had downloaded vfp2c32.fll into c:\temp\test
CD \temp\test
SET LIBRARY TO vfp2c32.fll
ADirEx("adirex", "c:\*.*", 0, 0x100)  && 0x100 = Recurse to cursor

It took about three minutes to complete, and produced a cursor with ~1.5 million records. No Access Denied errors. It recorded one entry in c:\windows\csc\ -- "c:\Windows\CSC\v2.0.6\v2.0.6". When I try to access that folder with File Explorer in my current non-administrator user, I get a prompt for elevation, and then I also get access denied.

--
Rick C. Hodgin

 

[Chris Miller]

Rick always aid the new FLL is in the attachment of his postas, aka the files.engeneering link at the end of the post.

Rick, I made absolutely sure I downloaded that fll and named it vfp2c32.fll and that now works almost.

I can't get the full c:\ drive in a reasonable time and even in c:\Windows I didn't get through to the end, it seemed do to "nothing".

I used my sample code to create a directory c:\test\accessdenied and your FLL reported a file c:\test\accessdenied\accessdenied, which does not exist. The directory is empty. In comparison the usual VFP ADIR does list the directory name, when doing ADIR(laFiles,'c:\test\*.*','D') and it returns nothing if I try ADIR(laFiles,'c:\test\accessdenied\*.*'), which is what I expected as result, since the second version explicitly asks forr files within the "accessdenied" directory and even if there were some, it would not get them.

Maybe you test with such a directory and see why it reports the file inside the folder with the folder's name, I guess some code just adds the last file name found to the directory.
Edit: The FLL I compiled from source code does the same thing, reporting the directory name doubled.

Chriss

 

[Rick C. Hodgin]

Was I unclear that the github download was not working?

I apologize, Mike. I'll try to figure out how to add Releases to GitHub, and will post FLL binaries on that platform.

FWIW, I view source code management tools like git / GitHub as things to hold source code, not binaries. I've never had the mindset of storing binaries in git or on GitHub. I've done so for 3rd party libraries I included, so they would compile on other people's machines. But I have a website for the executable downloads.

In addition, I didn't know there were FLL files in the source code / subdirectories in Christian's repository.

I apologize for my ignorance / lack of knowledge.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Maybe you test with such a directory and see why it reports the file inside the folder with the folder's name, I guess some code just adds the last file name found to the directory.

Will do.

--
Rick C. Hodgin

 

[jamesrav]

I finally have something to contribute here. The pre-check routine ran slow (13 minutes) and the next attempt wasnt much better. So I tried the sys(2000) statement in place of Adir, and indeed that does read in the directory very fast( 1 second, although Adir was not bad at 5-10 seconds). But the cycling thru and Use of each table was not improved, and the next attempt was going equally slow, so I interrupted it. I then exited VFP 9, closed Chrome and everything else (but did not log off) and re-entered vfp 9. First attempt of the pre-check routine was also slow (but not 13 minutes) and then voila! the 2nd attempt zipped thru in 30 seconds. So that will be how I handle this in the future. If exiting VFP and then re-entering does not give a good result on the 2nd attempt I will post that here, but hopefully it doesnt happen. Thanks to all, and I will try to read at least some of this thread to see if I can pick up some useful knowledge for the future.

 

[Rick C. Hodgin]

Attached is the updated FLL which keeps the folder name from showing up twice on inaccessible folders.

Here's the [url

https://github.com/RickCHodgin/vfp2c32/commit/5fcd19b2ac4b1dcad2238931c0bb1a09f4239e9b

]GitHub commit[/url].
Here's the [url

https://github.com/RickCHodgin/vfp2c32/releases/tag/2.0.0.19-recursive

]2.0.0.19-recursive release[/url].

SET LIBRARY TO vfp2c32.fll
lcDir = "c:\temp\test\accessdenied"
IF NOT DIRECTORY(lcDir)
    MD (lcDir)
    RUN cacls "&lcDir" /e /c /d %username%
ENDIF
? ADirEx("c_Files", "c:\*.*", 0, 0x100)
LOCATE FOR "accessdenied" $ LOWER(Filename)
BROWSE LAST NOWAIT

--
Rick C. Hodgin

 

[Chris Miller]

Rick,

that works perfect, now. Thanks.

Chriss

 

[Rick C. Hodgin]

Third time's a charm.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Based on your treatment of me, Mike, I'll just let you be all to yourself from here on out.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

That is deliberately ignoring my request per your imagined feelings of the author. Chriss confirmed you could have done what I originally asked for, and not your ridiculous imaginary consideration. Only a religious person could make the claim to know what God wants, while having no clue. I won't wish you have a good life, because you're too inconsiderate to worry about a real person's request versus your imagination of someone else's request. Typically stupid like so much of humanity.

I don't know why you don't understand this. It's Christian's project. He has a horse in the race. In fact, it's his horse. I contacted him, but didn't hear back from him, so it may be perfectly fine with him for me to do whatever to his source code, and he might have issues. I don't know, so I err on the side of caution. In addition, I addressed your original request. I gave you a recursive function. You then asked that it be incorporated into the existing function. That's where I draw the line because I didn't want to be invasive to Christian's algorithm. Right, wrong, or indifferent, that was my thinking at that time. I have respect for Christian and his project, and I addressed your need within the confines of that respect.

In addition, you're a total jerk, Mike. I don't want to help you in the slightest to be honest. But I do you help you because I have skills you don't, and I'm a kind person. But you make it very hard to be a part of your life.

Every post I read from you is some slam against something. Nothing kind. Nothing helpful without also being hurtful.

You're a piece of work, sir.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

You know what, Mike ... I should've stuck to my guns and just left you to spew and hiss and hate and rail all you want to.

So, I'll go back to that policy now. Have a nice life. I truly do wish you well.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

Mike, my hand of friendship remains extended out to you.

Whenever you're ready to be civil, respectful, and helpful without also being hurtful ... I'll be here.

--
Rick C. Hodgin

 

[Rick C. Hodgin]

I have used some binaries that have been published on GitHub.com, primarily for Linux video drivers, for example, so my web browser could play HTML5 content, which it cannot do by the native installation. So, I had used binary downloads.

However, I had not equated that to a natural thing to do. In my mind, my use of git (not just GitHub, but git in general) has always been one of source code. I have a .gitignore file that lists out a host of files not to include from my project because I used it as a source code repository.

I don't know who Drew Speedie. You seem to think he's the end-all be-all, so I assume he's some at Microsoft? Or Google? Or some high coding place?

You seem to cast what YOU think on other people as though that's the norm. It's not always. There is common knowledge in various circles, and there is also learned knowledge separately which is common, but not due to being in the same circles.

I grew up on FoxPro, for example, coming from a Multi-user FoxBASE+ 2.1 and later dev environment. I literally learned nearly everything on my own. I studied manuals mostly, and I tried things. I didn't know about people like Tamar Granor, and that group she works with at Southwest Fox until something in the 2010s. So, I have lots of FoxPro knowledge, but it didn't come from common circles.

The same is true for my C/C++/assembly knowledge. I read manuals. I got the Intel IA-32 architecture manuals. When I wrote my OS back in the 90s, I literally did it from what those manuals said it should be, plus trial and error on an old Pentium-133 machine, but it wasn't from working with others or being parts of groups with shared, common knowledge. It was done in isolation because I had an interest in it, and with my brain such things just clicked. It was automatic for me to understand such things, especially in my 20s and 30s.

I tell you all of this to try and teach you that not everybody knows what you know, or thinks what you think, or places value on what you place value on. We are each unique people with our unique pasts and sets of abilities, experiences, and everything else. We often have common frameworks of knowledge, but not always.

My LITERAL mindset regarding GitHub was a place to publish source code. Even though I had used GitHub to download those binary HTML5 files for my browser, I never equated it to a place to publish binaries. I just didn't think in those terms or hook those two together. Since it's been suggested in this thread I realize it was an obvious oversight on my behalf, but it was a real thing nonetheless.

I bet someone has developed an AI or analyzer that can take published content and return a series of ratings on the tone, intent, friendliness, helpfulness, etc., of online posts based on word usage. I'd be very curious to see what that arbitrary algorithm would say about THE LOT of your posts here on Tek-Tips. I'm guessing it would score you very particularly.

In any event, I respect your abilities, Mike. You're obviously a great developer. But you are so far lacking in terms of interpersonal relationships and understanding people that I feel sorry for you.

BTW, I'll probably be banned for saying this, but it's not about religion. It's about salvation. I was an atheist not looking to be saved. I was looking to know what the Bible said explicitly so I could DEBUNK it. But in setting my sights truly on the truth of what it said, I found it. It was shocking to me that such a thing could happen, because as far as my former flesh-only mind knew, it was impossible. But nonetheless, here I am now almost 18 years later and the change is real.

I hope it happens to you because it really is something more than the former existence. I am different now than I was before. I am forever changed. It is a great sense of inner joy and peace and love and hope ongoing. I reach out to people to help them, even the unfriendlies, because I have a new nature on the inside. Even to this day I'm amazed sometimes at my reaction to people who are mean, rude, hostile toward me. It's not one of getting even, it's one of concern, outreach, prayer, and real world assistance where possible.

I don't know if this will mean anything to you or not, but there it is. I am a 52 year old man who, at the age of 34 almost 35, was transformed into the person you see today, the man who spent nonstop dedicated 30-60 hour weeks from July 12, 2012 through the time I got sick in 2016 working on giving people the next version of Visual FoxPro for free, including source code and non-restrictive licensing. I sought out help, but could not find it because of the source of the gift I was giving to people. It could've been something really great, and it's really quite amazing what I accomplished in my project over those years. I go back to it and work on it to this day and I'm amazed at all I did on it. It is something that deserved to see the light of day, but did not because of my source for why I was doing it.

You have many admirable traits, Mike, specifically in your abilities and knowledge. But you do need help in working with people, because people are what's important, not things, not abilities, not gifts, but actual real living people. We are the source of why this world exists. We are the source of why this universe exists. And we are to love and respect and show each other the proper regard as best we're able, always improving and refining ourselves as we go.

That's the teaching, and that's the thing you're railing against, though you ascribe something else to it to make it easier to disregard. If you ever take down those shields and investigate the truth, you'll find it's markedly different than you think it is. You'll find it's truly amazing in fact.

Peace, Mike.

--
Rick C. Hodgin