path or cache/memory issue in VFP9 1

[USMA99]

In a job shop setting I have a template folder structure with some starter programs in various locations, a fresh copy of the tree is used for each job. This means we have a prg in the corresponding folder under each job folder with the same name. Like:

P:\ABC\001\some.prg
P:\ABC\002\some.prg​

We use mapped network drives and relative pathing all over the place, and have for many years. As a habit, we also close files and set the default directory to start off a program. Just recently, VFP9 seems to be ‘remembering’ the prior same-named program run regardless of the default directory or search path. Like:

SET DEFA TO P:\ABC\001DO some && runs ok
SET DEFA TO P:\ABC\002DO some && runs the prg from ‘001’ folder​

This happens even after restarting fox or rebooting (first run is ok, second job picks up the first, and works vice-versa. I’ve played around with deleting/compiling anew each time as well as using CLOSE ALL, CLEAR ALL, and SYS(1104) in between … nothing works. No program editor windows are open. There is no other same-named program or compiled program in the search path, windows temp folder (or default directory) as shown by SYS(2019) or set(“path”).

I've removed (renamed) the previous job folder before running the second one with the thinking that it would surface a file not found error ... but instead it then behaves and runs the correct one from the default directory!

At this point, I literally have the first line of the program identifying it’s current directory and then the very next line of code identifies itself (001 or 002 respectively). It shows that it is indeed in the correct default directory just set, but then shows the wrong program is running, like:

Command Window:

SET DEFA TO P:\ABC\002? CURDIR() && shows \ABC\002DO some​

First two lines in prg:

? CURDIR()
?"002"​

Displays:

\ABC\002001​

I just don't understand how this could be. As stable as VFP9 is, this issue must ultimately tie in to Windows somehow, but I'm at a loss. What would cause a recently-run same-named program to run instead of the one in the current default directory?

Many thanks in advance

 

[Chris Miller]

See the help topic of CLEAR commands.

Visual FoxPro keeps a buffer of the most recently executed programs.

The question becomes why you never had problems so far.

I checked and using DO actually poses no problems despite of what you should expect from buffering executed programs. I can see a problem when calling the prg as a function without DO by calling prgname():

CLEAR 
CD GETENV("Temp")
MKDIR "foo"
MKDIR "bar"
STRTOFILE("? SYS(5)+SYS(2003), SET('default'),'hello'",".\foo\some.prg")
STRTOFILE("? SYS(5)+SYS(2003), SET('default'),'world'",".\bar\some.prg")
CD foo
DO some.prg && expect 'hello'
CD ..\bar
DO some.prg && expect 'world'
CD ..\foo
DO some.prg && expect 'hello'
CD ..\bar
DO some.prg && expect 'world'
CD ..\foo 
some() && expect 'hello'
CD ..\bar
some() && expect 'world'
CD ..\foo
some() && expect 'hello'
CD ..\bar 
some() && expect 'world'

The output is alternating between hello and world until some() is executed in \foo\, which should print 'hello', but all calls of some() without DO print 'world', which is the output of the last Do some.prg

The other thing I see on my Windows version is that Set('Default') only returns C:, I expect the full path of the current directory. I would have recommended Do sys(5)+sys(2003)+"some.prg", but Do some.prg seems to work all right even with buffering of executed programs.

So further experimenting could try to DO some without .prg extension to see if that uses the program buffer like some() or what Do some.fxp does, provided the fxp exists. Then I'd like to know whether the bug is just about SET('Default') only returning Sys(5) and not Sys(2003) and what causes that.

Chriss

 

[USMA99]

Many thanks, there's a lot of good stuff there. In following your reference to the CLEAR command in help, I found this also:

CLEAR ALL does not release system variables, and does not clear the compiled program buffer. Use CLEAR PROGRAM to clear the compiled program buffer.

and further:

PROGRAM
Clears the compiled program buffer. Visual FoxPro keeps a buffer of the most recently executed programs. In rare cases, Visual FoxPro might not recognize changes made to program files on disk. CLEAR PROGRAM forces Visual FoxPro to read the programs from disk, rather than from the program buffer. The most common reason Visual FoxPro might not recognize changes made to program files is your using an external or terminate-and-stay-resident (TSR) editor to modify a program file. With this exception, you should not have to use CLEAR PROGRAM.

So, I've been using CLEAR ALL incorrectly since the 90s, thinking that it, well, cleared all. Using CLEAR PROGRAM like this resolves the issue when not using the .prg, though I can see how your explicit use of the extension .prg would cause VFP to use the file rather than the buffer. Adding CLEAR PROGRAM to your example does do the trick:

CD foo
some() && expect 'hello'
CD ..\bar
some() && expect 'world'

hello
hello
​

CD foo
some() && expect 'hello'
CD ..\bar
CLEAR PROGRAM
some() && expect 'world'

hello
world​

For me, adding CLEAR PROGRAM to start my programs isn't a bad deal, no worse that CLEAR ALL in any case.

As to why "in rare cases, Visual FoxPro might not recognize changes made to program files" and why that's being noticed only now, it certainly could be windows related, but the help suggests with what the text is edited is the likely culprit. I do indeed sometimes use the built-in editor, sometimes a 3rd party text editor, and also throw in the fact that don't often run same-named programs close in sequence. That said, you used STRTOFILE to create your programs in the example ... and still experienced the issue. This isn't what the help would suggest should be, but it is.

As to SET('Default') providing the drive letter only, that happens to me also, but I'd agree the help suggests it should be otherwise. This may have always been the case; I use Sys(5) & Sys(2003) (or FULLPATH()) whenever I wish to capture the current directory for later use; perhaps I began doing that because SET('Default') doesn't work, I no no longer recall. Why this doesn't work would be interesting, perhaps warranting another thread.

Thanks for having taken the time, it was very helpful.

 

[Chris Miller]

You used STRTOFILE to create your programs in the example ... and still experienced the issue.

It's easy for VFP to see a compilation is necessary the first time, as no FXP file exists, only the PRG source code.

What surprised me was already that Do some.prg after CD to the other folder did not take the buffered program. It was only after switching to the use of some( instead of DO some, that the program cache was used. So the help isn't exact about telling when and why program cache is in effect.

I know CLEAR PROGRAM SOME would have helped. By the way, CLEAR PROGRAM SOME, not just CLEARPROGRAM, though I guess CLEAR PROGRAM without name means clearing all programs from the buffer.

I'm pretty sure SET('DEFAULT') usually gives a full path and not just the drive. SO I actually expect that that's the reason for reusing the buffer wrongly and that this changed due to some API change of a Windows update, that indirectly affected that.

On the other side, if you always DO your PRGs and not call them with the function call syntax that looks more common from the perspective of other programming languages, you could be fine without CLEAR PROGRAM. I personally prefer the function syntax - that is prgname(parameters). I could live with DO instead, but calling PRGs with parameters by DO some.prg with parameters is so inconsistent with any other function calls, like calling native functions with function(), that I'd like to avoid needing it to work around that bug.

Good point to say that calls with or without .PRG file extension might make the difference, I'll make some more experiments about that.

If I remember correctly it also depends whether a PRG is a standalone file, as in my case, or part of a PJX. VFP keeps some meta information in the PJX file itself about PRG. But I don't remember what exactly. Last compilation can be readd from the last file write time of the FXP, last edit is the last file write time of the PRG, so that doesn't need PJX meta data about a PRG. Maybe it was only about PRGs that contain class definitions.

Chriss

 

[Chris Miller]

I made this experiment:

CLEAR 
CD GETENV("Temp")
MKDIR "foo"
MKDIR "bar"
STRTOFILE("? SYS(5)+SYS(2003), SET('default'),'hello'",".\foo\some.prg")
STRTOFILE("? SYS(5)+SYS(2003), SET('default'),'world'",".\bar\some.prg")
CD foo
DO some && expect 'hello'
CD ..\bar
? '----'
DO some && expect 'world'
DO some.prg
DO some
? '----'
CD ..\foo
DO some && expect 'hello'
CD ..\bar
DO some && expect 'world'
CD ..\foo 
some() && expect 'hello'
CD ..\bar
some() && expect 'world'
CD ..\foo
some() && expect 'hello'
CD ..\bar 
some() && expect 'world'

So it shows the difference using the file extensions prg vs not using it. Also Do some.fxp first time in the \bar\ subfolder triggers compilation of the prg and returns 'world'. So Do some.fxp with no existing fxp does neither cause a file not found nor a lookup of "some" in the program buffer, it looks for the PRG, compiles and executes it. I'm stressing this, as actually that's what you'd program if you're aiming for a consistent execution of object code. The downside is that this file distinction of prg ans fxp only exists for PRGs. Code in class libraries is in the memo file of a vcx, in the vct, but so is the object code, both are in the vct file. In Newobject you will refer to the vcx file, though.

Interestingly, the help of Newobject states about cModule:

When possible, specify the compiled program name (.fxp) when a class is stored in a program file (.prg). This ensures that the class is loaded from the proper class definition file

That's not explicit about the reason, but might point to when the program buffer has priority over the file: When you don't specify a file extension and leave oit to VFP to add the approriate one, as is usual habit for USE of DBFs without specifying the dbf extension.

The question becomes: Did you start dropping file extensions .prg and/or .fxp file in recent code? Becasue it seems you only get in trouble with the program buffer, if you do that. Despite how SET'DEFAULT') is broken or not.

Maybe it's best to get into the habit of being more explicit about what you mean, i.e. specifying file extensions, even though automatisms of VFP would get the code also without them.

And one step back on the more general topic of caches of code or data: It's not become a proponent of emptying caches to get what you want to have, caches are fine, you just have to program supportive of what you know about the failures of VFPs inner working on choosing the cache and when it should be hindered to do so vs when it's not just "fine" but actually a very vital way of getting closer to the performance of compiled code in the case of fxp and vct or of data already in RAM in the case of data. The downside of that is, that you can't rely on automatisms about this. That becomes obvious here. Fixing a bug about SET('Default') might not fix the whole topic of imperfect cache usage of VFP.

Chriss

 

[USMA99]

Mike - While I don't know what prior discourse on modularity you're referring to, I wouldn't disagree with you ... it is definitely better form to never call 2 pieces of code the same thing. Your suggestion of naming the program by incorporating the job# was a countermeasure that occurred to me, and in fact is something I already do as a practice elsewhere such as naming tables, etc. Most files created in these jobs are program-driven and done just like this.

Chris - I didn't change practice recently, I sometimes use DO WITH or other times use perenthesis with parameters, but without parameters I have never included the extension or parens and simply say DO basename. I agree awareness is the key takeaway and won't constantly empty the cache going forward. For these few certain programs that currently have the same name, they're infrequently called and not something with many calls where I'd lose any performance, and so clearing the cache is a viable option though I'll likely work toward getting rid of duplicated/ambiguous names altogether.

Boiling it down, you're both essentially driving toward being more specific. There are perhaps two reasons I hadn't been specific. First was just ignorance of how this works, thinking that the search path governed this differently than it actually does. The other reason though is that specificity does bear a small penalty, and the ROI on form versus practical application doesn't always favor good form, something that happens a little more in a job shop than in a dev situation. For example, if when typing a program hundreds of thousands of times over decades without an extension worked perfectly (which we now know it does not), it would always be better (less labor) than emitting the extra keystrokes for 'fxp' or 'prg'. In the file naming case, it is the way it is because a non-technical person can take a template folder structure, copy and rename it, change any folders/files they'd like and everything is good to go ... without the need for them to be aware of program files or rename them and also precludes the need to crystallize the folder structure and build an app to handle these tasks. Perhaps this new issue renders this obsolete, but this is why it is as it is.

As to exactly why or when this began to surface, it is difficult to say since several things mask the issue:
[ul]
[li]99% of the programs we have are in a shared location in the search path rather than replicated as they were in the above problem case; I've been blissfully ignorant of how the program buffer truly works and just enjoy it's efficiency, and since the program names are unique there hasn't been an issue.[/li]
[li]The same-named programs often run further apart in time, or even on different workstations, making it rare that the cache still bears code with the same identifier[/li]
[li]No doubt some of the times we thought "Hmm, thought I had changed the default directory to my current job" were in fact this issue and not forgetfulness. Since the folders bear the same structure it can be easy to miss this at first. Unless that person toggled immediately between two jobs again, at same exact stage (running the same same-named program) the issue wouldn't recur. My suspicion was only raised due to a rare need to toggle back and forth between two related projects in sequence.[/li]
[li]This issue occurs extremely infrequently, so any changes in the environment (here's looking at you, windows) would be difficult to correlate in time to this problem.[/li]
[/ul]


For someone coming across this thread later as VFP continues to age (gracefully), this issue with SET('DEFAULT') being broken is a key loose end:

I'm pretty sure SET('DEFAULT') usually gives a full path and not just the drive. SO I actually expect that that's the reason for reusing the buffer wrongly and that this changed due to some API change of a Windows update, that indirectly affected that.

Thanks again for everyone's input, all set here

 

[Chris Miller]

Okay, Mike.

overlooking several things here, but now I'd be interested what you think is the case then?
If this construct, as bad as it is, worked so far.

Do you really mean to say that it never worked, instead?

If SET('DEFAULT') only returns the drive, the culprit may not be Windows but some other VFP setting that effects this. And the connection to the program cache isn't given, as long as you assume the program cache only works based on the stem name of the prg, not making a difference of some.prg in pathA and pathB.

I conclude from the concept working before, that the program cache made that distinction, so you can have two some.prg in the cache, when they are in two places. Otherwise you just have to state that this whole construct never worked. And that's not even pointing to me, then, make of it what you like. VFP would internally perhaps not use SET('DEFAULT') anyway, to see how a call of DO some.prg or some() has to be resolved from the cache, but I just observed there is something buggy going on in that direction. And that might effect how VFP handles its own cache, no matter if SET('DEFAULT') is directly involved or not.

It's yet to be investigated why the behavior changed. I don't bet 100% on a windows change, as usually VFP only depends very little on windows, it's using memory management, the NTFS file system and windows forms, graphically gdiplus, and that's all, essentially. Pardon me, if I forgot a windows related dependency of VFP, I didn't mention process and threads, because that can't be avoided for anything running on Windows. Still I don't see why there isn't something worth looking into, too, despite the fact you could handle the problem with a solution that doesn't need a distinction between two same name prgs.

You seem to not have looked into how far things do work despite of same name, still, but skipped that part of sample code. In short with a model of the cache in mind, that does only distinguish by a main stem name, the call of some.prg after CDing away fromm the foo to the bar directory would already not work, Do some.prg would then always and only print "hello" as the prg printing "world" would never become active.

I don't know how what you'd predict how the program cache or buffer works, but as you seem to know so much more about it, I think not pointing out what our or just my ignorance is, is just due to your recent decision only select people deserve your advice.

That is indeed your decision and I'm neutral to that.

Chriss

 

[USMA99]

Apparently I'm missing some historical context. And that's ok.

Mike,
It would be beneficial to the greater good to fully pin down how the program cache works and determine and/or resolve SET('DEFAULT'). If the latter isn't windows, then why not? Also, I don't think this thread subject would be a likely place someone looking for a solution to SET('DEFAULT') would ever find it. I will open a new thread for that if you care to show it isn't windows or otherwise slay it for posterity.


This discourse is getting above my pay grade, but there are two I/O things I've seen, which may or may not have any bearing on program cache/search paths and be worth mentioning. I've seen them only in recent years on things that worked for a long time, which is what makes me suspect changes in the windows environment:
[ul]
[li]Once in a great while a COPY FILE will fail and we cannot find a reason for a file lock. No AV running, no backup running, no other user accessing the file, same user running the app, same PC, same mapped drives, same years-old code ... seems to eventually go away on it's own on a re-run, but not always on a first try.[/li]
[li]Even more rarely, a DELETE FILE will fail. This give a message, something about a windows path containing 'SysWow', which I'll have to note more specifically on the next occurrence.[/li]
[/ul]