Project Compiler suddenly stopped including newer files 3

[Joe Crescenzi]

Hi Everybody,

I've been developing xBase programs since dBase II in the early 80s and all of a sudden I'm having the craziest problem, the project manager is skipping files on my most critical project and I have no idea why.

The strange this is that this is the same program I've been continuously developing since 1985, and I've compiled newer versions of it at least once a week without a single problem.

Now, when I compile the program, all the newer PRG procedures, and SCX forms are NOT in the EXE file... and are NOT in the Project File.

I'm wondering if anyone else has seen this problem before.

I assumed the project file was corrupt, so I created a new one and just added the main PRG file, expecting it to rebuild the whole project, but it didn't. In fact based on opening the project file directly as a table, the newer project file literally skipped nearly 600 files that otherwise were in the original, and it's basically saying in the error file that the files were not found. All the files are in the same directory as the project file.

I even tried compiling it on an entirely different computer, so it's not a corrupted version of VFP.

I'm starting to wonder if the compiler somehow hit some sort of limit to the number of files in either the directory or the project file itself?

There are over 4400 files in the directory, including about 1400 PRG files, 400 SCX files and a ton of report and other files.

My workaround so far was to add some of the missing files by hand to the project file, but I worry that I won't know which files are missing without either waiting for a customer to say "there's an error saying procedure xys is not found".

I'm thinking of writing a script to look at all the PRG and SCX filenames that are in the director, but not in the project file so I can know which ones to add, but that's not not the best option because there are other prgs and scx files that aren't actually called, so I'd need to start maintaining my own list of files that can be excluded.

This problem really has me confused. I've been developing code for 40 years ands it's like somebody suddenly telling me up is now down, or blue is now red. There's just no reason for the project to think files are missing when they're right in the same directory.

Any thoughts?

 

[Chris Miller]

Sounds strange, the first thing I'd try is build with the Recompile All Files option.

With compiling on another PC you've eleminated a lot of reasons already, like is a worn out hard drive failing, so the reason isn't VFP, nor Windows but just hardware getting flaky.

I don't know of a limit of project files and since a pjx - as you know and looked into it already - is just a free table the number of records and project size has no practical limit with the 2GB limit of VFP, most records refer to all the files - scx, frx, prgs, whatever, so it's not even limiting the overall project directory size, as long as the built EXE doesn't have 2GB size.

Is there anything special about any file? Are ole public classes invovled and/or ActiveX controls?

The strangest things I experienced once was compiler creating very small EXE sizes and what fixed it for me was starting VFP9.exe itself elevated to compile which not only automatically registers any COM servers but overall went smoother somehow.

Chriss

 

[TamarGranor]

The most common reason that files don't get pulled into a project is that they're referenced only indirectly, for example, using a macro or name reference. Could that be what's going on here?

Tamar

 

[Joe Crescenzi]

The most common reason that files don't get pulled into a project is that they're referenced only indirectly, for example, using a macro or name reference. Could that be what's going on here?

Tamar

Not in this particular case. These are just new modules, called directly from the main prg file with "Do SomeNewName". The only thing different here is that the procedure itself is new. The time consuming thing is that since it doesn't bring in the SomeNewName, it also doesn't parse that one for other missing files, so I keep adding the new PRG, then compiling and looking at the .ERR file for whatever it skips next. Rinse. Repeat.

The frustrating thing is remembering to check the .err file for missing parts or it actually generates an EXE with those parts missing, and naturally as I test my code in my dev environment, it runs without any problems because in it's uncompiled state, it can execute the PRG directly.

For now, I keep adding the new stuff after I look at the log to the project, but if I ever want to create a fresh project file, I'll need to add at least 600 program files one by one.

Sounds strange, the first thing I'd try is build with the Recompile All Files option.

I tried that first, and even creating an empty project with just the main prg. It seems to arbitrarily grab only some of the files and leaving out over 600 files, so I'm sticking to the old project for now and just adding the new stuff as it comes along.

With compiling on another PC you've eleminated a lot of reasons already, like is a worn out hard drive failing, so the reason isn't VFP, nor Windows but just hardware getting flaky.

I don't know of a limit of project files and since a pjx - as you know and looked into it already - is just a free table the number of records and project size has no practical limit with the 2GB limit of VFP, most records refer to all the files - scx, frx, prgs, whatever, so it's not even limiting the overall project directory size, as long as the built EXE doesn't have 2GB size.

Is there anything special about any file? Are ole public classes invovled and/or ActiveX controls?

Since VFP hasn't changed since 2009, I kept looking for reasons why Windows wouldn't find the files, even looking for patterns like whether the prg file had capital or lower case letters. I can't find a pattern.

The strangest things I experienced once was compiler creating very small EXE sizes and what fixed it for me was starting VFP9.exe itself elevated to compile which not only automatically registers any COM servers but overall went smoother somehow.
Chriss

I've tried changing the permissions on the folder and even copying the files to a FAT32 filesystem and back so it strips any ownership from the files, but I like that idea. Perhaps if I run it as an admin it could eliminate a rights problem entirely.

I'll experiment with a copy in a fresh directory and empty project file to see if it can find everything and report back in case somebody else stumbles on this thread.

PS. I'm not really new to these forums. I just can't find my credentials and the Forgot Password option didn't work.

 

[Chris Miller]

I can't promise it's a wonder cure, but to see what I mean, create a shortcut to VFP9.exe and look at its properties. In a tab "Shortcut" you find a button "Advanced" and the advanced properties allows making the elevated start the default:

Little remark on the notice text: Running elevated with this option is the deal with UAC to have a less globally unrestricted access to anything when running Windows under the Administrator account. You're asked to confirm the elevation everytime you start VFP9.exe and can only do so as user of the admin group. That justifies the notice Microsoft gives in this screenshot.

Any processes you start from within VFP9.exe don't inherit the elevated state, so it's less risky than running your whole Windows session as an admin. There's still a risk involved as the VFP9.exe runs elevated and you can potentially delete system files and other stuff you normally couldn't simply damage.

So it's still not fully protecting, mainly from yourself, though. A bit caution is good when you run the IDE elevated.

Chriss

 

[Joe Crescenzi]

For the record, I just tried two more fixes, including Chris Miller's suggestion of checking the Run as Administrator box and creating another empty project file and it still leaves out exactly the same 590 files. Meanwhile 590 is just the tip of the iceberg, because many of those missing 590 would've called another 500 or so programs. I also tried re-applying the ownership of the folder to my own account without any change.

So, as it stands I'm still able to build using the older project file, but I literally need to keep adding any new functions, forms, procedures, and even image files that I use for icons after I dig through the error file for what wasn't added.

It's kind of crazy when you can literally see the names of each missing function in the .err file, but every single file is right there in the folder.

What bugs me most is that I've been continuously developing and maintaining this code for nearly 40 years, and this problem only popped up within the last 60 or so days. Since it happens on multiple computers, including Windows 10 and 11, I'm completely puzzled why it suddenly started. I used to just run a quick PRG I called "MakeVovio" to copy everything to a staging folder, generate the EXE from the project, and build a distributable zip file I could deploy without needing to touch the project at all.

At least I know I can keep adding the missing stuff as it comes along, but it's hard to imagine what's going on under the hood.

 

[Joe Crescenzi]

I tried another test by copying everything over to an exFAT USB flash drive, which strips the permissions entirely.

As I was copying the files over, I was encouraged because I saw dozens of popups telling me that I would be losing file attributes, even on files from 2009 and earlier.

I thought for sure this would play some role, but in the end, when I tried to build a new project file, the PJX file had a record count of only 252 rows, but my live project file has over 1400 records and that grows every week or so as I continue to develop new features for my customers. Every attempt to build a new project file always results in only 252 rows in the PJX file, even after copying the whole folder to exFAT, FAT32, and other drives. Spooky.

It actually creates the EXE file, and the ERR file tells me there are 590 errors, for missing functions. This means that if I ever need to build a new project file, I would need to add those 590 files one by one to the project... THEN, it still would be another 600 or so files short, so I'll need to keep manually adding files until all 1400 files are there.

So, as it stands, I'll keep adding files one by one when I see new files pop up in the .err file when I compile, and making sure that if I ever need a new project based on this codebase, I'll just need to copy that older project file.

If anyone else with a large project has had similar problems, I'd love to see if they can identify the cause and share it here.

I'm tempted to build a Windows XP or Windows 7 Virtual PC just to see if it can build my project file from scratch without needing to tinker with it. In fact, that's exactly what I'll try next just to see if this has something to do with some recent Windows updates.

 

[mJindrova]

Joe,
you can upload pjt and pjx file for analysing.


mJindrova

 

[Gerrit Broekhuis]

Joe,

Did you already PACK the project (it’s a DBF file) before compiling?

Regards, Gerrit

 

[Joe Crescenzi]

Did you already PACK the project (it’s a DBF file) before compiling?
Regards, Gerrit

Yes. Right now, my primary PJX has about 1400 records, but the strangest thing is that if I create a brand new project file and add just the main PRG file, it will only have 252 records after building.

In the past, it would normally parse everything, and I'd have a perfect project file with all 1400 records for everything referenced directly or indirectly from the main prg without having to add any file other than the main PRG to build it.

When starting from an empty project file, the resulting tiny EXE file will report 590 errors in the .ERR file, which means the parser couldn't find hundreds of files that are literally in the same folder. Chances are, even if I added the 590 "missing" files one by one, it would still not parse them and find the remaining files.

Thankfully, I still can use the project file, but I find myself adding anything new I create one by one, which I never needed to do in the past.

Joe,
you can upload pjt and pjx file for analysing.
mJindrova

There's nothing unusual in the pjx file. It will do this even if I create a brand new project file and give it the main prg as a starting point.

 

[Chris Miller]

Did you know you can add files to a project with code?

To me it seems a major part of your function is never directly referenced nor indirectly by macro substitution. Do you use PRGs for functions you then call in other code siumply by PRG name? That does not suggest to the VFP build process that such files should be looked for and added into the pjx, the err file would just report missing functions.

So I think the problem is easily healed if you know you want to include all PRGs in one or more folders, just write a loop to add them in:

For lnI=1 To ADIR(laPRGs,'c:\some\folder\*.prg')
_vfp.ActiveProject.Files.Add('c:\some\folder\'+laPRGS[lnI,1])
EndFor lnI

Chriss

 

[Steve Meyerson]

A wild guess: Seems to me a last ditch effort in the worst case might be VFP itself or even the operating system, depending on when they were last installed. Were they all installed recently from the same source (possibly defective?)

Steve

 

[Joe Crescenzi]

To me it seems a major part of your function is never directly referenced nor indirectly by macro substitution. Do you use PRGs for functions you then call in other code siumply by PRG name? That does not suggest to the VFP build process that such files should be looked for and added into the pjx, the err file would just report missing functions.

Most of the missing files are PRGs that are simply run via "DO XYZ" without any macro substitution or special handling... Just "DO XYZ", where XYZ.PRG is a regular procedure with just that code.

Normally, if you have a main program that says "DO XYZ", the project builder will include XYZ.PRG automatically. The funny thing is in the main PRG, there are probably 30 "DO" calls to various features. Out of that list, 25 were pulled in, and there is no apparent reason for the other 5 to be missing.

You can tell it definitely parses the other 25 including some of the DO calls and function calls from those, which is how it ultimately ends up with 252 records in the project file... but it's nowhere near the 1400+ in the project file I've used for decades.

At least I've started to monitor the .ERR file and I know when to add files manually, but it's still a mystery. I wrote 100% of this code myself over the past 40 years, with two exceptions, I have XFRX and FoxyPreviewer source code in that folder. Now that I think of it, I recently downloaded newer versions of these, so I wonder if there are any preprocessor commands or issues that would interfere with the project building process. I think I'll tinker with that next. I know my own code builds flawlessly every time.

Did you know you can add files to a project with code?

So I think the problem is easily healed if you know you want to include all PRGs in one or more folders, just write a loop to add them in:

For lnI=1 To ADIR(laPRGs,'c:\some\folder\*.prg')
_vfp.ActiveProject.Files.Add('c:\some\folder\'+laPRGS[lnI,1])
EndFor lnI

Chriss

I've considered that, or creating a report that just tells me which prgs and scx files are in the folder and not in the PJX, but there are hundreds of other programs in there that are just for testing or other projects, so it would be easier to just keep adding files for now as they pop up in the .err.

The main problem right now is that I'm just plain curious why the project can no longer be built from scratch using the main prg. It's been a while since I've needed to do this, but it worked perfectly in the past, and now it doesn't. The programmer inside of me loves solving puzzles, but this one has me stumped.

A wild guess: Seems to me a last ditch effort in the worst case might be VFP itself or even the operating system, depending on when they were last installed. Were they all installed recently from the same source (possibly defective?)

Steve

I tried it on multiple computers using identical copies of the code on flash drives. It's just plain strange. One of the things I love about FoxPro is that it's been the same for years, so it should be predictable. I keep wondering if I stumbled on some sort of size limit, as if it asks Windows for a list of files and it only returns the first 1000 or so. I know it's not a nesting depth limit, because 5 of the missing files are just "DO XYZ" from the main PRG, so it should've included all the DO files first. There's one section that says "if (something)" followed by 5 different Do this, do that, and do theother, and I can see four of them in the project, but not the 5th. There's nothing different in that 5th DO. It's just a PRG just like the rest.

Very very strange.

 

[Chris Miller]

Notice it can play a role where you CD to (OR SET DEFAULT) when you build. Also if you SET PATH in your development environment.

So it can make a build difference whether you're in the right default folder while building.

Chriss

 

[Joe Crescenzi]

I always start with a CD into the directory with the source, and the path and default folders match the Home Directory that the project uses, which is why the rest of the project files are found without issue.

My next experiment is to remove XFRX and FoxyPreviewer from the folder. Those are the only third party programs, and they both were recently updated to the latest paid versions with source code. Perhaps one of them triggers is confusing the project manager.

 

[Gerrit Broekhuis]

Joe,

I can hardly consider XFRX to be causing this. I’ve used many versions for more than 10 years and nothing similar ever happened. I think Martina can confirm this.

Have you checked user rights and file attributes (NOT read only or hidden) for the directories and files you want to have in your project?

Regards, Gerrit

 

[Joe Crescenzi]

I can hardly consider XFRX to be causing this. I’ve used many versions for more than 10 years and nothing similar ever happened. I think Martina can confirm this.

I've used XFRX for quite a few years too, but I'm just looking for things that changed recently other than my own code and I know I renewed XFRX and FoxyPreviewer. In fact, in the .err file, XFRX is one of the countless files it skips.

Meanwhile, all the XFRX files are right there in the same folder as the rest of the source.

Have you checked user rights and file attributes (NOT read only or hidden) for the directories and files you want to have in your project?

That was one of the first things I did. As I mentioned, I even copied the whole folder to an exFAT formatted drive which has doesn't have NTFS permissions, and I made sure the entire whole folder wasn't marked read only. It resulted in exactly the same 252 records in the PJX instead of the normal 1400+.

 

[Chris Miller]

Would you care to run a simple test to check whether there's some kind of misconfiguration or limit?

Think of a new directory, specify it here and run:

#DEFINE HOMEDIR "C:\buildtest\"
MKDIR (HOMEDIR)
CD (HOMEDIR)
StrToFile("","main.prg",0)
For I = 1 to 2000
StrToFile("do program"+PADL(i,4,'0')+Chr(13)+Chr(10),"main.prg",1)
StrToFile("*","program"+PADL(i,4,'0')+".prg",0)
EndFor
Create Project test.pjx nowait
_vfp.activeproject.Files.add('main.prg')

and then build it.

The overall main.prg code is much simpler than yours, naturally, and it's all a very flat structure, but I end up with all 2000 PRGs added through the build.

One reason you might have problems is codepages. But before we'd go into that direction, this should not fail with whatever codepage you currently use. Well, and if it works, there are a few less assumptions to worry about, like the number of PRGs limit.

Chriss

 

[Joe Crescenzi]

Would you care to run a simple test to check whether there's some kind of misconfiguration or limit?
...
...
One reason you might have problems is codepages. But before we'd go into that direction, this should not fail with whatever codepage you currently use. Well, and if it works, there are a few less assumptions to worry about, like the number of PRGs limit.

Chriss

I like what you're suggesting and I'll give it a try. I'll also need to modify it to add all the SCX files too.

I'm 100% sure it will compile, and I'm sure nothing will be left out, but it will definitely also include about 300 unrelated or older PRGs that are not part of the real project. If nothing else, perhaps the compiler will suggest I remove them and it's a winner.

The thing that still boggles my mind is how it ended up happening in the first place. Normally, just adding the main program is all you need to build a complete project.

I'm intrigued with the idea that codepages are playing a role. You may be on to something. There was a time where I edited some PRGs using a couple of different editors and search and replace programs like Notepad++ and another one that had file compares for example.

I would use them to simplify refactoring variable and field names. Perhaps one of those programs saved the resulting files in another codepage without me knowing about it and FoxPro is looking for direct matches where it is trying to match a different character for the names of the functions or procedures. I'm going to tinker with that idea by taking the main program and re-saving it into notepad or something that lets me change the codepage, then re-building a new project. It normally can't find about 3 files. If it doesn't skip anything, we've got the answer and I'll just need to see which other files are also in the wrong codepage.

 

[Chris Miller]

I wonder if you confuse my earlier idea with the last test. This generates 2000 PRG files but only adds main.prg to a new project and then build should add the 2000 PRGs. And that tests whether that mechanism is brokwn for you, doesn't work on your computer or with your VFP version and may find problems

As you mention Notepad++: I'm not sure 100%, but I think by default it would use UTF8 and that has an overlap with ANSI in the first 128 characters, but then when something is special you get problems finally compiling that in VFP. But you already said

looking for patterns like whether the prg file had capital or lower case letters. I can't find a pattern.

If you write á in Notepad++ and save it as UTF8 encoded file VFPs editor would show it as á (at least in ANSI codepage 1252.
It's not the only problem with file contents, the file systems also don't just use ANSI for file names, and even something as simple as starting to use spaces in PRG file names, obviously DO my program.prg would fail and you should use DO "my program.prg" or even more generally DO ("my program.prg"). Because once you have the brackets - aka name expressoins - you can also start using concepts like DO (BASDÈDIR+lcPrefix+"hardcodedpart"+PADL(number,4,'0')) using expressions that combine constants, variables and functions, whatever, if only you get a string finally - in short name expressions.

Chriss