Project Compiler suddenly stopped including newer files 3

[Joe Crescenzi]

Hi Everybody,

I've been developing xBase programs since dBase II in the early 80s and all of a sudden I'm having the craziest problem, the project manager is skipping files on my most critical project and I have no idea why.

The strange this is that this is the same program I've been continuously developing since 1985, and I've compiled newer versions of it at least once a week without a single problem.

Now, when I compile the program, all the newer PRG procedures, and SCX forms are NOT in the EXE file... and are NOT in the Project File.

I'm wondering if anyone else has seen this problem before.

I assumed the project file was corrupt, so I created a new one and just added the main PRG file, expecting it to rebuild the whole project, but it didn't. In fact based on opening the project file directly as a table, the newer project file literally skipped nearly 600 files that otherwise were in the original, and it's basically saying in the error file that the files were not found. All the files are in the same directory as the project file.

I even tried compiling it on an entirely different computer, so it's not a corrupted version of VFP.

I'm starting to wonder if the compiler somehow hit some sort of limit to the number of files in either the directory or the project file itself?

There are over 4400 files in the directory, including about 1400 PRG files, 400 SCX files and a ton of report and other files.

My workaround so far was to add some of the missing files by hand to the project file, but I worry that I won't know which files are missing without either waiting for a customer to say "there's an error saying procedure xys is not found".

I'm thinking of writing a script to look at all the PRG and SCX filenames that are in the director, but not in the project file so I can know which ones to add, but that's not not the best option because there are other prgs and scx files that aren't actually called, so I'd need to start maintaining my own list of files that can be excluded.

This problem really has me confused. I've been developing code for 40 years ands it's like somebody suddenly telling me up is now down, or blue is now red. There's just no reason for the project to think files are missing when they're right in the same directory.

Any thoughts?

 

[Chris Miller]

The depth/width first approach of parsing code can be tested with a much smaller project, and I did this:

1. main prg:

dothings()

Procedure dothings
   Do program1
   Do program3
   Do program4
   Do program5

2. program1.prg:

Do program2

program3 and 5 are just RETURN and I didn't write program4, so a real missing program is involved now. To see whewther that stops analyzing mein.prg and not including program5.prg

Only main.prg is part of the PJX before building, then debugoutput says:
[pre]Build Project: Program main
Module: dothings
Analyzing: MAIN
Module: DOTHINGS
Analyzing: PROGRAM1
Analyzing: PROGRAM3
Analyzing: PROGRAM5
Analyzing: PROGRAM2
Build Application: Program main
Build Application: Program program1
Build Application: Program program3
Build Application: Program program5
Build Application: Program program2[/pre]

And, finally, using the pjx file as table I see the files added after main are program1,3,5,2 in that order, just as the debugout lists it. If depth first would apply program1 should be analyzed when its DO call is found and since it calls program2, program2 should be added to the project a second, but it only comes in last, because first all programs called in main.prg are added to the pjx, only after that's done program1 is analyzed and program2 is found to be called and therefore included.

A single experiment can never prove any assumptions about, but I can reproduce this and haven't found an instance where depth first parsing/processing happened.

Also, program4 not existing didn't make the build stop, program5 is still included into the project, even before program2, which comes in last as expected for breadth first processing, or simpler said analyzing one program after the other. It's also telling me there's no pack and sorting done in the pjx records, that's only done in the project manaager listing of programs.

None of the DO calls after a certain point were found, meaning it stopped looking at some point before reaching the end, which also means it doesn't just add all the direct referenced files before parsing subsequent files.

Well, I don't see that happening. There must be another reason why only 252 of about 1400 files are added.

That shouldn't stop you from trying what you do, but it may speak for a totally different reason stopping to include files and I hope you find it. Maybe you find something interesting in the debugout window, when you open it duringa a build.

Chriss

 

[Joe Crescenzi]

I can see in your example that it did add everything in main before diving into the called program2 was added.

There is one more wildcard in play in mine, the main.prg has several declared functions and classes defined at the bottom that are frequently used. The four missing DOs out of dozens that at the bottom of the last function in the PRG.

So a simplified view looks like this:

**
** Main.prg

**
do while .t.
   ** setup my environment with variables that needs to be global
   userid = ""
   username = ""
   do asklogin
   do form mymenu  && returns a variable called CHOICE, based on what they select from the menu 
   do case
      case Choice = "ADD-CUSTOMER"
          do addcustomer
      case choice = "BROWSE-SOMETHNIG"
          do browsesomething
      case choice = "EXIT"
          do signoff 
   endcase
enddo

** I'll define some classes like this so I can pass and return a city, state, county, and zip as a single object
** 
define class CSCZ as custom
   City = ""
   state = ""
   County = ""
   Zip = ""

   ** The class may have some functions too
   **
   function xyz
   parameters something
   do case
      case something="1"
         do something1
      case something="2"
         do something2
      case something="3"
         do something3
   endcase
   return something
enddefine

**
**
function SomethingINeverActuallyCallDirectly
   do prg1
   do prg2
   do prg3
   ** etc
   ** etc
   do prg21
   do prg22
   do prg23
   do prg24
   do prg25
endfunction

So the PJX will have all the DO and functions called from the main function (in main.prg), plus all the functions declared in the classes, and all of the DO Prg# calls except the last four... 22,23,24,25.

So, the last four are still a mystery.

In case you are curious why I have a function that is never called at the bottom of main that has nothing but 20+ DO calls, it's just a trick I've used for decades to force the compiler to bring in a bunch of stuff that is called by things like customizable reports (that are intentionally excluded from the project), I have have a DO FORM Artwork that has nothing but pictures on it, so the art gets embedded in the EXE for things like reports.

For example, I'll put a function like ShowPickupDays(customerID) in the box on a report. Since ShowPickupDays is only called by the FRX report file, the compiler doesn't know it exists. By putting all those report functions in that last function ensures they are included. It's worked that way for decades.

 

[Chris Miller]

Okay, but then what you experience is not what I experience about missing PRGs 22,23,24,25. In my experience main.prg is fully analyzed and all DO calls cause included prg files. And all that happens before any of these called programs (or scx/vcx, whatever) are analyzed themselves.

Your theory of something within the other files stopping the extraction of files to include into the project does ont happen in my experiment. I could try if it makes a difference to put some calls into a function, also if it's never called. I don't think VFP is that picky or clever, wheatever reason you assume behind picking the files it actually includes, because in my experience VFP even analyzes and compiles excluded files, the only thing the exclusoin controls is whether that file is embedded into the final app/exe or not. So there is nothing clever excluding files because code analysis shows they are in code calling them which is never called, there's no such analysis taking place.

I added a function in my main.prg sample:

dothings()

Procedure dothings
   Do program1
   Do program4
    
Function nevercalled()
   Do program3
   Do program5

Program 3 and 5 are still included. And I doubt a compile error in program3 would effect program5 to be included or not, as you can clearly see from the include order main.prg is fully processed before program3, so program5 is already in the pjx before program3 is compiled and could cause any problem.

It would be good to find a way program5 would not be included, so that you could find a reproducible reason. Your theory it has to do with order of analysis doesn't fit the experiments, so I strongly doubt it's that. There's always a chance that some more specific setup or configuration option influences it, but the pure fact that not all files claled from main.prg are included into the project points out a source of a problem outside of VFP, because all my experiments don't show a problem of suddenly stopping to include further files at some point. I even tried to expülicitly stop the bukild process through an actually not existing file, and even that didn't stop it.

I bellieve you get to such an incomplete project state, but the reason behind it seems to be outside of VFP itself.

Chriss

 

[Chris Miller]

I don't see you mentioning VFP verion. I know VFP9 can handle unlimited lines in a prg, but there was a limit in earlier versions. I don't reacp if it was 65000 lines or that only was a limit even VFP9 still documents for number of variables and arrays (not array elements, though). I think there was a 64kb limit for fxp files or prg files, you're not by chance hitting that limit?

Chriss

 

[Joe Crescenzi]

I'm definitely using the last VFP9 SP (9.00.0000.7423).

My latest experiment was pretty amazing in both expected and unexpected ways.

Since the main program only skipped four PRGs, I decided to just move those four DOs about 10 positions higher in the PRG to see if it would include them, and still skip whatever four are at the new bottom.

The result was simply nuts. The error file now has 72 undefined objects (in the main.prg), and the PJX dropped from 252 to 184 rows, a difference of 67.

The plot thickens.

Then I removed the 4 prgs... and it generated a PJX with 1374 rows... so apparently one of those prgs freaks out the build, and where it's located in the main determines just how deep it will crawl subbed files.

All four files compile without errors, so its not the code itself, but something not obvious.

Chances are it's just one of those four that confuses the compiler, so I'll just need to isolate one at a time to see which one is to blame. I'm super curious which one, and what in particular confuses the compiler so much that it decided to stop parsing the rest of the files without even a single error in the debuginfo window (there's nothing unusual there).

Once I know, I'll share the results.

What a ride.

 

[Chris Miller]

apparently one of those prgs freaks out the build

I agree, of course. I can't for good or bad think about what could achieve that. Because even drastic code like CLEAR ALL is just compiled, not executed during a build. The only thing I can think of stopping a build is when the build encounters a first not found file and pops up the Locate file dialog that asks you to Locate, Ignore, Ignore All and Cancel. But since you posted lengthy .err report files, you do use the "Ignore All" option.

One more straw to grasp: Do you make use of the IDE setting "save with end-of-file marker"?

Chriss

 

[Joe Crescenzi]

It actually isn't doing an ignore all. I like to see each error as it comes up rather than looking at the .err.

The .err for the 252 row projects is actually pretty short because it doesn't see about 1200 files.

I'll check into the EOF markers. I haven't had to deal with those in decades because as you remember FoxBase DOS used to have a bunch of Nulls at the end that I eventually removed as I'd come across them, but as I mentioned, perhaps an update to some of those third party editors and search / compare tools may have inserted something that VFP doesn't like in at least one file.

The strange thing is all 4 compile without errors, so it must potentially only exhibit this when crawling objects to build the project.

I'll be tied up today, but this is definitely something that should finally be solved today.

 

[Chris Miller]

It actually isn't doing an ignore all.

It? How do you then go through hundreds of errors, even more than 1000 missing files?
Do you CANCEL?

Regarding EOF marker: Helkp says it introduces CTRL+Z at the end of files and even if you turn this on I don't find that at all in the files, so I think you can forget that.

Chriss

 

[mJindrova]

Hmm,

Joe , do you use project hook or not?

mJindrova

 

[Joe Crescenzi]

It? How do you then go through hundreds of errors, even more than 1000 missing files?

I meant I. I normally don't click ignore all, so I can write down errors as they come along. Normally when I build from a live project, it doesn't find anything except for some reason it has trouble understanding two of my array names that it thinks are functions, even though I always explicitly declare them as external arrays just before using them.

In this case, it doesn't actually stop over 1000 times. Remember, when I build from an empty project, it only see the first 252 programs, so at that point the compiler has no idea those other 1200 or so files exists, so it never parses or looks for most of them because they're called by just the ones that were found.

 

[Joe Crescenzi]

Joe , do you use project hook or not?

No. Never tried it, but it would be funny after nearly solving this to discover there was a tool to help me find the answer.

 

[Chris Miller]

I meant I. I normally don't click ignore all, so I can write down errors as they come along.

Well, for once: After the builkd VFP will pop up the .err file, so there's no reason to note down one error after the other.
If you cancel, are you not awre that you are cancelling the build and thus cause the rest of the PRGs to not be processed and n wonder you don't get a complete project at all.

I mean, you already showed longer lists of undefined functions and not found files, they have to come from builds you did with ignore all, don't they?

I'd always recommend to ignore all unless you have to do with single files not found you can locate, but even that is better fixed in the project than during the build process. See how far you get and what's really finally missing when you build ignoring all errors. It doesn't mean they will not be reported, you get a full overview of problems if you build that way. And, of course, you get an incomplete project if you stop.

Chriss

 

[tomk3]

Hello,

strange.
Maybe you can watch the compiler run with process explorer which reports any fileactivity, maybe an access denied by an AV program,...
procmon (freeware, portable) from sysinternals/MS :

https://live.sysinternals.com/

regards
tom

 

[Chris Miller]

Remember, when I build from an empty project, it only see the first 252 programs, so at that point the compiler has no idea those other 1200 or so files exists

Don't you see the error in your line of thinking? When you stop build, of course VFP doesn't analyze all the further PRGs and doesn't incude all further files, not only the ones after a certain line in main.prg, every other file. You only get a complete overview of what's found and not found when you complete a build run by using the ignore all option when the first file/function/class/whatever is not found.

Chriss

 

[Joe Crescenzi]

Well, for once: After the builkd VFP will pop up the .err file, so there's no reason to note down one error after the other.
If you cancel, are you not awre that you are cancelling the build and thus cause the rest of the PRGs to not be processed and n wonder you don't get a complete project at all.

I mean, you already showed longer lists of undefined functions and not found files, they have to come from builds you did with ignore all, don't they?

I'd always recommend to ignore all unless you have to do with single files not found you can locate, but even that is better fixed in the project than during the build process. See how far you get and what's really finally missing when you build ignoring all errors. It doesn't mean they will not be reported, you get a full overview of problems if you build that way. And, of course, you get an incomplete project if you stop.

Chriss

In most cases with my live project, I get exactly two errors, just the two arrays that for some reason pop up in spite of declaring them as external arrays in the line just before they're being called, so I don't mind clicking ignore twice.

I never cancel.

Since I've been tinkering with the sandbox copy of my code, I always got the same 252 rows whether I ignored all or one by one. I tried it both ways, so my interaction was never a factor. What did turn out to matter was moving or removing the four last DO calls.

All four compile with zero errors, so it's going to be interesting to see why one of them chokes the compiler. As I mentioned, my code is tight and I'm the only developer and the only third party code that lives in this project is XFRX, but as I mentioned, I deleted it in the test folder with no change, so it's not an overlapping function or object name confusing the compiler.

So here's where things stand.

With those four files as the last four DOs, it always builds a project with only 252 records.
With those four files moved to an earlier position, it skips about 70 more files, and the project shrinks to about 180 records.
With those four files removed, it builds a project with nearly all 1400 files.
One of the four is just XFRX, so I can eliminate that since I tried it with it removed with the project being reduced by just one, itself.

The remaining three are super short programs that literally do virtually nothing important.

Proc./Func. __ID2ID2 - Undefined
Proc./Func. CHANGEACCOUNTID - Undefined
Proc./Func. FIXORPHANS - Undefined

__ID2ID@ just copies a string (ID) to a second variable (ID2) if it's left out (there's a search box with a start and end. If they only enter a start value without an END value, it makes them the same). It's probably only 10 lines of code, at most.

CHANGEACCOUNTID just takes two parameters to change an account ID from one value to another. It checks to be sure the account ID isn't in use, and changes all the child tables accordingly.

FIXORPHANS just makes sure that there are no records in secondary tables that have no parent client ID. Theoretically, it never needs to be called because the ChangeAccountID has every conceivable child update fully updated. I only include it in the system with a backdoor hidden menu option in case some of the clients who've been using the system 25+ years ago still have orphans before I fine tuned the changeaccountID code.

So as it stands one of those is the culprit and none look like anything unusual. I'll explore them tonight and I'd really love to know the answer.

 

[Joe Crescenzi]

Don't you see the error in your line of thinking? When you stop build, of course VFP doesn't analyze all the further PRGs and doesn't incude all further files, not only the ones after a certain line in main.prg, every other file. You only get a complete overview of what's found and not found when you complete a build run by using the ignore all option when the first file/function/class/whatever is not found.

I don't think you understood. I never stop the build. I always use Ignore and in all those sandbox starts, I did it once or twice with ignore one by one to see what it would do, then dozens of other times, I used Ignore All because I always got the 252 rows regardless of whether I hit ignore a few dozen times, or hit ignore all.

I always let it complete and it always generates an EXE (but with missing functions).

By removing the four DOs entirely, I do the same ignore or ignore all, but this time I got nearly all 1400.

The only difference is still the four files, and since one is XFRX, and I think I've already tried a version with JUST that removed from the main. To rule it out entirely, I also removed the XFRX.* from the sandbox, leaving just the remaining 3... all of which compile with zero errors when compiled individually (compile changeaccountid for example).

 

[Chris Miller]

all of which compile with zero errors when compiled individually

The major topic is not finding/includng files, not errors in code, isn't it?

Chriss

 

[Joe Crescenzi]

Exactly. None of the three have errors in the code. If I typed compile *.prg, there will be zero .err files, because none of the programs has compile errors.

The mystery is why one of those 3 programs is causing the compile to not include the majority of code.

Before all these tests, I used my real project file, and it was only missing the programs I created recently because the rest of the files were already in the project. When I add them by hand, I get a perfect EXE without any missing functions.

In summary:

By creating a new project with just the main, it always included just 252 files, and didn't parse the rest.

By moving the last four DOs, the project is even smaller, skipping about 70 more.

By removing the four DOs, I get virtually everything, 1377 rows in the project.

That's where I stand. I know those 3 files are super short and simple PRGs with nothing obvious in them. I'm sure it's just one that is the culprit, so I'll work out why and report it tonight.

 

[Chris Miller]

Okay, I'm stumped.

But going to mJindrova asking whether a project hook would cause this and your answer to that:

funny after nearly solving this to discover there was a tool to help me find the answer.

I'd be curious if you write a project hooks AfterBuild event maybe just MESSAGEBOX("afterbuild") whether you get there or something actually interrrupts the build.

Then also the procmon tom mentioned started before build could list all files the vfp9.exe looks for and whether it finds them or fails. Both things could tell more about what happens during the build and whether you actually get to after the build.

Not to forget AV might intervene, especially since you build an EXE.

Chriss

 

[Joe Crescenzi]

Not to forget AV might intervene, especially since you build an EXE.

I only use MS Defender, but anything is possible. It's still crazy to think one of those 3 innocent looking PRGs is the root cause when they do virtually nothing and compile flawlessly on their own.

But, facts are facts. Moving the 3 DOs 10 lines higher makes it even worse skipping 70 more, and just commenting out those 3 DOs yields an almost complete project of nearly 1400 records.