Project Compiler suddenly stopped including newer files 3

[Joe Crescenzi]

Hi Everybody,

I've been developing xBase programs since dBase II in the early 80s and all of a sudden I'm having the craziest problem, the project manager is skipping files on my most critical project and I have no idea why.

The strange this is that this is the same program I've been continuously developing since 1985, and I've compiled newer versions of it at least once a week without a single problem.

Now, when I compile the program, all the newer PRG procedures, and SCX forms are NOT in the EXE file... and are NOT in the Project File.

I'm wondering if anyone else has seen this problem before.

I assumed the project file was corrupt, so I created a new one and just added the main PRG file, expecting it to rebuild the whole project, but it didn't. In fact based on opening the project file directly as a table, the newer project file literally skipped nearly 600 files that otherwise were in the original, and it's basically saying in the error file that the files were not found. All the files are in the same directory as the project file.

I even tried compiling it on an entirely different computer, so it's not a corrupted version of VFP.

I'm starting to wonder if the compiler somehow hit some sort of limit to the number of files in either the directory or the project file itself?

There are over 4400 files in the directory, including about 1400 PRG files, 400 SCX files and a ton of report and other files.

My workaround so far was to add some of the missing files by hand to the project file, but I worry that I won't know which files are missing without either waiting for a customer to say "there's an error saying procedure xys is not found".

I'm thinking of writing a script to look at all the PRG and SCX filenames that are in the director, but not in the project file so I can know which ones to add, but that's not not the best option because there are other prgs and scx files that aren't actually called, so I'd need to start maintaining my own list of files that can be excluded.

This problem really has me confused. I've been developing code for 40 years ands it's like somebody suddenly telling me up is now down, or blue is now red. There's just no reason for the project to think files are missing when they're right in the same directory.

Any thoughts?

 

[Joe Crescenzi]

I didn't read your script fully, but now that I do, I see exactly what you were thinking. You are creating an entire directory of 2001 program files with just a comment that does nothing, plus one main program that calls all 2000. That's a great experiment.

As I mentioned, I reference XFRX in the main program and that was listed as not found too and I never touched or edited that one.

I keep wondering about the codepage, but then again, it's not just skipping PRG files. It misses class libraries, forms, and even image files like PNG.

Here's an except from the 590 skipped files in the .ERR file. Keep in mind 100% of these files are in the same directory as the main PRG and all 252 files that do get included in the project rebuild. As you can see it's a mixture of just about every type of file.

Program f:\cartpro.new\cartpro.fxp has the following errors:
    Proc./Func. XFRX2 - Undefined
    Proc./Func. __ID2ID2 - Undefined
    Proc./Func. FIXORPHANS - Undefined
    Proc./Func. CHANGEACCOUNTID - Undefined

Program f:\cartpro.new\tce_lib.prg has the following errors:
    Proc./Func. US_OPEN - Undefined
    Proc./Func. DB_CLOSE - Undefined
    Proc./Func. VL_OPEN - Undefined
    Proc./Func. WDROP - Undefined
    Proc./Func. PAGEBRK - Undefined

Program f:\cartpro.new\logon.prg has the following errors:
    Proc./Func. ASKPW - Undefined

Program f:\cartpro.new\snoop.prg has the following errors:
    Proc./Func. WDROP - Undefined

Program f:\cartpro.new\fleetmenu.prg has the following errors:
    Form FLEETMENU - Undefined
    Proc./Func. FLEETMANAGE - Undefined
    Proc./Func. FLEETBROWSE - Undefined
    Form FLEETDAILYCHECK - Undefined
    Form FLEETMANAGER - Undefined
    Proc./Func. PICKFLEETCODE - Undefined
    Unknown MYGUID - Undefined
    Form FLEETEDIT - Undefined
    Proc./Func. FLEETREPORTS - Undefined

Program f:\cartpro.new\scalemenu.prg has the following errors:
    Form SCALEMENU - Undefined
    Unknown GEN_TSID - Undefined
    Form EDITSCALETICKET-V5 - Undefined
    Form EDITSCALETICKET-BASIC-V4 - Undefined
    Form EDITSCALETICKET-V2-GAETA - Undefined
    Proc./Func. BLANKSCALETICKETS - Undefined
    Unknown PICKSCALETICKET - Undefined
    Form SCALETEMPLATEBROWSER - Undefined
    Form TARESEDITOR - Undefined
    Form FINDSCALETICKET - Undefined
    Proc./Func. REPORT2PRINT - Undefined
    Proc./Func. SCALEREPORTS - Undefined
    Form HAULERMANAGER - Undefined
    Proc./Func. SCALE2PENDING_MANY - Undefined
    Proc./Func. SCALE2INVOICE_MANY - Undefined
    Proc./Func. SCL_STAT - Undefined

Form f:\cartpro.new\routemanager4.scx has the following errors:
    Proc./Func. REPORT2PRINT - Undefined
    Visual Class Library VOVIOMASTERCLASSES - Undefined
    Form ASKROUTEREPORTSTYLE - Undefined
    Proc./Func. REP115 - Undefined
    Unknown DAY2TXT - Undefined
    Unknown NIGHT2TEXT - Undefined
    Form SURVEYCONFIRMPOST - Undefined
    Proc./Func. QSURVEY - Undefined
    Form ASKROUTEEXPORTOPTIONS - Undefined
    Unknown CLEANFN - Undefined
    Unknown L_O_M - Undefined
    Unknown DAYS1E - Undefined
    Visual Class Library SYSTEM2CLASSES - Undefined
    Visual Class Library VOVIOEXTRAS - Undefined
    Form QUICKDISPATCH - Undefined
    File PRINTER-24X24.PNG - Undefined
    Unknown PICKVEH - Undefined
    Form ADDCLIENTTOROUTE - Undefined
    Unknown NEWSORT - Undefined
    Unknown NEWNUM - Undefined
    File G.PNG - Undefined
    Visual Class Library VOVIOBASECLASSES - Undefined

 

[Joe Crescenzi]

I just ran the test program and it created a test project with exactly 2002 records, as it should.

Still wondering why it only adds 252 records out of over 1400 when I try to build a fresh project from scratch. I'm going to find a way to tinker with the codepage on the main program to see if it picks up any additional files. If I get anything higher than 252, I'll try changing codepages elsewhere too.

 

[Chris Miller]

What about the pattern that all addressed errors are about the directory cartpro.new?

What happens, if you remove the dot in that directory name and instead call it cartpro_new? And rename all references, of course.
Just checked, this form of directory name does not cause a problem. But, of course, if your new PRGs use old PRGs and address them as if they were in the same directory, that will cause missing procedure/function errors, wouldn't it?

Chriss

 

[Chris Miller]

By chance: is cartpro.new a zip file and not a directory?

Chriss

 

[Gerrit Broekhuis]

Joe,

Another wild guess: do you have enough headroom in the dbf files in the user folder \AppData\Roaming\Microsoft\Visual FoxPro 9.

I recently had an error because the file refdef.dbf had become too large (FPT > 2GB).

You may check the dbf's in this folder to be sure.

Regards, Gerrit

 

[Joe Crescenzi]

The directory is just a complete copy of the newest source code directory. As I mentioned, I'm doing all my testing on a complete copy on the exFAT formatted drive so I can tinker with it in a sandbox, and so that I can rule out any NTFS permissions, because FAT32 and exFAT don't support ownership.

All the files are in the same directory, and there are no references in any of the code to a path.

My last experiment was particularly fun. I just wrote a newmain.prg that had nothing but the same four DO lines that were note included in the real main to see if it would still exclude them. Strangely, it did include them, and even started to parse the references in them, which still eventually led to a long list of different files that are being excluded.

I have to say this is the strangest thing I've ever encountered in my 45+ years of writing code. I'm glad I have a workaround by adding files by hand, but it still doesn't make sense. I should always be able to just create an empty project, add the main prg or scx, then let it rebuild the project. There's nothing different about the PRG, SCX, VSC, or PNG files that are being excluded.

The one thing that somehow still seemed possible is the codepages, but that's proving tricky to confirm. I edited my main.prg using notepad and saved it as ANSI and got the same result. Of the four skipped files in that prg, three of them are called back to back with no code in between. I looked at the codepages in the live and generated projects and everything says 1252, except the kinds of files that have none like vcx, png, etc.

An idea just hit me. I'm not sure of the exact method the project manager uses to add files to the project, but all the files skipped in main are referenced as a DO at the very bottom of main. Perhaps the exact sequence that it adds files is a factor. I noticed that the records in a PJX file are not alphabetic, so the question is what order are they added when building from scratch and how does it play a role, if any.

When looking at the PJX of any of the fresh 252 row versions, it seems to flow like a tree, where the rows flow in something close to the order they were recursively found, with the last rows representing objects found in subsequent files, but clearly not including any of the DO calls made at the tail end of the main, which means it doesn't add all the references in main first, followed by a chain of subsequent parsed files, or those DO calls in main would absolutely be found.

This leads me to the conclusion that somehow there's potentially something that interrupts the parser. I checked row 252, it's just a simple text formatting function with no child references, so it wasn't that one. Perhaps whatever file it would've looked at as #253 stops the normal flow. Finding that will be a challenge, but it's the best guess right now.

 

[Joe Crescenzi]

By chance: is cartpro.new a zip file and not a directory?
Chriss

Just a plain folder.

Another wild guess: do you have enough headroom in the dbf files in the user folder \AppData\Roaming\Microsoft\Visual FoxPro 9.

I recently had an error because the file refdef.dbf had become too large (FPT > 2GB).

You may check the dbf's in this folder to be sure.

Regards, Gerrit

My dev machine has a tiny RefDef, about 15,000 rows.

It also does this on a PC with a freshly installed copy of Windows and VFP.

 

[Chris Miller]

This leads me to the conclusion that somehow there's potentially something that interrupts the parser. I checked row 252, it's just a simple text formatting function with no child references

What do you mean by that? Is there a function definition in main.prg and you have DO calls after that? Those DO calls can never run and thus are not parsed by building, indeed.

The firt line in a prg that defines a procedure, function or class ends the code section before it that has potential code that can run when you DO the PRG, the code in procedures, functions and classes runs when you call them or instanciate a class, but any single command outside a procedure, function or class after such definitions is regarded as non existant by the compiler, how would you reach it, there is no GOTO linenumber in VFP, it's not BASIC.

Chriss

 

[JackTheC]

I had the same thing after I included a .h file with #Define instruction in .prgs (with #include command) and forms (through menu).
After I removed these all was back to normal. This problem occurred in several of my projects.
I couldn't reproduce in a simple example so there might be a combination with another thingy.
Question: did you include a file in a form? (Modify form, menu Form, Include file)

 

[Joe Crescenzi]

No. That's not what I meant.

I'm saying that those were just DO calls that coincidentally are just the last DO calls within the file, and are part of the same procedure as those that did come through.

Do calls higher in the main were found. None of the DO calls after a certain point were found, meaning it stopped looking at some point before reaching the end, which also means it doesn't just add all the direct referenced files before parsing subsequent files.

It starts at the top of main, finds a new file, parses that and its subsequent file references, then goes back to main, and repeats with the next (new) object.

If something in the middle somehow fails (for reasons still unknown), it never finishes adding the referenced files at the bottom. If it did, every simple DO within the main would be in the projec first, followed any objects called from those objects, but based on the sequence in the PJX, it looks like it adds them to the project in order found as it branches out.

Example.

PROCEDURE main
   DO ABC
   DO DEF
   DO GHI
   DO JKL
   DO MNO
ENDPROC

If the parser just adds objects in main first the project rows would be:

MAIN
ABC
DEF
GHI
JKL
MNO

In that order.

What I'm seeing is more like:
MAIN
ABC
XYZ (a procedure potentially called by ABC)
PDQ (a procedure potentially called by XYZ)
DEF
GHI
(other procedures called by GHI)
... etc.

BUT... NO JKL, or MNO.

So something in the middle must be stopping it from adding anything past a certain point in main. The trick is finding which file is breaking the parsing and understanding why it happens.

As I said, my workaround lets me deliver updates, but my curiosity leads me to know why. To do this, I think I'll keep tinkering with variations. I may even resurrect an old DOS technique using the legendary "FOXBIND" to put all the prgs into one big procedure file and seeing if it compiles. In the old FoxBase / DOS days, I would foxbind all my prgs into one file I called overlay.FP, then compile it as overlay.fxp and use SET PROCEDURE TO OVERLAY at the top of my main. Back then you didn't compile to an EXE, but ran your programs with the runtime at the command prompt from a batch file that said "FOXR MAIN".

This brings back other memories, such as having MFoxPro for the "Multi-User Version".

 

[Chris Miller]

I see.

The trick is finding which file is breaking the parsing and understanding why it happens.

I agree, that's the bet lead you have.

I'd wonder if you have any double file names, but that would only work with multiple folders of files.

Chriss

 

[Joe Crescenzi]

I had the same thing after I included a .h file with #Define instruction in .prgs and forms.
After I removed these all was back to normal.
I couldn't reproduce in a simple example so there might be another thingy.
Question: did you include a file in a form? (Modify form, menu Form, Include file)

No, but you may have found the answer.

I don't use any .h files, but I recently updated to newer paid versions of FoxyPreviewer and XFRX and I purchased them with source code so I pull in the PRG versions, and as far I know XFRX has some .H files. I'm going to give it a try in a copy of my folder with XFRX source code removed or reference it as an external procedure so it doesn't try looking for it.

 

[Chris Miller]

I don't see a major advantage of pulling in the FoxyPreviewr source code. I know they suggest that in some circumstances, but I would rather just use the app.

Chriss

 

[Joe Crescenzi]

Just checked. I don't have FoxyPreviewer in source, and it's only called in some test modules. I still use XFRX as my main PDF tool.

I picked up FoxyPreviewer for testing only after learning that they were starting to fix some of its older bugs, especially with 4K and higher screen DPI, which made the old version useless.

That leaves just XFRX, which is definitely being pulled in as a PRG, and that calls some .h files. If the compiler struggles with certain .h files, making it an external reference should do the trick.

 

[Joe Crescenzi]

For the record. I copied everything to a new folder again, then deleted XFRX.* so it couldn't possibly try to find it or any of the .H files, and the only difference is that the PJX has 251 rows instead of 252. The only row that is different is just the lack of XFRX in the project.

The mystery continues.

That said, I'm sure there's at least one module that confuses the builder. I'll try using some old fashioned document generators to see if there's some sort of oddity, like defining a function at the bottom of a procedure file that is also a standalone PRG, or a class name that's defined in two places. There's 40 years of code in there, but I only noticed it missing files a few months ago, so I'll focus on modules with newer date stamps and work backwards.

I'll report any findings when I work it out in case anyone runs into a similar problem.

 

[Steve Meyerson]

Perhaps mark some suspects "Exclude"??

I know you have many many files, but perhaps they can be marked so in the pjx file. Don't know, just a guess.

Steve

 

[Chris Miller]

So something in the middle must be stopping it from adding anything past a certain point in main.

I modified my test project so that programN calls ProgramN+1000, if you know what I mean, to check whether I see depth first analyzing of the code or breadth first, and the records in the pjx are still main, then program0001 to program2000, not programm0001, program1001, program0002,... so I don't see that.

You said...

I used to just run a quick PRG I called "MakeVovio" to copy everything to a staging folder, generate the EXE from the project

Are you waiting for all copies to finish and be complete files in the staging folder before you start the build with a BUILD commad?

Depending on what you use for copying code can finish and the OS/Filesystem has still work to do.


Chriss

 

[Joe Crescenzi]

I know you have many many files, but perhaps they can be marked so in the pjx file. Don't know, just a guess.
Steve

Technically, since I'm doing my testing on a copy of my original, I can just intentionally delete the files with the newest timestamps, and then see if there's any improvements to narrow down the specific date that may have an impact. Once I see improvements, copy some back until it breaks the compiler and then I should be able to find the one(s) that could be the cause.

I modified my test project so that programN calls ProgramN+1000, if you know what I mean, to check whether I see depth first analyzing of the code or breadth first, and the records in the pjx are still main, then program0001 to program2000, not programm0001, program1001, program0002,... so I don't see that.

Interesting. It's still hard to know for sure what role sequence plays, but I can absolutely say that when I look at the live PJX, the newest programs are definitely at the bottom and that when I start with an empty project, the order I see is definitely starting with the first objects found in main, and that that there were only 4 errors that are part of Main.prg, and all 4 are the last 4 DOs.

It's a real mystery and since there's a predictable pattern, where it always builds a project of 252, instead of 1400+, I'm hoping I only need to check those 252 files to find the answer. Like I said, it could also be the file that would've been #253. To test for this, I can write a program to copy the 252 to another test folder, then add another DO at the end and see if it finds it.

Are you waiting for all copies to finish and be complete files in the staging folder before you start the build with a BUILD commad?

Depending on what you use for copying code can finish and the OS/Filesystem has still work to do.

Technically, I do it in two parts. First I run a batch to do the copying, then the MakeVovio is a PRG within the folder that builds the program and generates my Zip files to deploy updates.

I do this in two parts because I like to keep a copy of the last version deployed, including the source frozen in case I need to confirm or test any issues found in the last update. Otherwise I can't re-create or test problems 100% if I test it using source that is newer.

So, I always build the shipped version on a copy. I tend to keep the last 2-3 updates until I no longer need them, and it also lets me revert to a known safe version of a change in case something goes wrong with an edit. I also keep up to 31 copies of my code on a NAS that makes it easier to compare changes over the past month (I use a great compare tool for this) for the same reason.

The MakeVovio fails even in the original location, and all the copies of the directory I've made for testing this. In this case, all my recent tests are either in a temp folder on C:, or the exFAT drive I created to eliminate the potential for permissions.

I've done all subsequent tests on copies made earlier in the week. All versions yield 252 rows when using a fresh project. I've deployed perfectly working updates to 3 clients this week using the project file that I've been manually adding any files that are reported undefined. Thankfully, that works, but it still bugs me knowing that the compiler will not build fresh project files. I'll keep tinkering and see what I find.

 

[Chris Miller]

when I look at the live PJX, the newest programs are definitely at the bottom

Yes, new records are last.

when I start with an empty project, the order I see is definitely starting with the first objects found in main, and that that there were only 4 errors that are part of Main.prg, and all 4 are the last 4 DOs.

Obviously that's more interesting. I was also thinking does VFP maybe PACK the PJX data with a certain index order set to order by filename. Also since, the project manager lists prgs alphabetically, no matter in which order they were created/added to the project.

If you open up the debugger and its debugout window during compilation you can see it's analyzing full files, so that speaks against depth first processing. My build text project - even in the more compicated structure - will not do depth first analysis, so all programs are added merely because main.prg calls them all. That all likely differs in a project you maintain and grow over 40+ years, of course, but when you copy over to a staging folder the way the build process works and adds in files of course matters and it's good to know. Since I can't confirm what you see in that respect.


Chriss

 

[Joe Crescenzi]

It makes sense that the order in an older project file will seem random at first, then chronological over time because VFP definitely just adds things to the bottom when it encounters something new. As you pointed out, they can sort it on the fly anyway.

So as it stands now, the order that it parses main is my only lead.

I only know for sure that the top rows are definitely exactly in the order of the first DOs, and that all 4 unknown records just happen to be the last 4 DOs in main, so I can confirm that it doesn't just add all the DOs in main first before crawling them.

I could really shake things up by moving those last 4 DOs to the top, then seeing if it includes them this time, and leaves out different DOs at the bottom that were originally included, or removing the DO just before the last 4... perhaps the DO that is currently 5th from the bottom is the cause. So many fun experiments left.