Project Compiler suddenly stopped including newer files 3

[Joe Crescenzi]

Hi Everybody,

I've been developing xBase programs since dBase II in the early 80s and all of a sudden I'm having the craziest problem, the project manager is skipping files on my most critical project and I have no idea why.

The strange this is that this is the same program I've been continuously developing since 1985, and I've compiled newer versions of it at least once a week without a single problem.

Now, when I compile the program, all the newer PRG procedures, and SCX forms are NOT in the EXE file... and are NOT in the Project File.

I'm wondering if anyone else has seen this problem before.

I assumed the project file was corrupt, so I created a new one and just added the main PRG file, expecting it to rebuild the whole project, but it didn't. In fact based on opening the project file directly as a table, the newer project file literally skipped nearly 600 files that otherwise were in the original, and it's basically saying in the error file that the files were not found. All the files are in the same directory as the project file.

I even tried compiling it on an entirely different computer, so it's not a corrupted version of VFP.

I'm starting to wonder if the compiler somehow hit some sort of limit to the number of files in either the directory or the project file itself?

There are over 4400 files in the directory, including about 1400 PRG files, 400 SCX files and a ton of report and other files.

My workaround so far was to add some of the missing files by hand to the project file, but I worry that I won't know which files are missing without either waiting for a customer to say "there's an error saying procedure xys is not found".

I'm thinking of writing a script to look at all the PRG and SCX filenames that are in the director, but not in the project file so I can know which ones to add, but that's not not the best option because there are other prgs and scx files that aren't actually called, so I'd need to start maintaining my own list of files that can be excluded.

This problem really has me confused. I've been developing code for 40 years ands it's like somebody suddenly telling me up is now down, or blue is now red. There's just no reason for the project to think files are missing when they're right in the same directory.

Any thoughts?

 

[Joe Crescenzi]

I only pasted the XFRX missing parts, which in itself is a sign that you really can't just use the XFRX.prg in your project without a struggle. The rest are about 100 or so PRGs which are in plain sight in the same directory, but since the compiler breaks... are never found, nor are their nested calls. This time it's not as many, but it's still quite a bit to add back by hand.

From what I can see, I still have a project file I made in 2013 to build the (easier to work with) version from some time around then.

The instructions for installing and using XFRX are different from distributing it, which adds to the confusion.

The distribution instructions contain a list of this is dependent on that, which in turn is dependent on something else. Some of the files are in (home)\ffc, others are inside of some zip files along with other files, and others are simply various GDIPlus, and zip libraries.

What I'd really like to see is a folder called "COPY-TO-YOUR-DEV-FOLDER-TO-COMPILE" and/or "COPY-TO-YOUR-DISTRIBUTION". Frankly, I'd love to see it coexist with my own code, but considering all the problems, I'd love just a simple handful of pre-compile files that I can just drop into my dev folder and go. I located about 30 required files and it runs, but as I said, that seems to break the compiler again and it stops looking at my own code files.

I'll also need to make sure that even if I manually add any missed code, that the distribution will work anywhere. For example, he instructs you to add a "SET CLASSLIB TO (HOME()+"ffc\_reportlistener")" just before calling it. I need to ensure that the compiler pulls that into the compiled exe because my clients won't have a VFP IDE with an FFC folder (I don't think the runtime includes it).

 

[Joe Crescenzi]

RESULT: If Joe compile xfrx.prg to xfrx.app then must initiate XFRX object follows: loxfrx = EVALUATE("XFRX('XFRX#LISTENER')")
mJindrova

Thanks for clarifying this. So, when you said "The evaluation version of XFRX cannot be included into VFP projects, it makes VFP crash.", the bottom line is you can't include ANY XFRX.App, even a compiled version of the paid version.

So, basically any .APP version of XFRX will crash the compiler.

In the last version I used, I did not bring over an APP. I just added the XFRX.PRG and about 30 other files. That was a mess and it didn't compile correctly, so I'll try to put everything into one folder, compile it into an App, and try the macro substitution.

 

[mJindrova]

Joe,


XFRX#LISTENER is derived from vfp report listener and depends on next files which are not parts of xfrx distribution package.

All writes here:

https://eqeuscom.atlassian.net/wiki/spaces/DOC/pages/3899429/Installation

mJindrova

 

[Chris Miller]

he instructs you to add a "SET CLASSLIB TO (HOME()+"ffc\_reportlistener")" just before calling it. I need to ensure that the compiler pulls that into the compiled exe because my clients won't have a VFP IDE with an FFC folder (I don't think the runtime includes it).

That shows some missing knowledge about how things work during IDE usage of code and within a final EXE. If you SET CLASSLIB To any directory not existing in the target computer (the end users machine) then that still just looks inside the EXE and the build includes the _reportlistener.vcx into your project.

you really can't just use the XFRX.prg in your project without a struggle

You only have one main.prg in any pproject, so if you expect moving the main program of a separate project into your project would work the same way as starting with just your main.prg

Overall, it sounds like it could be better documented, but your way of building up a fresh project everytime you build also isn't the normal way to work in projects and it seems to me all your problems mainyl stem from that way of working.

Chriss

 

[Gerrit Broekhuis]

Joe,

Being a XFRX user for more than a decade I allways try to use the latest version. First in a special test application I built for this purpose, then in our main applications.

Certain versions of XFRX have had their issues, as you can see in the XFRX release notes. The latest versions work perfect for us and our customers.

So you should be able to get this running.

Regards. Gerrit

 

[Joe Crescenzi]

Overall, it sounds like it could be better documented, but your way of building up a fresh project everytime you build also isn't the normal way to work in projects and it seems to me all your problems mainyl stem from that way of working.

I'm not sure if I made this clear. I haven't needed to generate a new Project file in at least 15+ years. I've always use the same project file.

This whole thing started a few months ago, when an older prg made its way into my project that had "Set procedure to XFRX", causing XFRX to compile into a dead end, and then it stopped finding any new files. I kept adding anything new by hand, but wondered why it stopped automatically crawling my code to add anything new. That's when I posted here to see if anyone else had that happen.

So, in order to isolate the problem, I copied my development folder to a different drive, and tried to just build a project from scratch. Every time I did this, it picked up exactly 252 files, and didn't crawl and discover anything else.

With that in mind, I systematically kept tinkering until I realized that from the main itself, there were only 4 missed files, and all four were at the bottom. By moving them higher, it was even worse, skipping at least 80 more files... removing them made it find everything (about 1400). That's how I knew I only needed to explore those 4 files. Eventually I discovered the set procedure line and removed it and it worked as it did for decades, capable of finding everything.

With everything working, I was inspired to finally bring in a newer XFRX, but by using the PRG, the original problem is back. I'm going to take mJindrova's advice and compile it to an App and only reference it by macro substitution. The main thing is not allowing the compiler to parse XFRX.

 

[Joe Crescenzi]

Here's where things stand.

I changed all references of:

m.loSession = xfrx('XFRX#INIT')

To

m.loSession = EVALUATE("xfrx('XFRX#INIT')")

Now, the compiler doesn't even know XFRX.PRG exists.

Then I simply compiled XFRX.PRG into XFRX.APP. It generates a bunch of errors, but the APP works perfectly, and in my sandbox no longer messes up the compiler.

I'll just need to make sure I include the .APP and the other required files are in my distribution and that should do it. From this point, I should always be able to swap out the newest XFRX by pre-compiling to an APP like I did here and things should be good.

 

[Chris Miller]

Good to hear you found a solution.

Sorry, that I took it for granted you build this way, since you mentioned a statging folder and this method seemed to me motivated by only inclluding everything into a project that's actually used.

Good news, or bad news - depends on how you look at it - I just tried to start a simple project only doing SET CLASSLIB TO _reportlistener.vcx, not literally, I used both the syntax with HOME() and the absolute path and most of the time, even though the _reportlistener.vcx file clearly exists, the VFP project manager reported to not find the file and the vcx was not included into the project. So there is so,mething going on with VFP not adding files into a project, even when they are explicitly specified in source code.

That means the issue you have with too small projects could be an issue completely independent of the XFRX issue. As if there is a change in the file system api regarding the way VFP looks for files.

To reproduce the problem, all you need to do is start VFP fresh, default directory (for me) then is HOME(), i.e. ? Sys(5)+Sys(2003) prints the same as HOME().
Creating a new PJX with just SET CLASSLIB TO ("c:\full\absolute\path\to\_reportlistener.vcx"). Build and the vcx is not included, sometimes with reporting the "undefined", sometimes not.
Only when CD into the ffc folder before the build, the vcx is found. If that's normal, I must have added anything I used manually, so far. Of course it's never a problem. And if I think of how I usually introduce a new PRG, I write a new PRG by using the NEW button of the project manager, which means the new prg is added that way before build. Anyway, it's a well known feature of VFP to include files during build and the experiment with the 2000 programs also worked that way. There was no MS update since then. So whatever does not work is unclear and the exact circumstances, too. And, well, this is Windows 10, not even 11:

Edition Windows 10 Pro
Version 22H2
Installed on ‎11/‎03/‎2022
OS build 19045.4170
Experience Windows Feature Experience Pack 1000.19054.1000.0

I'm not sure, but it seems not normal, even if the absolute path is not within the project home directory. If you use vcxes, prgs from several folders within Home() or your own directories, you can't obvioulsy CD into all of them. What also works is Set Path To ".\ffc" additive, but that means PATHS relative to the default directory you set when building, not relative to the project home directory. That's not new, I already said build can depend on the current/default directory, but then absolute paths are absolute paths. I wonder what is going on.


Chriss

 

[Joe Crescenzi]

As a followup... Distribution.

Unfortunately, since "SET CLASSLIB TO (HOME()+"ffc\_reportlistener")" is evaluated at runtime, it didn't just pull it in, so my test distribution failed to find it when I ran the exe, so I copied it locally, along with a few other files referenced from the (external) XFRX.APP, including GDIPlus and added them to the batch that builds my distribution and everything is running smoothly.

The newer XFRX seems to work faster than the older version, and the PDFs somehow look cleaner. It's hard to say why, but the margins seem more accurate, formatting and some of the typography looks cleaner, and they handle some things the older one couldn't do, like angled text.

I just need to confirm important things like addresses showing up invoices in the same places, so they fit windowed envelopes the same way, etc.

 

[Chris Miller]

so I copied it locally,...

I would rather include it manually into the project - especially since what I found is current behavior with automatic including project files, see above - it then will work without any external extra files and no matter if HOME() exists n a target machine or not.

Just double checked, once the _reportlistener.vcx is in the project (added manually, for example) you can even write

SET CLASSLIB TO "I don't care\_reportlistener"

This will only cause an error when you run this PRG within the IDE, the EXE will ignore any path, even HOME() and just look for _reportlistener.vcx in the EXE, find it and not error.

Chriss

 

[Joe Crescenzi]

I'm not surprised (home()+"\ffc") didn't work. Because of the parenthesis, that's just another macro, so it's evaluated at runtime so the compiler didn't bring that over. I guess I could've used the physical path because most VFP installs map HOME() to the same place, but it was easier to just copy over.

My method of staging builds in different directories is just an audit trail and it helps me reproduce problems customers may find in earlier versions.

On a simplified level it goes like this:

I develop in a folder with .NEW that I never compile from.

When I want to distribute a new build, I like keeping everything from that build together frozen in time, so that goes to a staging folder, normally with a date like .319. I compile in this folder, and if there are no build errors, I run a batch file to zip up a distribution.

If a customer calls with a problem and I can't recreate it using the newest version, I like to re-create it running the same version of the source they have installed (3/19) to see exactly what they see. Also, since disk space is dirt cheap, by keeping the builds from the last 30 days, its also easier to compare changes. Technically, I also periodically push code to a GitHub repository, so I can compare unlimited older versions of a file that may have gone off the rails a long time ago, but sometimes it's quicker to just compare local folders. I use a great program called ExamDiffPro for that.

 

[Chris Miller]

Joe,

you miss that I said I used an absolute path and replaced HOME(), so that's not the fault.

There really is a bug about including files that are not in the default directory or SET PATH, even if they are specified with absolute path.

And I know from the past experience HOME() is a constant for VFP, so it is not only evaluated at runtime. It shouldn't be a problem, just like an absolute path shouldn't be a problem. There's something on top of the XFRX difficulties hindering VFP to include files into a project when it's not manually added. And the only chance that is normal is I never used this as I always created or added necessary files before building a project anyway.

Chriss

 

[mJindrova]

Joe,


Yes, because vfp can't find procedureas and function from other parts:
A file xfrxproxy.prg
[pre]
Proc./Func. XFRX_EXTRACTFILEFROMAPP - Undefined
Proc./Func. XFRXPROXY - Undefined
Program XFRXPROXY - Undefined
Proc./Func. XFRX_TESTFILEINAPP - Undefined
Proc./Func. XFRX_USETABLEFROMAPP - Undefined
Proc./Func. XFRX_SETPICTINAPP - Undefined
Proc./Func. XFRX_EXTRACTREPORTFROMAPP - Undefined
Proc./Func. XFRX_RUNMACROINAPP - Undefined
Proc./Func. XFRX_RUNPROGRAMINAPP - Undefined
[/pre]

A file xfrxlib.fll/xfrxlib64.fll
[pre]
Unknown _XFGETLTI - Undefined
Unknown _XFBMP2 - Undefined
Unknown _XFCROPIMAGE - Undefined
Unknown _XFCONVERTIMAGE - Undefined
Unknown _XFIMAGEBPP - Undefined
Unknown _XFWMF2IMAGE - Undefined
Unknown _XFWID - Undefined
Unknown _XFDEC - Undefined
Unknown _XFIMAGES - Undefined
Unknown _XFGETVERSION - Undefined
Unknown _XFIMAGEGETDPI - Undefined
[/pre]

mJindrova

 

[mJindrova]

Chris,

IMHO, VFP ignore - at buildind project - all macros, EVALUATE() function and all name expression at finding file dependies.

VFP try find file:
[pre]
DO PROCNAME
DO PROCNAME IN PROCEDURES_FILE
[/pre]


VFP don't try find file (name expression):
[pre]
DO ("PROCNAME")
DO PROCNAME IN ("PROCEDURES_FILE")
[/pre]


VFP don't try find file (macro):
[pre]
lcc="DO PROCNAME"
&lcc.
lcc="DO PROCNAME IN PROCEDURES_FILE"
&lcc.
[/pre]

VFP don't try find file (EVALUATE()):
[pre]
EVALUATE("PROCNAME()")
[/pre]

Joe,
Becase xfrx.prg contains command SET LIBRARY TO (name_expression) VFP can't find xfrlxib.fll at building project.
But (xfrx.prg) contains DO XFRX_TESTFILEINAPP WITH ... IN XFRXPROXY. In this case VFP try find XFRXPROXY.prg, but if a file can't find, then wrote messages to error log.


mJindrova

 

[Chris Miller]

mJindrova,

let's be totally clear: I said I use an absolute path, I have VFP installed in C:\USERS\PUBLIC\VFP9, and VFP does not include _reportlistener.vcx, if I write:

SET CLASSLIB TO C:\USERS\PUBLIC\VFP9\FFC\_REPORTLISTENER.VCX

And that removes any cause of evaluation or anything like that. It does not work and should. The build only finds _reportlistener.vcx whemn I CD into C:\USERS\PUBLIC\VFP9\FFC and that's the current topic.

The other thing is, once you have _REPORTLISTENER.VCX in the project you can change that to as little as
SET CLASSLIB TO _REPORTLISTENER.VCX, use any non existing path and the built EXE will still find it insde the exe, if you don't exclude the VCX library. And that's two different topics.

Chriss

 

[Joe Crescenzi]

As a rule to thumb, I don't normally reference files outside my source directory, but it's definitely strange behavior that even referencing them with a full path doesn't just bring them in. We both tried it and it fails.

The HOME() version is for different reasons, it's dependent on where you installed VFP, so (HOME() + "ff\") in parenthesis basically is only evaluated at runtime because the compiler probably sees it as an on the fly macro substitution.

Like you, I also was thinking even the quotes around the file path cause the compiler to just tokenize it as a string that will be evaluated later when you use SET CLASSLIB TO "I don't care\_reportlistener". Even when we eliminated the spaces in typical VFP Home directories and copy the file to the source directory, it still fails. For no apparent reason.

In the end, my solution was to just copy them and even include them in my batch file that builds my distribution zip file. I know I could manually add them to the project, but since I already needed to push a bunch of FLLs, and DLLs into my batch builder, I just added a few of the other required XFRX dependencies because they're only actually called by the (ALWAYS external) XFRX.APP file, which we know now is toxic to the project compiler. So now, the external APP can always see the always external dependent files.

****
**** LESSONS I'VE LEARNED
****

1. XFRX works best external from your project as an APP or as an FXP file.

2. Referencing it inside the project can break the compiler's ability to parse the rest of your files.

3. VFP also has trouble pulling other types of files into the project like the _ReportListener.vcx. It doesn't crash it like XFRX, but it doesn't seem to include it.

4. XFRX has a ton of dependencies, including _ReportListener.vcx, some DLLs and FLLs. Once you include them in your distribution, things work fine and dandy.

I had no reason to expect the #1-3, but once we discovered it, the easiest solution still seems like #4.

I know a lot of developers use it, because it does a lot of great things, but documentation isn't clear about the potential problems when it's unusual implementation is done with the source code version.

Perhaps most people didn't pay extra for the source, and only use the precompiled version. Frankly, I'd prefer it if I knew I just needed to stick a few files in my directory and include them in the distribution. For the record, when you purchase the source version, they don't include the pre-built version, so I can't confirm what it's like using the pre-built version.

Another complication is since it's including the source for everything, including things like the DLL, you don't know which of those files are part of the required distribution itself. The answers are divided into 4 potential places, installing, compiling the source, using it in code, and distributing, without a black and white set of steps to see the big picture in one place. Perhaps mJindrova can help us with that.

So, I basically kept it external, built my normal project and added dependencies one by one to my distribution zip until I no longer got runtime errors.

I can list the changes to my zip later for those who are curious or have a better solution.

 

[Joe Crescenzi]

The other thing is, once you have _REPORTLISTENER.VCX in the project you can change that to as little as
SET CLASSLIB TO _REPORTLISTENER.VCX, use any non existing path and the built EXE will still find it insde the exe, if you don't exclude the VCX library. And that's two different topics.

The funny thing is that files located inside an exe can always find other files inside that exe. It looks there first, then when they're not found, it looks outside, however an external file like an APP may or may not see those files for some reason even when that APP is called from the exe, which is odd.

So, another potential fix is to build the APP with those dependencies baked in. Perhaps the non-source versions do this, but I don't know because I purchased the source version and it's not pre-built.

 

[Chris Miller]

Joe,

thanks for confirming that.

Re 3: That's not the whole truth, because when I CD HOME() then CD ffc and then build the project, during the build the _repertlistener.vcx and _gdiplus.vcx are pulled into the project.
But if I take it as the way it ever was, then I wonder about the often recommended solution when a PJX breaks to create a new PJX, just add in main.prg and build. It would only work, if you
a) have everything in one folder
b) CD into that folder before building
or
c) alternatively to b) have all necessary PATHs in SET PATH so the files that need to be pulled in are visible from either default directory or SET("PATH"), but don't assume subdirectories aer automatically included. Nmobody hinders you to expand to as many directories including HOME() and subdirectories to not need to instead copy over source files from the VFP installation into your own project directory. It works better, if UAC doesn't sabotage things and you have VFP installed into a directory fully under your control with write permissions to compile not into VirtualStore directories.

The idea that VFP automatically pulls in files is therefore something that only works partially. Another working trick that's also often recommended is just drag a bunch of files or directories and drop them on the project manager window to let the files be added, instead of letting build discover and pull them in.

I have to say I rarely use this pull-in mechanism anyway, as said when I create a new PRG for a project I usually do it from the project and if I put in something from other projects I do use the project managers ADD and don't just wait for BUILD to pull it in.

What that tells me is that CD into the project home directory before building may give you a better experience in your compilation.

And the lesson I learn, if there is no other behavior on older Windows versions, too, is that the pull-in of files is surely not a total myth, but this has no recursive effect and does not at all work like a flood fill to complete a project just from a new empty pjx with main.prg. And, Joe, I understand by now that's also not what you're doing in general, it's still something I investigated into as you clearly state in the thread title that including new files should be automatic, when you just DO them from somewhere else that's already included in the project.

Chriss

 

[Joe Crescenzi]

Lots of lessons learned from this XFRX experience, especially that it was the main problem in the first place.

My main product has about 4800 files in it, with about 1400, prgs, vcx, and scx that become part of the project file. I intentionally don't include any of the FRX files in the project, because I like to be able to modify them on the fly as needed for customers.

My program evolved over 40 years, so modules are refactored and moved or deleted when I no longer use them. I generally trust that the project file is accurate and that if I write a new PRG or create a new SCX, it will end up in the project without needing to remember to do it. As a developer, I like predictability like that.

When I accidentally added the older PRG that had that SET PROCEDURE TO XFRX line, it broke reality for me because for the past few months, new features were no longer being added automatically. I'd only know they were being skipped when a customer would send me a screenshot with "xyz not found". It wouldn't fail for me in my dev environment, because the IDE always found the PRG as I ran it uncompiled. I kept adding them by hand only after getting another screenshot.

Thankfully now that it's external again, all is well. I added a new test PRG and called it, then when I built it, the project picked up the new prg, so I know all this led to unbreaking reality for me so I can focus on the program itself knowing that I can keep adding code and it will always compile without issues.

I thank you all for helping solve this mystery.

 

[Chris Miller]

I added a new test PRG and called it, then when I built it, the project picked up the new prg, so I know all this led to unbreaking reality for me so I can focus on the program itself knowing that I can keep adding code and it will always compile without issues.

I still rather keep the responsibility to add in a PRG to a project before even building it. Just to be sure the EXE will also have it.

Your way will work mostly because everything is in the main project directory anyway. But you could use the PATH setting and environment manager for ensuring that things are finding their way into the project and EXE, too.

Chriss