FileToStr and StrToFile 1

[viclap]

Hi All,

I'm wandering this odd behaviour of VFP on how to solve this problem of mine. When the code below runs on the VFP IDE environment it works fine and generates an output of csv file. When the app was an exe file already, it hangs on the code of FiletoSr and StrToFile, it doesn't continue to next line of code which is the wait window message and the cleanup code. Is there anything I've missed in the code?


StrHeader1 = "DN,Device ID,EMS Device ID,Device Alias,Device IP,EMS ID,Vendor ID,NE Type,Model,Hardware Description,Functional Description,Parent DeviceID,ParentDN,SiteID,Device State,Software Version,"
StrHeader2 = "Integration Date,End of Support,TSA Scope,Product ID,Serial Number,FREQ (TX/RX),Hardware Capacity,Domain,NE Owner,TX Clustering,TX Type,NATSPCODE,Admin State,SUBDOMAIN,FUNCTION,IUBCE DL LIC,"
StrHeader3 = "IUBCE UL LIC,S1CU LIC,Cluster Region,Cluster Sub Region,Cluster Province,Cluster City,Address,Site Type"

USE C:\AddIns\Master_DeviceData_File IN 0 ALIAS MDF
SELECT MDF

INDEX on SubDomain + "," + EmsID + "," + DN TAG DnOrder UNIQUE
SET ORDER TO TAG DnOrder

&&Creating Output file for Report generation
COPY TO "C:\AddIns\Device_Data_Report\FxDeviceDataInv_" + DtOS(DATE()) + ".Csv" DELIMITED WITH CHARACTER ','
FinalHeader = StrHeader1+StrHeader2+StrHeader3+CHR(13)+FILETOSTR("C:\AddIns\Device_Data_Report\FxDeviceDataInv_" + DToS(DATE()) + ".Csv")
STRTOFILE(FinalHeader+CHR(13),"C:\AddIns\Device_Data_Report\FxDeviceDataInv_" + DtOs(DATE()) + ".Csv")


WAIT WINDOW "End of Device Data Report.Output File at C:\AddIns\Device_Data_Report Folder." AT 20,60 TIMEOUT 3
CLEAR ALL
RELEASE ALL
CLOSE ALL
QUIT

Please help me on this.
Vic

 

[Olaf Doschke]

Maybe you hit the 16 MB limit of strings.

https://fox.wikis.com/wc.dll?Wiki~VisualFoxProLimits~Wiki:

Maximum # of characters per character string or memory variable 16,777,184

It's not a hard limit, you can generate longer files easily, for example, using SPACE() to create an empty 32MB file:

StrToFile(Space(2^17),ADDBS(GETEENV("TEMP"))+"empty.txt")

In general VFP can deal with up to 2GB files, just not safely have longer than 15MB string variables.

I don't see how that would make VFP stop. It would error and either exit or display an error. Maybe log an error and exit, maybe RETURN TO MASTER, which is a rarely used option. So I'd look what error handling is done.

Then the IDE vs EXE difference might be because of embedded file in the EXE, FILETOSTR() picks them up, but I don't see how that could be a problem with files including DTOS(DATE() in their name, that varies per day, so that would have weird requirements of how this adds in to included files of a project compiled into an exe making problems there.

The only hint is how error handling can change the course of how code execution continues or stops. Find out what ON('ERROR') is before the COPY TO.

If you don't find anything, turn this usage of FILETOSTR() and STRTOFILE() into the usage of low-level file functions FCREATE/FOPEN/FPUTS/FCLOSE and that'll surely solve any issues with too large files, as you can work in say 8KB blocks, something far below 16MB.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name

 

[Olaf Doschke]

And another hint, if the file size os the rot problem, as nice and easy as it is to create files in single commands like COPY TO And STRTOFILE, when you write out data without header, then combine a header with the data read back in to finally save it, you write twice and read once, where you could also just write once. Even if read is faster due to caching, you can get this to less than half the process time, if you take the effort of generating CSV yourself and you don't even need to write this, MS has given a KB article because of the missing ability to write out Memo into CSV or multiline strings in general, and I built on top of that, you can use this to generate your CSV with this function:

faq184-7770

Note I write out a header with the field names, so if your header strings are exactly that you don't need to add them, if you want another header line just replace the section outputting the first line by going through the first element (field names) of the AFIELDS() array I create in the firs loop after the Fcreate() and instead of that just add a parameter for your header line and output it there.

Bye, Olaf.



Olaf Doschke Software Engineering

https://www.doschke.name

 

[viclap]

Million Thanks Olaf,

Good suggestion to create a log for errors or something to track what going on COPY To. I think i didn't hit the 16 MB string limit.
Also, very nice suggestion to try the low level file functions of VFP which is tremendously very fast.....

Vic

 

[Olaf Doschke]

Yes, you would expect an iteration on VFP record by record becomes slow, but even when you don't buffer before output the file system does on the software OS level and hardware, also even such a multitude more of commands doesn't make it slow. VFP is more like an interpreted than compiled language, but the bottleneck still is the file write.

That output does only need to create the file once, not twice, not read the csv back into a string. That's making it good enough at least, to be faster than creating a file without headers to prepend them in a second pass. And it fixes some of the issues VFP has with CSV creation like creating a CSV with headers.

Bye, Olaf.

Olaf Doschke Software Engineering

https://www.doschke.name