modified fgets() function

[Steviebone]

Thanks in advance to anyone that can help me.

I need to run some FoxPro programs that make extensive use of the fgets() function. These programs were written some years ago and unfortunately Microsoft altered the function in later versions limiting its maximum length return to 8192 bytes. Earlier versions of FoxPro did not have this limitation.

After experimenting with various ways to get around this problem I've come to the conclusion that I need a modified low-level function to replace fgets that does not place a limit on maximum string size. Because the function is called recursively on large files it needs to be efficient and likely needs to be coded in C++ and placed in a FoxPro Library fll. Experiments with writing code within native FoxPro proved to be too inefficient to be useful.

It's been some years since I've written C++ and I thought that before I delved into creating this myself I would see if anyone in the community had something to offer. Essentially all I need is a one function library that I can use in lieu of the fgets in native foxpro. Ideally it would function exactly as the native version but without the size limit just like in the older versions of FoxPro.

Can anyone help me with this? I'm sure this is just a few lines of C++ and I know some of you are experienced at creating the flls. I can dig up a compiler if I need to.

Thanks again for any assistance. Why Microsoft decided to alter this internal function without any backwards compatibility is beyond me?

Thanks again.

 

[Dan Freeman]

I don't remember any version of FoxPro (going back to FoxPro 1.0) that didn't have the 8182 byte limit on fgets(). If I recall correctly, the limit is the same as Clipper (where the function prototypes came from).

If you need to absorb a larger string, something like this would work:

REPLACE memofieldname WITH FileToStr("filename")

Where did you get the idea that 8182 bytes is a new thing?

Truth be told, there's rarely a good reason to use fgets() and the other LLFNs any more. There are so many other options.

Can you elaborate on what your actual requirement is so maybe we can point out ways to get there?

 

[Olaf Doschke]

To complete what dan said, besides FILETOSTR() there is APPEND FROM and IMPORT.

If your files are really large, over 2GB, then you would need something else anyway, all file access in VFP is limited to 2GB, including low level file function.

The easy way out is using Windows Low Level File functions, eg as shown in thread184-1613571

Bye, Olaf.

 

[Mike Lewis]

I might be missing something, but why not simply use FREAD() to read a fixed number of bytes into a buffer (it can read up to 65,535), then use ALINES() to separate it out into separate lines?

Regarding Olaf's point about the 2 GB limit, the following might be helpful: Working with large text files in Visual FoxPro. But it would probably be much slower than native VFP functions.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Steviebone]

@danfreeman:

Im not kidding about the 8192. I have both versions of foxpro running. I have been using these programs for more than 13 years. When I upgraded to v9 the programs broke because of the limit. In previous versions (vfp5) you could specify the max return size as the third argument. In vfp9 for some reason MS decided to fix the limit at 8192.

From the vfp5 help file:

FGETS(nFileHandle [, nBytes])

nBytes Specifies the number of bytes FGETS( ) returns. FGETS( ) returns nBytes bytes unless a carriage return is encountered first. FGETS( ) returns data between the starting file-pointer position and the carriage return if a carriage return is encountered within nBytes bytes.

If you omit nBytes, FGETS( ) returns a maximum of 254 bytes by default.

--------------------------

you can specify a large number as nbytes, works fine. Been doing it for more than a decade.

from vfp9 help file:

Specifies the number of bytes FGETS( ) returns. FGETS( ) returns nBytes bytes unless a carriage return is encountered first. FGETS( ) returns data between the starting file-pointer position and the carriage return if a carriage return is encountered within nBytes bytes. FGETS( ) returns a maximum of 8192 bytes. If you omit nBytes, FGETS( )returns 254 bytes by default.

same program run in 9 generates string too long error.

 

[Steviebone]

@mike lewis:

of course I experimented with fread and other means of reading the entire file into memory and then processing. The problem with that method is that the native fp routines for processing chr by chr are just too slow. I'm simply looking for a way to make the old programs run in vfp9 without generating the string too long error. There are dozens of programs, too many to go back and completely rewrite. Not an option.

Essentially, the routines process files that see paragraph blocks, 1 at a time, by identifying crlf pairs (blank lines between paras). These routines must also remove hard returns within paras.

The best solution for me is to replace the fgets calls with an equivalent function that does not have the 8192 limit. It needs to be coded in C to keep speed satisfactory.

The fp help file mentions "_fgets" (note the underscore). It even shows sample c code for the function. But it must be compiled into an fll. Anyone who has ever created an fll for foxpro could probably do this in 15 minutes.

Alines() might work, but would require extensive rewrite of more than 100 programs. A one to one equiv for fgets would be better... (set lib to, ... global s&r fgets to _fgets) much simpler

Thanks for your input. It is appreciated.

 

[Steviebone]

Alines shows some promise if all else fails... Funny, been writing FP code for more than 20 years and never fully appreciated this function... thanks,.

Would still prefer a one to one fix, however I may be able to code an fgets replacement that simply acts on the array instead of a file ptr. Because of the number of calls/programs tho this would be messy.

Anyone coded an fll before?

 

[Olaf Doschke]

OK, thanks for sharing. My VFP versions just go back to VFP6, and I didn't used it then, I used APPEND, IMPORT and FILETOSTR.

Your observation goes wrong about the error "string too long", though. To try to get more than 8192 bytes you do h=FOPEN(...) and lcString = FGETS(h,16384) for example, and that causes error "Function argument value, type, or count is invalid.", not "string too long". I could imagine you get "string too long error" at some time, if you repeatedly do lcString = lcString + FGETS(h,8192), but you would not read the whole file, just until a CR = chr(13) is found in lcString.

Have you already modified to using windows API or the VFP2C32.FLL proposed by Pavel at MSDN forums, or FREAD (allowing to read in longer blocks disregarding line feeds) and ALINES?

Strings have a limitation of 16MB, but it's not a hard limit, eg you can read in longer files via FILETOSTR(), eg I just tried and read in a 123MB file this way (indeed a video animusic.mp4, I'm sure you'll even find larger files on your drive) with no problem. Also ALINES() does not error on splitting this "too long" string into 990006 lines, which by the way makes use of the unlimited array size VFP9 offers. The VFP help tells about the 16MB string limitation in it's chapter about system capacities, but does not tell which functions have problems with longer strings, storing longer strings in variables is no problem, indeed, LEN() and as already used, ALINES() also work fine on oversized strings.

So I can't reproduce your problem about "string too long". What are you really doing, causing a "string too long" error? Do you have an error handler telling you ERROR(), MESSAGE(), LINENO(), PROGRAM() and further info via AERROR()?

Irregardless of all this, any of the proposed alternatives should help with your problem, it was fine of you to quote from VFP5 help, but now it's time to move. Changing most probably gets rid of the "string too long" error, anyway. But that error surely does not come from FGETS.

Bye, Olaf.

 

[Steviebone]

Here is the answer:

FGETSEX() from VFP2C32.FLL (

http://vfpx.codeplex.com/wikipage?title=VFP2C32&referringTitle=Home)

does not have 8192 chars limitation.

this fll library is full of useful stuff and the replacement fgets works without the 8192 limt.

Thanks to Pavel Celba at the msdn forum!

http://social.msdn.microsoft.com/Fo...6-b74ff339a017/modified-foxpro-fgets-function

 

[Olaf Doschke]

Meanwhile you posted more answers.

I don't get your complaints. You bark up the wrong forrest.

You are overlooking the proposed Windows API functions I linked to, scroll down in that thread to find my answer declaring API functions.

Also I don't know how ALINES would be too slow, it cuts a string in lines in one call. Are you checking for CR via looping from 1 to LEN(string) and inspecting SUBSTR()? This is nonsense. You would just need to check RIGHT(lcLine,1) after each FGETS, and repeat to read up to 8192 bytes more, until RIGHT(lcLine,1) is CR or LF. And that is only an idea to stay with FGETS, it's much easier to read in the whole file and do ALINES, it's even more easy to IMPORT or APPEND FROM the file. And overall it's nonsense you need a C++ function to replace FGETS.

Bye, Olaf.

 

[Mike Lewis]

Stevie,

If your VFP2C32 solution works, that's fine. But I think you were wrong to dismiss my FREADS() / ALINES() solution so quickly.

You wrote: "I experimented with fread and other means of reading the entire file into memory and then processing. The problem with that method is that the native fp routines for processing chr by chr are just too slow".

I wasn't necessarily suggesting you read the entire file into memory, and I certainly wasn't suggesting chracter-by-character processing. I envisaged a new function that would first read a chunk of the file - say 64KB bytes - into a variable, then use ALINES() to copy it to an array. It would do that the first time you called it. On that call, and on each subsequent call, it would return the next element from the array (which is what FGETS() would have returned). When the entire array was processed in this way, it would read the next chunk and start the cycle again.

Because the whole thing is a single function, with the same calling sequence and returned value as FGETS(), you could use a simple search-and-replace to incorporate it in the "dozens of programs" that are affected.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Olaf Doschke]

Actually, as I already wrote at MSDN forums,

As the nature of your files is texts (perhaps articles, perhaps even whole books) split into paragraphs with a CRLF, the whole process to read a file chopped into paragraphs is done with 2 lines

lcText = FILETOSTR("filename.txt")
ALINES(laParagraphs, lcText)

That is limited to 2GB but has no other limitations, as VFP9 allows more than 65000 array elements, though I doubt that's a
limit you'd break considering 10 paragraphs per page of a 1000 pages book you get to about 10000 paragraphs.

It may take longer to get the first paragraph this way, but for the simplicity and better maintainability I'd prefer this fetching of the whole file anyway. You can't read a whole file in faster than via FILETOSTR in VFP, even with support libraries, any file read is imited with the speed of the hdd, which may be an ssd, but even than that's the bottleneck more than any RAM. And to argue about the secondary step to chop the whole text into paragraphs, this is done in split seconds, considering todays RAM bus speeds.

The only thing that works faster with an FGETS approach is to get at the first (few) pragraph(s) and only continue reading, if a user navigates to next chapter(s). But that's the only advantage I can think of, aside from keeping the rest of your code logic. But how much easier can it be to replace the whole process of reading the file in with two lines of code and then work on the array elements?

Overall I think you were and you are still too focused on just replacing the FGETS, that you don't even try to see the bigger picture and make a deeper cut of old code in advantage of new much shorter code.

Bye, Olaf.

 

[Mike Lewis]

For what it's worth, it appears that Stevie is correct about the change in functionality of FGETS().

I don't have 3.0 to 6.0, but I have tested FGETS() in 2.6, and 7.0 to 9.0. In 2.6, I could read 15,000 bytes with a single FGETS() call. When I tried to do that in 7.0 and later, I got a "Function argument, count or type is invalid" error.

So that's a new entry in my personal knowledge base.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Dan Freeman]

The Hacker's Guide to VFP3.0 doesn't mention any limit on FGETS(), but would you like to guess the size they use in their sample code?

I suspect what you're seeing in versions earlier than 7.0 is a bug. The main difference between FGETS() and FREAD() has *always* been that one of them (fread) will read a whole file at a time while the other goes in smaller chunks.

 

[TamarGranor]

The Hacker's Guide to VFP3.0 doesn't mention any limit on FGETS(), but would you like to guess the size they use in their sample code?

We're guilty of missing the change, I think. Mea culpa.

Tamar

 

[Steviebone]

I know I'm right... I been using the same prgs to process massive files (statutes, laws and codes) for more than 15 years. I'm not sure what version the change happened in... I just know that v5=no error, I can read 500000 bytes if I want. V9= anything over 8192 generates an error msg.

It's not something many would ever notice. Let's face it, I doubt there are many people using fp in this manner where the 8192 limit would ever come up. I'm not really processing lines, I'm processing paragraphs, which in some cases (especially in the law) can be quite large.

In any event the aforementioned fll solved my delima and I am happy to report that I can now run my existing prgs in v9 without incident and without extensive rewrite.

FWIW, that fll has more than 100 calls Windows function wrappers... a most useful library... no serious vfp programmer should be without it. Full open source code provided as well.

@Mike @Olga:

Yes thank you. Future code may very likely make use of ALINES! That's a very nice function I was basically unfamiliar with! However, replacing the existing fgets calls with it would have been very messy. I wish I would have known about ALINES 10 years ago when I first developed some of these routines! lol

Thanks again for ALL your responses. They have all been most helpful.

 

[Steviebone]

OLGA:

Let me be clear, I wasn't dissing you're comments. The ALINES approach is most elegant and by far the better method if coding from scratch. I experimented with it today and it was a thing of beauty, flexible and fast. Again, if I had only known about it back in the day!

PS: Alas I feel somewhat vindicated from my stupidity, as ALINES was not around in vfp5. No mention of it in the vfp5 documentation.

 

[Steviebone]

One last PS: sorry Olaf, i am tired and my typing not so good, didn't mean to call you Olga my bad

 

[Olaf Doschke]

Two things:

1. I observe this behavour of myself, too. You have a solution, one thing doesn't work and you look for a fix of exactly that failing command. It's a bad behaviour pattern, and if I'm on the outside, I tend to point people to it. In this case you were just lucky, in the general case you always better widen your sight and see what you need on a more general level.

2. If you have so many places to change from Fputs() to Set Library and FputsEx(), and it takes you hours to change, then you have copy-pasted code, which is another result of a - let's say - pragmatic behaviour and you should bite the bullet and refactor your code anyway, no matter if you already fixed it and expect no further change causing the need to go through all the routines again for another function failing. What took you so long, anyway? Did you not use the project wide search and replace tool "Code References"? replacing FGETS with FGETSEX is all it takes, the parameterization doesn't change, does it?

Even in a good OOP design it's not that any function is only used in one class, but the tendency to centralize and generalize functionalities is much better in the OOP approach to development, and so the number of places to change in general are fewer.

This obviously is advice from a "youngster" to an old school developer. I never did FPD, I started in VFP6 with VFP. Nevertheless the rule to never change a running system is a bad rule.

Last not least a way to natively change FGETS with FGETS:

Function FileGetString(h, nbytes)
   * disregard nbytes, always read a full line
   Local lcLine, lnLen
   lcLine    = ""
   lnLen     = -1
   Do While Not Right(lcLine,1) == Chr(10) And Not lnLen = Len(lcLine)
      lnLen  = Len(lcLine)
      lcLine = lcLine+Fgets(h,8192)
   Enddo
   Return lcLine
Endfunc

To make this function available in any place a SET PROCEDURE in main.prg is sufficient or you put the function in your main prg, the prg set as starting point of your application. If you use a general header file in all classes, forms etc, you only need to add a #DEFINE FPUTS FileGetString and all FPUTS are compiled as FileGetString calls instead, or you replace with Code References.

Bye, Olaf.