STRTOFILE() Large Files not written out

[GriffMG]

I use an included table in many of my applications, in there I store binary files that may be
useful at runtime, documents, .dlls, executables, anything I need after the install.

I search the table, find the binary and use STRTOFILE(binaries.binary,m.Filename) to drop
them where required.

Today someone brought to my attention that some of the files were not being created - in this case
large PowerPoint .pptx files.

It turns out that STRTOFILE() was not able to read the binaries.binary field as it was too big
(16MB or so) so I had to store it in a memvar and then write it out

PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME)

This seems to be good for up to the file limit we are all used to 2GB

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[mJindrova]

Yes, it's limitation for STRTOFILE() (VFP6-VFP9). But you can use COPY MEMO [memoname] TO [filename].

mJindrova

 

[Mike Lewis]

Griff,

Are you sure that this is a size issue? As a test, I just did a STRTOFILE() on a file that was 44.8 MB, and it worked OK.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Mike Lewis]

And no problem, with a POwerPoint file of 23.1 MB.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[GriffMG]

Hi Mike

StrToFile can write a 2GB file, from a memory variable, or possibly from a filetostr() - not tested, but if the source is a binary field
in a table it did not (for me at least) want to take one in excess (I think) of 16MB directly.

I tried

Didn't work

StrToFile(tablename.fieldname...
StrToFile((tablename.fieldname)...
StrToFile(Evaluate("tablename.fieldname")...

but this works

PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME)

Not tried copy memo.... yet

** edit **
This works:

COPY MEMO BINARIES.BINARY TO (M.FILENAME)

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

The help topic on System capacities actually says you can't store more than 16,777,184 bytes into a memory variable. Whereas a blob field is documented to enable up to 2GB in it. I know the SPACE() function ends at that limit. I already know the variable length limit is not as hard as the system capacities tells it is, but it's odd you couldn't just write out a Blob ia FILETOSTR and you indeed used a variable, that by documentation should work worse than the blob field.

There is a simpler solution to write out a blob field to a file, though, Griff, much more straight forward: Using the COPY MEMO command. I think that was extended to also work on blob fields, which are just a binary memo field anyway. Let me try... ...yes.

Create Cursor Files (wBinary Blob)
Append Blank
Append Memo wBinary From (GetFile()) && optionally with OVERWRITE clause, but unneccesary for a blank Blob
* and to write it out
Copy Memo wBinary  To (Putfile())

I would prefer it over using a variable as intermediate storage for the blob field, even though I know the variable length capacity of the documentation is wrong. It's not clear to which extent you can disregard this limit or whether this only is a limit of some functions handling string parameters, or what actually are the limits. It's clear that COPY MEMO is always working.


Edit: Sorry, I just saw you came to that conclusion after posts of mJindrova and Mike. So I think COPY MEMO is the best solution.

Chriss

 

[Mike Lewis]

I'm aware of the 16 Mb limit for memory variables. But I just did this:

[tt]x = FILETOSTR("bigtable.dbf")
? LEN(x[/tt])

and it showed 47,000,050, which is 44.82 MB. And when I did this:

[tt]STRTOFILE(x, "bigtable1.dbf")[/tt]

the resulting table had exactly the same data as the original.

I can't explain that.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[GriffMG]

Little oddity

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Mike Lewis]

Another little experiment:

x = REPLICATE("A", 100000)
y = REPLICATE("B", 100000)
DO WHILE .T.
  x = x + y
  ? TRANSFORM(LEN(x), "999,999,999,999")
ENDDO

The length of the variable got up to well over 100 million before I killed it.

The Help gives 16 MB as "Maximum # of characters per character string or memory variable". But can that be correct?

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[Chris Miller]

Mike,

yes, the limit for variables is a soft limit in some way, I knew that. And indeed Scott Griff used it too in getting STROTOFILE to work with a variable longer than 16MB, but not with a binary field longer than 16 MB (Well, I assume that limit was crossed). The point is Scott Griff already knows STRTOFILE() works with larger strings, too.

For sake of its simplicity and foreseen usage, I'd prefer to use the pair of commands APPEND MEMO and COPY MEMO to store a file and get it out.

I remember we experimented with STRTOFILE already and I find that thread: thread184-1813541. think it has no definitive result, though. I would accept from your's and Scott's Griff's new experience, that STRTOFILE also has limits, somewhere over 16MB, but not clearly definable, if it matters whether the source str is a variable or a memo/blob field. If that matters by Scott's Griff's experience it's not a matter of the length only, but the storage of "too long" strings in a blob/memo vs a variable. The only thing that bugs me is that the blob source didn't work while that's perfectly fine storing anything up to 2GB. So the "defect", if there is one, is with STRTOFILE. Must be with the function handling the string parameter weirdly.

Chriss

 

[Chris Miller]

I think, Scott Griff, STRTOFILE(FILETOSTR('embeddedfile.xyz')) is usual, if not the only way to get an embedded file out of an EXE back to the hard drive and perhaps that's why you got into the habit of using STRTOFILE. If you embed a DBF with a Memo field and hold files in there, you can of course also use COPY MEMO.

It's all because the basis of getting files out of an EXE can't be FOPEN nor COPY FILE, you have to use FILETOSTR(), but since you can simply USE an embedded DBF, that's not your concern, and then you also don't need to use the other half of it, the STRTOFILE() function.

We can conclude that if you embed files to write them out with the STRTOFILE(FILETOSTR()) trick, you are limited in file sizes. A DBF also limits you to 2GB overall, but you could also use more DBFs. I think the limit could be determined exactly, but I'd be lazy to find out if a DBF and COPY MEMO can be the solution that's safe to use. The only overhead is putting files into a DBF is one step more than just adding them as project files and including them in the build.

Chriss

 

[Chris Miller]

I don't know why I did that again, Griff. Sorry.

Chriss

 

[Chris Miller]

Mike,

your test is not that wrong. But to illustrate where the problem was in the thread I referred to with occurs - It's not just a problem of a specific length limit, it's also a problem that occurs with memos and not string variables! That's strange, because whereas variables are documented to be limited and memo length lmit is at 2gb, it's actually memos that cause the errors!

I said I'm lazy to find out the exact limit, well I now did for the case of the OCCURS function:

Clear
Create Cursor StringLimits (mString M)
Append Blank
x = Space(16777184)
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString)
x = x + ' '
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString)

Notice the "String is too long to fit" error happens with Occurs(' ', mString), not with Occurs(' ',m.x). So it's a problem with memos and not variables.

Astonishing.

And what happened to Griff (Griff?, yes Griff) with StrToFile() from a memo.

The limit 16777185 also is strange as 16MB is 16777216, 31 bytes more. The more interesting point is that the problematic case is happening for memos only, not for string variables.

I'm not astonished from the earlier discussions, that longer strings are possible, I'm astonished that memos cause problems with lengths that are far below the limit of memos. And since the functions can process longer strings in variables, what's causing the error with memos? So, it's not just the functions limit, but a combination of which functon with which type of string (variable vs. memo) you use.

Len(), as you can also see, Mike, is a function that fortunately has no problem also with longer memos, so it is indeed a problem matrix of function, and type (var/memo) of the string.

Chriss

 

[GriffMG]

Hi

I'm still here, just reading...

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
​

 

[Chris Miller]

What I wanted to say, Griff, is.

Your case with StrToFile() was just like the case of DanielEvansJr with Occurs() and the error also only happened with memos, not with string variables.

Chriss

 

[stanlyn]

No issues either way for me. I even bumped it up to 66,777,184 instead of 16,777,184 and still no issues.

Stanley

 

[Chris Miller]

Are you using VFP9 or VFPA?
Well, rhetorical question, your screenshot shows the caption is VFP Advanced.

Chriss

 

[Chris Miller]

If you use VFPA, try SPACE(2147483647), that's exactly 2 GB-1 byte and should be no problem for a 64bit process, though it won't be stored into a memo, even with VFPA. As far as I know, the table structure has still the 2GB limits and besides 2GB being the theoretical upper limit for a memo field, the FPT file for such fields also has a header and can only grow to 2GB overall, so the limit is a bit below 2GB. Surprise me, how far VFPA can take it. In theory, you can up things to 4GB, also file sizes, as unsigned 32bit integers go up to 4294967295.

No matter if you think about 2GB or 4GB, though. The C struct storing VFP variables of all types, including string variables, has a 32bit value for length and the length could grow to at least 2147483647. If used unsigned it could even be 4294967295. So that higher limit would even have been possible in VFP9, the reasoning for the system capacity limitation in VFP9 has to be a choice of the developers, and I can imagine memory management nightmares of strings the size of almost 2GB. The different limits of string vs memo is the only new thing I learned and that strings can hold more bytes than memos is the biggest surprise about that.

Chriss

 

[mJindrova]

Notice:
Large memory support will be in VFPA 10.2.

But MEMO field max size is 2GiB. FPT file size is unlimited (10.1)

mJindrova

 

[stanlyn]

Hi,

I tested Chen’s new downloads and found them to work. I posted my findings and a screenshot to

https://www.foxite.com/archives/0000487428.htm

and very happy to what I see. Here are the screenshot:

Hopefully that answers your questions...

Stanley