Foxpro 2.6 file size limitations 1

[ITfromZX81]

Hello all,

We use a program that was developed with Foxpro 2.6 for Windows.

One of our datafiles has a memo file (FPT) file associated with it that is passing the 2 Gigabyte mark.

We are having all kinds of problems with it - when we use the pack and reindex utility that the program uses we get internal consistency errors.

My guess is that there is a size limiation and that we are passing it - the only way we can get things working is by restoring from a backup file, which works for a few days until the problem reappears.

I have heard of an issue with 2 Gigabyte files for DBF files - is it the same for FPT files? It would explain what is going on.

If anyone can give me a detailed breakdown of file size limits for various files in Foxpro 2.6 it would be appreciated. I need this info asap to explain what is going on to the powers that be.

Thanks in advance,

ITfromZX81

 

[Stella740pl]

Yes, 2 Gb file size limitation in FoxPro applies to .DBF, .FPT, and .CDX files (and also .IDX, I believe) files. Moreover, not only for Foxpro 2.6 - it's still there for all existing versions of Visual FoxPro, and there are no known plans for it to go away.

Detailed breakdown of file size limits for various files in Foxpro 2.6 should be found somewhere in the Help. Try to search your Help for "System Capacities".

As for dealing with the limitations, you have an option of a table's breakdown, either vertical or horizontal, depending on your case; or switching to a different back-end, say, SQL Server.

 

[ITfromZX81]

Hi,

Thanks, I found the System Capacities item.

The program was not written by us - we pay for it - it is a program written by another company for our industry.

I think the problem is that this specific file is for archiving data and we are a larger company and have been using this program the longest, so I don't think any other user of this program is having the same problem.

I think these guys are going to argue that the 2GB limit doesn't apply to FPT files and I think it does. The only way I see to fix it is to split it into more than one table.

Is there any other Foxpro docs which clearly state that there is a 2GB limit on FPT files?

Thanks in advance.

 

[DSummZZZ]

Whether you/we find documentation or not, there is indeed a two gig limit on the memo file size. And since all memo fields in a particular table are all stored in the same memo file, it can grow to be quite large.
One way or the other, the table structure hasn't really changed all that much from the original, which was designed for 8-bit systems.
Even with VFP 9, the 2 gig limitation is still there.
Here is documentation:

http://msdn.microsoft.com/library/d...html/6e27027b-806b-4f96-b7e5-5f24af2901e0.asp

-Dave Summers-

Even more Fox stuff at:

http://www.davesummers.net/foxprolinks.htm

 

[DSummZZZ]

See what happens when you start a post, get busy and come back to it? It really sounds snide!

I really wasn't trying to sound that way, and I apologize if that's the way it sounds. I should have re-read it before clicking.

Anyway, take a look at the link. It gives the information you requested.

-Dave Summers-

Even more Fox stuff at:

http://www.davesummers.net/foxprolinks.htm

 

[JRB-Bldr]

ITfromZX81 - When you say...
"I think these guys are going to argue that the 2GB limit doesn't apply to FPT files"

Perhaps it should be asked by your management if these guys are as knowledgeable as the gurus with an accumulated 100+ years of FP/VFP development experience from these FP/VFP forums.

When the preponderance of people here are certain that the 2 GB limit applies to ALL individual FP/VFP tables and their separate component parts (DBF,CDX, & FPT files) then I'd consider it "gospel".

Good Luck,


JRB-Bldr
VisionQuest Consulting
Business Analyst & CIO Consulting Services
CIOServices@yahoo.com

 

[ITfromZX81]

Hey guys,

No worries DSummZZZ - I didn't consider the remarks snide - I have a built in filter for emails and postings - I understand that sometimes the typed word doesn't convey what a conversation would if you were standing in front of me.

To JRB-Bldr - yes, I wholeheardedly agree - I'm just worried this is going to turn into a peeing contest without an effort to resolve the actual problem.

Thanks again everyone - at least I have a smoking gun...

ITfromZX81

 

[dbMark]

So likely there are two major ways to handle this. (1) Remove and archive some of the data that is not longer needed. (2)If it all must stay accessible, especially since you described it as already being an archive, the data will have to be split into 2 or more tables. If the division will be visible to the user, figure out how best to split it apart, possibly by year, type of event, or some other intuitive manner. If it needs to stay in one big bucket so-to-speak, then you can still keep one huge master lookup table but add a field (or use a unique existing one) as a key so the program cn determine which of a group of tables with the memo fields is desired.

Oops, I better not say any more, I don't want to do your vendor's work for them. They should have anticipated this eventuality. For them not to be aware of the 2GB limit does not necessarily mean they're not good programmers. Apparently they hadn't been bitten by this issue before. But now that they know, they ought to get down to business and determine how best to fix the issue and quick. It's not going to go away and data will continue to be corrupted until it's fixed.

Also, it sounds like some of your archive has already been corrupted and lost. Keep enough backup copies so they can also reconstruct as much as possible. Of course this probably won't be free, so your company has to decide which are the most important issues overall, both to rescue archived data and storing future data.

Since you said the company has other customers, they also will also need the solution at some future time. Unfortunately, your company needs the fix first, so your company may have to shoulder a large proportion of the costs, although perhaps a version upgrade for a fee could help spread the cost among customers.

dbMark

 

[Brak]

Another solution is to break the database into several databases (by year or such) and span them together.

B"H
Brak

 

[ITfromZX81]

Hi guys,

Well, the problem is solved for the moment. The software maker ran a utility on our files that removed any memo fields of around 1000 lines or longer in the tables in question and this allowed the FPT file to pack correctly - it is now around 380MB.

It turns out we had a lot of old data from a data conversion from our old unix software of many years ago - it is mostly pages and pages of junk but we have reference copies if need be.

This still doesn't fix their 2GB problem but it should be okay for a few years yet...

Thank you for all of your help and input.

ITfromZX81

 

[JRB-Bldr]

As we have said above the 2GB issue persists in the Visual Foxpro world as well.

BUT as for "This still doesn't fix their 2GB problem"...

A re-design into Visual Foxpro would allow you to:
1. Utilize most of your existing code thereby preserving the data handling capability advantages of FP/VFP over VB, etc.
2. Use a variety of back-ends such as VFP, MySQL or SQL Server -- some of which would not have the 2GB limitation.

Good Luck,
JRB-Bldr

 

[RichardMoran]

About 9 or 10 years ago I looked after a marketing database that had several files close to the 2GB limit. We ran systems written in Foxpro for Windows 2.6 and the DOS equivalent which was virtually identical. We spoke to MS who just said that there was a 2GB limit and there was nothing we could do if we were to continue using Foxpro.

I think the limit is to do with this which I copied from the MS site :-

FAT16 volumes larger than 2 GB are not accessible from computers running MS-DOS, Windows 95, Windows 98, Windows Me, and many other operating systems. This limitation occurs because these operating systems do not support cluster sizes larger than 32 KB, which results in the 2 GB limit.

Foxpro for Windows 2.6 was barely a Windows application! It looked more like some sort of port, so I imagine that the 2GB limit imposed by DOS was carried over at the same time. The restrictions imposed by the other operating systems available at the time would have also reinforced the 2GB limit.

I must admit, not having used Foxpro for about 8 years, I was surprised that nothing was done to address the issue in VFP.

I wish you luck!

Rik

 

[Stella740pl]

...said that there was a 2GB limit and there was nothing we could do if we were to continue using Foxpro.
...
I must admit, not having used Foxpro for about 8 years, I was surprised that nothing was done to address the issue in VFP.

Well, the master table of a database I look after (and more) for a few years already ran close to just 1GB some couple of years ago - and it doesn't have any memo fields, and not many fields at all. So I split it into separate tables by year well before it got anywhere close to the 2GB limit (it would have by now if I didn't), made a few changes to the code, and it runs just fine, and probably will for some time more in the foreseeable future.

 

[TamarGranor]

Unfortunately, as I understand it, the 2GB per file limit is so tightly wound into the FoxPro engine that removing it would be a major and risky operation. Since there are lots of alternatives and the limit is well-known, it makes more sense to leave this as it is.

FWIW, I understand that in Access, the entire database (included forms, reports, etc.) is limited to 2GB. In VFP, it's each file.

Tamar

 

[dbMark]

The 2GB limit probably is related to standard (until now) internal data maximums. 4GB is 32 bits, so 2GB is 31 bits, just the right size for a 32-bit integer which reserves one bit for the sign.

In thread184-57123 TomasDill said: "... 2GB is a limitation of the integer number used to perform seek functionality over the file on the low level. When moving to larger number, it will require major reworking of the basics..."

That about sums it up. The old proverb, when there's a will, there's a way, is still true. If they really wanted to, they could rework the code to replace the 32-bit integer logic with 64-bit integers. Hard but not impossible. Of course, then they'd be petitioned to allow more than 254 fields in a table, 254 characters in a character field, larger field names in free tables, etc. (Looking at it that way, I guess we should be glad that we got all those fancy new data types!)

SQL Server was designed from the start for huge volumes of data and doesn't have that limitation. So does open source Python, it's fully 64-bit and roughly equivalent to VFP, so I've heard.

I saw a comment somewhere on the forums quite a long time ago, can't find it now, that the last time there was a major upgrade for FoxPro (was it from FP to VFP?) there were lots of complaints, so maybe that's an excuse not to do it.

dbMark