32-bit VFP9, 64-bit OS, and more than 4GB RAM

[USMA99]

Some background:
We use large free tables in a processing-intensive environment. We would LOVE to see the 2GB barrier eliminated ... but that's the subject of another post. Before you tell me to take this out of Fox and move into Oracle or SQL Server or something, note that other data platforms just won't do. We have data processing programs that need direct access to the data which is something uniquely available in Fox's handly free xBase tables and something SQL Server just won't allow for. I suppose if there were handy ways of working with flat text files then that could be an option but I digress.

The question:
If we have 64-bit hardware, go to Vista 64-bit, run VFP9 in 32-bit compatibility mode (which sounds to work fine), and have say 8 or 12GB of RAM, would we be able to open large tables directly into RAM and use more than a total of 4GB of ram? (For example, simultaneously open 3 2GB files which total 6GB) I'm unclear on whether or not the 32-bit VFP9 program needs to worry about managing the memory or if the OS does all of that. A corellary question, if the answer is "no" can we use some sort of RAM-disk solution to get around this?

 

[craigber]

No. VFP will not support tables larger than 2 Gig. Period. It has nothing to do with the OS, processor, or amount of RAM.

Craig Berntson
MCSD, Visual FoxPro MVP,

http://www.craigberntson.com

Author, CrysDev: A Developer's Guide to Integrating Crystal Reports"

 

[DSummZZZ]

Sounds like an OS question, not VFP. You need to determine if any sort of RAM disk even can be used, which I don't think you have any control over. As far as I know, Windows pretty much assumes all control over RAM and other hardware.
But if it were even possible, you would end up with two copies of the table. One in RAM (which would get there by using COPY TO...), and one on a hard drive somewhere. I'm guessing you would still have to keep them in synch somehow, which would still involve disk I/O.
that said, take a look at SYS(3050) and see if it helps any.


-Dave Summers-

Even more Fox stuff at:

http://www.davesummers.net/foxprolinks.htm

 

[craigber]

Doh! I reread the question and got a different meaning from it than the first time.

The big question is why do you thing you need direct access to the data? I can't imagine that your needs are any different that hundreds of other applications that run on Oracle or SQL Server.

Craig Berntson
MCSD, Visual FoxPro MVP,

http://www.craigberntson.com

Author, CrysDev: A Developer's Guide to Integrating Crystal Reports"

 

[Mike Lewis]

USMA99,

Could you clarify .... What do you mean by "open large tables directly into RAM"?

Do you mean you want to copy the entire table from disk to RAM, and then open it from there? Why would you want to do that? Or are you under the impression that VFP opens tables in the same way that Word opens document files - by reading the whole thing into RAM and operating on it there?

In general, there's no relationship between the size of a table that VFP can process and the amount of RAM installed.

Certainly, installing a RAM disk is not going to help. By reducing the amount of available RAM in that way, you'd simply be increasing the amount of data Windows pages out to virtual memory. And, as Dave says, you haven't really got any control over that anyway.

Mike




__________________________________
Mike Lewis (Edinburgh, Scotland)

My Visual FoxPro site: www.ml-consult.co.uk

 

[Imaginecorp]

(For example, simultaneously open 3 2GB files which total 6GB) I'm unclear on whether or not the 32-bit VFP9 program needs to worry about managing the memory or if the OS does all of that.

A little lost about the RAM part too, but can you open 3+ 2GB tables simultaneously… Sure we open over 20+ table each approaching 2GB etc. But a little programming is involved such as clearing the Cache and believe it or not Putting a limit on the memory VFP can use, so 8 – 12 GB of RAM is of no use

If you mean run a query which will return a result set larger than 2 GB… I don’t think so

 

[USMA99]

To a couple of your points:

Our needs are that we run the dbf through a software package. (It qualifies records to meet USPS deliverability requirements and add's ZIP+4 data, etc) It's niche to our industry but a pretty first-rate and commonly used package. It parses the data within certain fields, analyzes the data and then creates an output file with modified data or appended fields. It also accesses other large reference tables. We cannot access the data in a truly relational setting such as SQL Server/Oracle.

Yes, I wanted to copy the tables right into RAM, I realize we don't have a temp file like MS-Word would for example. I will admit, I don't fully understand how Fox & memory work together. I thought perhaps Fox itself is limited to managing 4GB total of RAM as it is 32-bit and having a 64-bit OS with 12GB or RAM might not help as Fox couldn't take advantage of it. If Fox can use the RAM, I'll try loading tables into RAM first. (maybe this isn't something you can even ask Fox to do, I'm not sure)

In any event, in my past I've tested a RAMdrive situation and found that running this software package on tables located on this RAMdrive (taking RAM and pretending it's a hard-drive) ran like greased lightning. From that I thought, hey, if I can open a few large tables in RAM (via a RAMdisk situation) then with the 64 bit OS I'd be set, I wouldn't be limited to what Fox can do itself. I want to open a few 2GB tables simultaneously (purely within foxpro now) and have them completely working in RAM instead of going out to disk (although I'm thinking as far as Fox knows they'd be on disk).

 

[Imaginecorp]

If I understand your requirements correctly, what you are trying to do is pretty simple and is done all the time. Whether you put your tables on a RAM drive or not is not the issue, VFP will Not read or handle a Native VFP table which is more than 2 GB Data and 2GB Index. You can open as many 2GB tables as you like, it make no difference to VFP and if the tables are optimized, the processing will be extremely fast.
Though a point to be noted, VFP does not thrive on unlimited memory, it’s actually a memory hog and will try to utilize all the memory it can get and like a glutton will crap out. The worst part is the error message will say “Not Enough Memory to File map…” etc. This is misleading. You Must limit the memory to under 618 MB if running on a machine with over 2+GB of memory. This is done through a program or the command window before the program can run (=SYS(3050,1,618000000)
I am sure somebody in here can add to my suggestion…

 

[USMA99]

Just to reiterate for everyone who keeps bringing it up, I'm very clear that I can't open a single table larger than 2GB. I know this is an internal fox issue and not a RAM issue.

To paraphrase the last poster: VFP is a memory hog and will grab all of the memory that it can. To me, this is actually what I desire if I'm reading you correctly. This means that the total amount of RAM in-use by FoxPro would not be limited to 4GB just because it's a 32-bit application. It would be able to utilize more RAM if I wished to throw RAM at it (which I'd like to try).

Limiting memory to 618MB seems a little odd when compared with our experience. (also, note we're discussing VFP) We routinely have more than 2GB of memory on our workstations, 4GB in fact, and have nice snappy performance that way with Fox. (the full 4GB is not available since we're at 32bit, but close enough for sake of discussion) I am concerned by some comments including the "Not Enough Memory to File Map," you aren't saying that performance *decreases* with additional memory in that Fox doesn't know how to manage it are you?

 

[USMA99]

That last most had a typo, it should have read "... also note that we're discussing VFP version 9

 

[craigber]

Fox may not be able to take advantage of 4 Gig even on a 64 bit OS. I don't know this for sure, but my hunch is it won't use that much. At some point, even using files from a RAM drive, Fox's internal data cache will fill up and it will start swapping to where its TMPFILES setting is pointing to. By default, that is the physical disk, but you could create a RAM disk and point temp files there.

Craig Berntson
MCSD, Visual FoxPro MVP,

http://www.craigberntson.com

Author, CrysDev: A Developer's Guide to Integrating Crystal Reports"

 

[Imaginecorp]

I am concerned by some comments including the "Not Enough Memory to File Map," you aren't saying that performance *decreases* with additional memory in that Fox doesn't know how to manage it are you?

Yes: In our experience, VFP9 or earlier does not know how to handle it effeciently and if you do not limit the memory you will get the error message.
Even With 4 - 12 GB The performance may not decrease but it will Not improve either.
Though I have not tried it on a 64 OS...

 

[Imaginecorp]

I should add: the above scenerio is with large tables with millions of records and over 1 GB in size.
It does not apply to files in the MB catagory.

We routinely have more than 2GB of memory on our workstations, 4GB in fact, and have nice snappy performance that way with Fox.

Sure: VFP is the fastest that there is, but if you check the windows Task manager when VFP is processing on the desktop, I am pretty sure under "process" it does not exceed 512MB...If you want it be even more "snappy" limit the memory to 512 with the SYS(3050) function and optomized tables...

 

[USMA99]

Thanks for the input all. If anyone has first-person experience on the 64-bit & RAM question please let me know. I'll post what I find when I test this but it won't be for some months.