What is the fastest looping within a file???

[agit72]

To all expert..
What is your suggestion if I have the condition
to looping withing a file?
What should I use? is for... next, scan.. enscan
Please help, I neet to loop a file about 500.000.000 records.


Agit Permana (08561052915)
Mitra Solusi Pratama, PT

http://www.msp-works.com

 

[Mike Lewis]

Agit,

I don't think it will make much difference between FOR/ENDFOR and SCAN/ENDSCAN. My guess is that SCAN will be slightly faster, simply because you will have a couple of lines less code to execute, but the difference (if any) will be pretty small.

A more important question is: Why do you want to loop through such a large number of records in that way? Are you seaching for records that meet a specific condition? If so, creating a Rushmore-optimisable query might be a better way to go.

Mike


Mike Lewis
Edinburgh, Scotland

http://www.ml-consult.demon.co.uk

 

[tesar]

Agit72,
interest structure of file for 500 000 000 records...
(think max 3 byte length of record
+ recno() as additional field ?)
Consider file placement in alone hard disc,
if not, from time to time use Norton Speed Disc
or similar program...
Tesar

 

[mkrausnick]

Tesar's suggestion is very important - the program will run much faster on a local hard drive than if it is stored on a network volume, regardless of what language construction you use (FOR...ENDFOR/SCAN...ENDSCAN, or even DO WHILE !EOF() and SKIP).

The other consideration is indexing. If you will run this many times, then definitely, as MikeLewis says, you should look at creating an index to limit the number of records to process. However, if you really do need to read and process all the records, then DO NOT index the file - it will only slow down the scan.

In any case, 500,000 records is not really very many records for Foxpro to process. It shouldn't take that long - a few hours at most, depending on the speed of the computer and how much processing is to be done on each record.

Some of my Foxpro tables have 2 million records or more, and on the rare occasion that I have to process the whole table, it never takes more than an hour.







Mike Krausnick

 

[baltman]

It isn't so much a question of which is faster, it is a question of what is faster for what you need to do.

SCAN is supposedly fastest, but if you need to have a concept of a counter, then a FOR can be quicker.

If you need to *do something* to the entire file in the middle of your logic, you'll likely find yourself at the end of the scan loop without having executed the code you meant to.

e.g. this works fine:

scan
repl field1 with iif(field2=3,9,0)
repl field3 with iif(field2=2,9,6)
endscan


And this one never gets past the the 1st record because the 2nd line of code takes you to the end of the file
scan
repl field1 with 9 for field2=3
repl field3 with iif(field2=2,9,6)
endscan

If you post a sample of your data/layout and explain what you're trying to achive, I'm sure TT can help further.

Brian

 

[AirCon]

Agit72,

I'm a bit confuse with your records number. Is it 500 million or 500 thousands ? Cause I saw it as 500 million but mkrausnick figured it as 500 thousands.

If it is 500 thousands, I agree with MikeLewis. Go with query will be better. But if it is 500 million, I suggest you start to think to upgrade into SQL server. See this figure:

If you have 5 bytes (chars) for each record:
(500 mill * 5) / (1024^3) = 2.33 Gig

I know that max records is 1 billion but the figure already exceed the VFP max table capacity (2 Gig). Are you sure about the number ??


-- AirCon --