Low Level File Functions

[ARS1]

I have a coma delimited file that need to navigate thru and read data. There are 1,454,701 records and the total size returned from FSEEK(gnHandle, 0, 2) = 535,84,229

I need to locate about 170k records 3/4 of the way down the file and use it to create and populate a data table.

I can read the data from the first record using FGETS. How do I navigate thru the the rest of the records and make my way to EOF()?
Thanks!

 

[mmerlinn]

[ ]

It has been a while since I did low level scraping and I can't seem to find my documentation on FGETS().


However, if I remember correctly all you need to do is to put FGETS() in a loop and run the loop until you get what you want. FGETS() moves the record pointer to the beginning of the next chunk of data in the file.

mmerlinn

http://mmerlinn.com

Poor people do not hire employees. If you soak the rich, who are you going to work for?

"We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding. Answering questions for careless and sloppy thinkers is not rewarding." - Eric Raymond

 

[Mike Lewis]

Mmerlinn,

You're close, but not quite there. It's not FGETS(). It's FSEEK().

Ars1,

In brief, you need to calculate the approx number of bytes from the start of the file to the point 3/4 of the way through. You pass that number as the second parameter to FSEEK(). You then FREAD() the required number of bytes.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro tips, advice, training, consultancy
Custom software for your business

 

[Olaf Doschke]

FGETS() get's you through line by line. You can use FSEEK() to position anywhere, that's true. One FSEEK() to about the line you want and the next FGETS() will get from there to the end of the line, then read on full lines.

Bye, Olaf.

 

[MrDataGuy]

For good or evil it seems that there are at least a dozen ways to solve any issue with VFP.

When I first read your (ARS1) post I said to myself that the way to solve this is to read all of the file into an array and work with the array. (e.g. Then I read the solution again and realized that there was 1.5 million rows of information and VFP arrays are limited to ~64,000 elements (or has that limitation been lifted??)

Anyhow how about creating a temp cursor, reading in the information and working with that? (Low level functions may be faster, but I prefer what I think are clean and simple solutions when speed is not a concern, e.g. get it to work first, then only if needed make it fast.)

Example, say the data to read is:

1,a,120
2,b,232
3,c,345
4,d,465
5,e,567

Then the code to work it could be:

CREATE CURSOR qTemp ( PK I, Type_Code C(10), Amount I )
APPEND from c:\Temp\Data.CSV TYPE CSV

INDEX on pk TAG PK

BROWSE normal

Notice that an index can be added. Also notice the index is added AFTER the data is read, otherwise the computer may spend a huge amount of time reading in the data.

 

[cbasoz]

You move with fseek() and read the lines with fgets(). However, what you are describing is best done with "append from" as MrDataGuy said already. I jumped in to point out you can add a 'for' clause to append from to 'skip' the records if you have some criteria. ie:If you had a text file like:

id,fname,lname
1,john,doe
2,frank,smith
id,fname,lname
3,frank,doe

and you want to get lines that start with an integer:

create cursor myCursor (f1 i,f2 c(10), f3 c(10))
append from myFile.txt for f1 > 0

fgets() also have a drawback of reading up to 8192 bytes which may be too few for a record size (it generally is too much but thinking a record size could be 65000 it is few). Append from OTOH doesn't have such limitation, faster does the parsing for you.


Cetin Basoz
MS Foxpro MVP, MCP

 

[MrDataGuy]

Cetin Basoz,
good catch with the FOR clause I forgot about that.

I was wondering if the use of the FOR would be a speed issue. That is the figuring out which rows should be added might be a time consuming command. I wipped up a 1.5 million record CSV file ( 26 meg on my disk drive ) and tried reading the file into a cursor with and with out a FOR clause. Time 1.3 seconds with no FOR 1.4 seconds with a FOR. So no significant speed issue (And this is from the guy that said do it right and then do it fast <smile>)

 

[mmerlinn]

[&nbsp;]

For good or evil it seems that there are at least a dozen ways to solve any issue with VFP.

That is why FoxPro is one of the most powerful relational database languages around - It can be used to do almost any kind of data manipulation without any add ons. If one way won't easily solve a problem, there are many other ways available within the language to solve them. No need to go outside of the language.

mmerlinn

http://mmerlinn.com

Poor people do not hire employees. If you soak the rich, who are you going to work for?

"We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding. Answering questions for careless and sloppy thinkers is not rewarding." - Eric Raymond

 

[cbasoz]

But although that saying goes a lot in VFP community it is also true for many other languages, like for all the .Net based languages - and they are a lot in count To be honest, a task like this would be easier in a .Net language like F#/C# (especially using the ultimate powerful integrated query - Linq). For example consider a delimited file created like this:

SELECT ;
  cust_id,company,contact,city,country,maxordamt ;
FROM customer ;
INTO CURSOR xx nofilter
COPY TO c:\Customer.txt TYPE DELIMITED WITH ""

Now from within this delimited file you want to create a data structure and get the data where country is "USA":

var myData =
  File.ReadAllLines(@"c:\customer.txt")
  .Select (f => f.Split(','))
  .Select (f => new {
     CustomerID=f[0], 
     Company=f[1],
     Contact=f[2],
     City=f[3],
     Country=f[4],
     MaxOrderAmount=Decimal.Parse(f[5])})
  .Where (f => f.Country == "USA");

and this code benefits from intellisense! Isn't that cool
One of the many ways to do that in .Net and easier IMHO.

Cetin Basoz
MS Foxpro MVP, MCP