Managing a flat data file

[Archon]

Though I know it might not be the most efficient way of doing it, I've been instructed to use a flat, pipe-delimited, data file for a project I'm doing. Lets say, for the sake of arguing that the first column of data was a number index. Is there a convenient way with perl to jump down to a variable index number, and extract (or insert) the 'info' as it is in the place below?

1|...|...|..|info|..|
2|...|...|..|info|..|
3|...|...|..|info|..|

In the actual case, I'll be inserting and deleting data from various sections of the data file. But, first things first.

Thanks guys

 

[tsdragon]

The only convenient way to do this is to load the entire file into an array, update the records in the array, and rewrite the entire file. This is particularly true since it appears that the data in each record may be variable length, and the length of a record may change when it is updated. If the "index numbers" are going to be consecutive without gaps, you could actually omit those, since the records will be in an array and you can just index the array (but it will start with 0, not 1). Alternatively, you could load the records into a hash, with the "index number" as the key.

If there is not a lot of records in the file, and if you are not going to be updating it frequently, and if no more than one person is going to be updating it at a time, the flat-file method ought to work OK for you. Tracy Dryden
tracy@bydisn.com

Meddle not in the affairs of dragons,
For you are crunchy, and good with mustard.

 

[Archon]

Okay, I was thinking along the lines of what you said. However, It was the code portion of how to do that I was confused about. What's the easiest way to search through an array for what I want and either extract/insert data into that position?

Thanks for the speedy reply.

 

[goBoating]

First, I suggest that you spend a few bucks on 'Programming the Perl DBI' by Descarte and Bunce from O'Reilly. While it is primarily about RDBMS's, it has a chapter on playing with text files( mapping, using DBMs, using DBI with csv files, and using plain old text). It is well worth the $35 (US),( I think 35, about that, anyway).

It is possible to open a file for both read and write and to jump (seek) to a specific spot in that file, do some stuff and hop out. However, most applications that I have seen using flat files don't do it that way. Figuring out to where you want to seek can be more work than it is worth. Instead, I suggest a more blue collar approach.

open the output file.
flock - lock the output file.
quickly figure out what you want to do and do it.
flock - unlock the output file.
close the file.

Engineer the flocking into any application that uses the file(s). Each app will check to see if the file is flocked, and if necessary, wait its turn. As you might imagine, you will want to keep the flock periods as short as possible to keep the line of waiting processes short.

If flat files are sufficient for your purposes, then this approach suffers from very little overhead and is very simple. I have one web based help request tracking system that is up to about 800 requests. It produces a report of the key words sorted and printed back to a browser in about 2 seconds running on a fairly modest and slightly dated Sun machine. That system uses some XML-ish tags for tagging the data elements. Your situation should be faster since you can use split instead of pattern matching to get at your data elements. Don't for get to take the new lines and carriage returns out of the input before you print to that format.

If you anticipate the eventual need to go to a database server, then you might want to start with the DBI::Stuff for text files. That way, when you make the transition, all you change is the connect string(maybe a few other tweaks) and you are off to the races.

Feel free to ask for more specifics if you need. There are a few previous threads dealing with using flock that you can search for.

HTH


keep the rudder amid ship and beware the odd typo

 

[latch]

There is a provision in the Perl command line options to change the contents of a file without even having to open it.But first here is an example to extract the entire line for a particular index

perl -n -e 'print if /^2/;' <datafile>

This extracts the line with index 2 from your data file.
You can consider using a system call for the above statement or using backticks

Check out the -i option used in conjunction with the -p option....its more powerful.

regards
CM