What is the most reliable way of updating a test file?

[NeilFawcett]

I'm periodically getting corrupted records from my perls scripts. They are blank, etc. I recon this could be down to when the system hiccups/crashes and my perl scripts happen to be running at the time...

Is this the most reliable/best way to process a text file?

#!/usr/bin/perl
use Fcntl qw(:DEFAULT :flock); 
print "Content-type: text/html\n\n";

sysopen(AC,"test.txt", O_RDWR | O_CREAT) or die "message";
flock(AC,LOCK_EX) or die "message";

@recs=<AC>;

#Update @recs in someway 
seek AC,0,0;
print AC @rec;

truncate(AC,tell(AC)) or die "message";
close(AC) or die "message";

Can anyone suggest anything that can improve on this and make it any more reliable?

 

[PaulTEG]

Neil,

Why don't you use simpler filehandling?
read a file, review the content, output the update, and rename the file over the original

open FILE, "<file.txt";
open OUT ">file2.txt";
while (<FILE>) {
  #do something with $_
  print OUT "$_";
}
close FILE;
close OUT
rename "file2.txt", "file.txt"; #this will destroy the original file

BTW what version of Perl are you using because I think flock is implemented by default on later versions.

Slurping the file into an array is a good idea only for smaller files, and especially so on a shared system

HTH
--Paul

 

[NeilFawcett]

First, most of my updates are done by reading my data into an array, updating them and then printing them on a single line again (Eg: print AC @recs). This seems faster than looping around.

My bigger data files tend to be about 3-4meg. Some are only a few K though.


Surely your update there has fundamental issues. ie: If two or more people are attacking the same file at the same time.

Someone could start accessing the old version of the file, before the rename has finished or even taken place. As such you'd need a locking file around the whole process?

If adding that locking would then make your method the best way then I'll do some tests... Had any experience of this?

The update I describe seems pretty robust! I've even killed perl at various stages of update and the file survives (doesn't get blanked). It's also about 40% faster than the standard read in the and write back out to the same file with a lock around the whole process. I suspect as such, it'll also be faster than yours?


I'm trying to be as efficient & robust as possible...

Any other suggestions anyone?

 

[NeilFawcett]

I've just done a test... Running a loop and updating a file over and over, adding and removing entries.

My example look runs twice as fast as the suggested code of writing to a temp file and renaming it. Infact of the four different means of updating a file I've tried, the writing to a temp file and renaming it is by far the slowest.

So unless there are fundamental issues with my method at suggested at the top, it's by far the best solution.

So does anyone see any problems with my file update coding?

 

[ishnid]

Tie::File sounds like what you need. It allows you to access your file as if it were a regular array, doesn't slurp and supports file locking.

 

[NeilFawcett]

That uses SYSOPEN, be it in a lot more of a complicated way

 

[ishnid]

I know but it's easy to use and doesn't slurp.

I would argue that raw speed isn't everything. If you have even 10 users accessing 4Mb files at the same time, you're starting to eat up quite a bit of memory. As your server gets busier, your code's efficiency will deteriorate much more quickly than a slightly slower memory-efficient alternative.

 

[MikeLacey]

BTW what version of Perl are you using because I think flock is implemented by default on later versions.

Really?

Mike

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

It's like this; even samurai have teddy bears, and even teddy bears get drunk.

 

[PaulTEG]

Really? (Mike)

Well I make calls to flock, and I don't implicitly use Fcntl

Now whether it's been that way since the beginning I dunno, it could be another module is importing it downstream, but none of the modules I work with regularly.

Though this works without reporting any proiblems

#!/usr/bin/perl

    open HOOD, ">test.txt";
    flock HOOD, 2;
    print HOOD "1";
    flock HOOD, 8;
    close HOOD;

runs on XP box, with ActiveState5.8.2
--Paul

It's important in life to always strike a happy medium, so if you see someone with a crystal ball, and a smile on their face ...

 

[MikeLacey]

Oh, I see what you mean.

No - flock() is a builtin function which, if I remember right, has been there since at least Perl 3.

When you do this in a script:

use Fcntl ':flock'; # import LOCK_* constants

all you're really doing is importing the definitions of LOCK_SH, LOCK_EX, LOCK_NB and LOCK_UN into your script. Fcntl.pm doesn't redefine flock()

Mike

Want to get great answers to your Tek-Tips questions? Have a look at faq219-2884

It's like this; even samurai have teddy bears, and even teddy bears get drunk.