Sometime PERL only manages to partially write a file? Help! 1

[NeilF2]

I have a simple program that basically reads in say 15,000 lines and prints them back out again (eg: "PRINT DF $rec" in a loop)...

Over 99% of the time it works an absolute treat, but every now and then it seems to go wrong and only manage to right back a fraction of the records (eg: only 5,000 odd), therefore losing all the records towards the end.

It's almost as if part way thru writing out the new version of the file it goes wrong/crashes!?!?

Now the software is being run on a Unix box by hundreds of people so I do not see the problem happening myself, just the result. Ie: The fact that a massive number of the latter records are lost from the file.

It is likely that when PERL is dealing with a large (over 5 meg) file in memory it may go wrong randomly and therefore while writing the adjusted file out, simply give up?

Could this account for my partitially written back file?

Or is PERL rock solid and the problem must be in my code. My code is pretty simple and works hundreds of times a day with no problem though.

Thanks!

 

[MikeLacey]

It's difficult to describe *any* software as rock-solid - but Perl's "not bad".

Re dealing with large files, I've not come across the problems you describe.

Are you creating any temporary files or a file that is unique to the time that the script is run? How are these files named?
Mike
"Experience is the comb that Nature gives us after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[NeilF2]

No, I'm simply doing something along the lines of...

open(DF,"file.txt"
while($data_line=<DF>){push(@database_rows, $data_line);}
close DF;

I Adjust the records in DATABASE_ROWS and then...

open(DF,&quot;>file.txt&quot;
foreach $record (@database_rows){print DF $record;}
close DF;

This is basically all I am doing. One time recently I had over 14,000 lines (6meg+), and ended up with only about 4,000 lines afterwards...

The software is being run online by hundreds of people and just every now and then (less than once a month) is doesn't manage to write all the lines back again

 

[nexus1001]

You should try to use autoflush

 

[i2mc]

You say &quot;the software is being run on a Unix box by hundreds of people&quot; - sounds like a possible file locking problem? Perhaps your locking scheme is faulty and 2 processes are writing to the file at the same time?

Do you check the return error codes for ALL your file operations - including close? Your code sample isn't even checking the open which is a recipe for disaster.

Roger

 

[NeilF2]

I check my OPEN statements for errors, and I lock during all updates using calls to the following subs:-

#--------------
# Locking on
# Try for 14 secs, else overwrite it - Makes crash proof!
#--------------
sub lock_on{
$lock_file=$_[0];
for($i=1;$i<=14;$i++){if(-e $lock_file){sleep 1;}else{last;}}
open LOCK,&quot;>$lock_file&quot;;print LOCK &quot;lock&quot;;close LOCK;
}

#--------------
# Locking off
#--------------
sub lock_off{
unlink($lock_file);
}


Surely if I get an error with a CLOSE I really am f****ed? ie: If I'm writing my updated file and the CLOSE fails then surely it's already screwed in someway?

Thanks for you help!

 

[NeilF2]

I've just had a thought...

When people update the file I do lock it... But people can view data from the file (ie: not update it) at ANY time. There is NO locking around this reading.

If someone is writing to the file (eg: file.txt) using:-
open(DF,&quot;>file.txt&quot;
foreach $record (@database_rows){print DF $record;}
close DF;

and someone else is reading in the file using:-
open(DF,&quot;file.txt&quot;
while($data_line=<DF>){push(@database_rows, $data_line);}
close DF;

Would that have any effect on the first persons update/reading of the file?

The error I'm getting is very rare so a timing issue cuold explain it?

 

[i2mc]

A close failure often (but not always) indicates a disk space problem (which of course could be temporary and hence why your code may only fail once a month).

The thing to do is to check the return values from ALL your file system commands and record any errors in a log file if you don't want the user to be aware of them. You can then analyse your log file and see when / where the problem is.

Checking the return value also lets you write more resilient code - for example, you could do a similar thing to your file locking in that if any of your updates detect a failure then you could pause for a while and then try to write the whole DB again.

Another approach would be to ask the system to send you an e-mail when it detects a failure so you don't have to keep monitoring the log file for an error that happens rarely.

HTH
Roger

 

[i2mc]

&quot;Would that have any effect on the first persons update/reading of the file?&quot;

YES.

If you are on a unix system - why not use flock which allows for shared locks (read) and exclusive locks (write) etc. If you are using file locking then obviously every read / write request has to acknowledge the lock otherwise things can go pear shaped!

Roger

 

[NeilF2]

I currently DO NOT have &quot;$|=1;&quot; anywhere in my coding...!?!? Should I? Could this be the cause?

 

[i2mc]

<I>I currently DO NOT have &quot;$|=1;&quot; anywhere in my coding...!?!? Should I?</I>

Are you writing a CGI script? If so, $|=1 allows data to be flushed out to your client's browsers faster (depending on the httpd server, apache on win32 doesn't support this properly yet).

It isn't relevant to your problem though.

Roger

 

[NeilF2]

i2mc - I think you're wrong...

You say the fact I'm reading the file while others may be writing it could screw it up.

I've done a couple of tests this does not seem to be the case. I've got two browser windows. One writing the file (with a sleep statement between each PRINT), and another window running another program to read it in and show it the screen.

No matter how many times, I run the READ program while the WRITE program is running, it never screws the file up.

Note: Under certain conditions the READ program can return a blank file, but the WRITE is always successful and the resultant file is intact and not missing any lines.

 

[i2mc]

&quot;Note: Under certain conditions the READ program can return a blank file, ...&quot;

and hence if it (your program) then goes on to process the file and write the new (blank) one back to disk you now have a disappearing database. I think you've just proved me right, not wrong ;-)

Anyway, not going to argue with you - if you don't process error codes that the OS is providing you for very valid reasons then you are asking for trouble, period. Read the Perl man pages for flock as well.

Roger

 

[NeilF2]

i2mc - No you misunderstood...

When the file is updated. ie READ records -> update -> WRITE records, it IS locked... (There is a LOCK around the whole procedure)

But during this people can READ the data. Under tests the this can result in an empty read. So, this does NOT explain my half created file.

 

[MikeLacey]

Actually - the file can be written to even when there's a lock on it. Unix locks control locking, not file access. It's possible to write a program that writes to a file even if the lock attempt fails. Mike
&quot;Experience is the comb that Nature gives us after we are bald.&quot;

Is that a haiku?
I never could get the hang
of writing those things.

 

[jewel464g]

Hi Neil,

I had the same problem on a linux box. I was told to use autoflush and it fixed my problem.


use FileHandle;
autoflush FILE 1;


Jewel

http://www.geocities.com/jewel464g/

When faced with a decision, always ask, 'Which would be the most fun?'

 

[missbarbell]

<I>&quot;apache on win32 doesn't support this properly yet&quot;</I>

Roger, If only I'd seen this last week. I was running round in circles trying to figure out why my flushing of a continual data stream waiting until the </HTML> tag before displaying.

Thank you.

Barbie. Leader of Birmingham Perl Mongers

http://birmingham.pm.org