thoughts on file saving safety

[CADTenchy]

My app will be saving a growing text file to hard disk regularly.
This could be every 10 or 20 minutes or so, up to a maximum frequency of about once a minute. It will be used anywhere between 4hrs and 24hrs continuously when in use, but that might be once a week probably at most.

My data is held in a stringlist and I'm using

stringlist.SaveToFile('filename.txt');

to do the saving.

Plan A was just to use the line above each time a save is required.

Plan B is to do this at each save:

if exists filename.bak then delete filename.bak;
rename filename.txt to filename.bak;
stringlist.SaveToFile('filename.txt');

Which would you think has best integrity, all in all, given the save frequency?


Steve (Delphi 2007 & XP)

 

[Griffyn]

Clearly the second option that creates a backup has more integrity. You should obviously trap the SaveToFile line within a try..except block that restores the .bak file you just created in case of failure, along with notifying whatever that a save failure occurred.

 

[CADTenchy]

would this be a good offering?

[b]try[/b]
   CopyFile(PChar('fileName.txt'), PChar('fileName.bak'), False); [green]// False to overwrite 'fileName.bak' if already exists[/green]
   stringlist.SaveToFile('filename.txt');
[b]except[/b]
   on E : Exception do
   ShowMessage(E.ClassName+' error raised, with message : '+E.Message);	
[b]end;[/b]

Steve (Delphi 2007 & XP)

 

[DjangMan]

Is there any reason why you can't keep more than one backup?

 

[harebrain]

Is there any reason you aren't just doing sequential file output?

 

[CADTenchy]

Djangman- I guess not. I could keep two or three, but not get carried away..

harebrain- is there any better integrity with sequential file output?
Would a failure to save at crucial point be just as catastrophic in terms of data lost (the file would be corrupted) ?


Steve (Delphi 2007 & XP)

 

[roo0047]

Steve - Since you're being asked why not more than one and why not sequential, I thought I'd offer my solution as it applies to a log file. Rather that have numerous backups of either, I append the last run output (MyLogFile.txt) to an archive file (MyLogArch.txt). Rere's my code:

const
  MyLogFile = 'C:\Program Files\QCMenu\MyLogFile.txt';
  MyLogArch = 'C:\Program Files\QCMenu\MyLogArch.txt';

procedure TQbInvForm.ArchiveLogFile;
var
  fIn, fOut: textfile;
  s: string;
begin
  if FileExists(MyLogFile) then begin
    try
      assignFile(fIn, MyLogFile);
      reset(fIn);
      assignFile(fOut, MyLogArch);
      if FileExists(MyLogArch) then Append(fOut) else rewrite(fOut);
      while not eof(fIn) do begin
        readln(fIn, s);
        writeln(fOut, s);
      end;
    finally
      CloseFile(fIn);
      CloseFile(fOut);
      if not DeleteFile(MyLogFile) then
        showmessage('Unable to delete file ' + MyLogFile)
    end
  end else
    showmessage('File: ' + MyLogFile + ' not found!')
end; //ArchiveLogFile

procedure TQbInvForm.FormCreate(Sender: TObject);
begin
  Memo1.Lines.Clear;
  ArchiveLogFile;
end;

procedure TQbInvForm.FormDestroy(Sender: TObject);
begin
  Memo1.Lines.SaveToFile(MyLogFile)
end;

As you can see, ArchiveLogFile, which deletes the old one if archived successfully, is called on FormCreate and the new log-file is created on FormDestroy. As you've said, "yet another way to skin a cat."

Roo
Delphi Rules!

 

[harebrain]

is there any better integrity with sequential file output?

Repeatedly saving a growing stringlist, provided it's growing at the end, is just wasteful of system resources. The longer your program runs, the more time it will take to maintain the list and save the file.

Would a failure to save at crucial point be just as catastrophic in terms of data lost (the file would be corrupted) ?

My experience is that if the file is not properly closed for whatever reason, the worst thing that happens it that some record(s) (or a partial record) remained in the buffer, never having been written. That's what the close statement is for: to flush the buffer. Corruption, in the sense that the file becomes unreadable, doesn't happen. But if you require 100% capture, you can open/write/close for each write, although that's going to be a drag on performance, too.

 

[CADTenchy]

Thanks guys for more input. I think I may reconsider using sequential file output.

My requirement is slightly different to Roo's, I certainly wouldn't be comfortable with saving on Formdestroy. In my case the application is running once, continuously, until job is done. If the data gets lost it's lost forever, one time shot at getting it.
I will have another application for Roo's code though, further down the line.

The PC's my app runs on will be a mixture of normal environment and in the field (literally field often), and whilst when in the field odds are it will be a laptop (and thus integral 'UPS' if power fails as is likely) they could be older flakier laptops. Loss of the data for the user will bring tears.

In terms of resource use as I have it now, the file to save will start as one line of text, no more than 100 or so characters, and grow to probably never much more than 1000 lines.

Would this fall into light resource use per chance you think?


Steve (Delphi 2007 & XP)

 

[roo0047]

Good point! Every situation is different. Output should always be configured to fit each situation. For mine, the app exports invoice data from an in-house Delphi/SQL app to QuickBooks. The export is run monthly. The log lists all invoice numbers that were exported successfully and also lists those that failed, for whatever reason. Typically, the log file is ~9000 KB. The apps been run monthly since 02-2007, so the archive file is now ~164,000 KB. Disk space has not been an issue but performance is terrible, as writing to QuickBooks always is. My critical data has already been written when the log-file is saved. When accountants can't find something in QuickBooks, that archive has proved to be most useful. For my "situation", it must be in OnDestroy since so many other functions can write to the Memo strings (lines) right up to just before it ends. However, loss of the log-file would not be critical.

My suggestion was for the Append method, not really where to put it.

Regarding "wasteful system resources", it's only wasteful if you put things in the log file you don't really need to keep.

I use log-files a lot. They can aid in involvement of program development. Example: I was logging a lot of these

03/16/2007 10:33:24 Error adding invoice [99999] for customer [ABC Co.]

Turned out that the customer name was not in QuickBooks. So I created a routine to add the customer info to QuickBooks if not found.

Sorry if I've gotten too far off topic. Hopefully it's information someone finds useful.

BTW: I'm using LedgerLink to talk to QuickBooks. If anyone knows of something faster, I'd sure like to hear about it!


Roo
Delphi Rules!

 

[harebrain]

Well, if your program is already written and max data output is 100k chars, I wouldn't re-write it. I'd just think about doing things differently in the future.

As for data safety, you might consider writing to two devices simultaneously, such as a USB stick. That USB stick wouldn't lose data if there was a power drop, and in the field wouldn't be so susceptible to mechanical failure.

 

[whosrdaddy]

CADTenchy,

sequential file writing in your case is indeed the way to go. since you are logging continously, it is a good idea to write the data now and then to disk, so that in case of failure the data loss will be minimal.

I use a threaded log mechanism in all my applications, mainly for debugging purposes. but it would also suit your case.

to keep things easy, I use a TTimer in this code sample to simulate the threading part. I created a seperate class so you can reuse it in other applications...

unit LogUnit;

interface

uses
  SysUtils, Classes, ExtCtrls;

type
  TLogger = class(TObject)
  private
    { Private declarations }
    Timer    : TTimer;
    StrList  : TStringList;
    FLogFile : string;
    procedure TimerEvent(Sender : TObject);
  public
    { Public declarations }
    procedure Add(Str : String);
    constructor Create(LogFileName : string);
    destructor Destroy; override;
  end;

implementation

{ TLogger }

procedure TLogger.Add(Str: String);
begin
 StrList.Add(Str);
end;

constructor TLogger.Create(LogFileName: string);
begin
 FLogFile := LogFileName;
 StrList := TStringList.Create;
 Timer := TTimer.Create(nil);
 Timer.Interval := 1000; // write to logfile every second
 Timer.OnTimer := TimerEvent;
 Timer.Enabled := True;
end;

destructor TLogger.Destroy;
begin
  Timer.Enabled := False;
  // flush any text that's left in the stringlist
  TimerEvent(Self); 
  FreeAndNil(Timer);
  FreeAndNil(StrList);
  inherited;
end;

procedure TLogger.TimerEvent(Sender: TObject);

var FStream : TFileStream;

begin
 if StrList.Count > 0 then
  begin
   // open the file with sharing, this means that the user can read and write the logfile, change according to your needs
   // (replace fmShareDenyNone with other value like fmShareDenyWrite and so on)
   // create logfile if doesn't already exists
   if not FileExists(FLogFile) then
    begin
     FStream := TFileStream.Create(FLogFile, fmCreate);
     FreeAndNil(FStream);
    end;
   FStream := TFileStream.Create(FLogFile, fmOpenReadWrite or fmShareDenyNone);
   try
    // append text from the stringlist to our logfile
    FStream.Position := FStream.Size;
    FStream.Write(StrList.Text[1], Length(StrList.Text));
    // empty stringlist for new data
    StrList.Clear;
   finally
    FreeAndNil(FStream);
   end;
  end;
end;

end.

save this code as logunit.pas

to use it:

unit u_frm_main;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls;

type
  TFrm_main = class(TForm)
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
  private
    { Private declarations }
    Logger : TLogger;
  public
    { Public declarations }
    procedure LogSomething;
  end;

procedure TFrm_main.FormCreate(Sender: TObject);
begin
 Logger := Logger.Create('c:\test.txt');
end;

procedure TFrm_main.FormDestroy(Sender: TObject);
begin
 FreeAndNil(Logger);
end;

procedure TFrm_main.LogSomething;
begin
 Logger.Add('testing 123...');
end;

so Steve in you case you would this

memo1.lines.add(LogLineStr); // visual output
logger.add(LogLineStr); // write file

keep in mind this unit is provided as an example (and it is far from complete) but I hope it will help you to go in the right direction...

Cheers,
Daddy

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[Glenn9999]

Am I not understanding this entirely? Couldn't text lines simply be appended to the end of the current log file (and flush/close after each write) instead of rewriting it each time as it seems suggested here?

----------
Measurement is not management.

 

[whosrdaddy]

Glenn, isn't that what i'm doing?

/Daddy

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[harebrain]

>>isn't that what i'm doing?

In a word, no.

With the volume (low) and frequency (few and far between) that the OP suggests, your approach takes a sledge-hammer to gnats.

Open for append.
Write.
Close.

Simple.

 

[whosrdaddy]

harebrain, agreed (I like slegde-hammers) , but even this simple problem can be challenging:

- what if the disk is full (USB stick,...)?
- what if the maximum filesize has been reached? (FAT,..)g
- what if the disk crashes (but not the OS)
- and so on...

like I said, I have a complete multithreaded solution for this sort of problems, even if it means LOW volume logging.
I REUSE my object in a lot of applications/services, that's my point.



/Daddy

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[CADTenchy]

Thanks for the code daddy, and thanks again for pointing out the other checks to be made, which I'd completely overlooked!

Off to google for some code snips now..


Steve (Delphi 2007 & XP)

 

[CADTenchy]

Kind of final update...

I am going to rewrite my saving code to append the extra line.
Coupled with the idea which daddy germinated (and so far I'd missed completely) of usb sticks,(which I'm keen on,my apps are all stored on stick to commute between work and home) and the resource saving (intended pun) as sticks can be sluggish, I think it's the right thing to do.

X heads (where x>0) are much better than 1!

Steve (Delphi 2007 & XP)

 

[harebrain]

- what if the disk is full (USB stick,...)?
- what if the maximum filesize has been reached? (FAT,..)g
- what if the disk crashes (but not the OS)

All are trappable errors--which is how you handle them.

Kind of disingenuous to pose those questions as unsolved.

I'm returning to the OP question. An app that creates a log entry less often than once per minute, with 1000 max entries is not very demanding. You may rightfully be proud of your creation (GFY) but I still think in this case that your solution is overkill. I've seen a lot of simple applications made too difficult and too ponderous by some programmer's delusions of grandeur. KISS!

And a lot of foreseeable problems (disk/stick full) can be prevented the way we did it when we only had 360Kb floppies: procedures for using the program. ("Before you start the program, make sure you attach a formatted memory stick with at least 256Kb free space." Etc.) If the data is important to the user, he'll follow the procedures. If he doesn't, "Don't come crying to me!

 

[whosrdaddy]

*sigh* I give up, you have the last word on this, as you still don't get the point I'm trying to make here...



-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!