File IO

[andjem1]

Hi

I am working on a program to perform various alterations and calculations on CNC Gcode.
The problem I'm having is how to start an operation on an existing file, after the file is opened and displayed in a TRichEdit.
Id like to use 'in memory' processing instead of reading each line of the richedit to try to avoid performance problems.

I have a test procedure which doesn't work here, could someone point out what I'm doing wrong ?

I know its basic but it's got me stumped.
Hope I'm making sense.
Thanks

Andrew.

[b]procedure[/b] TForm1.InsertBlockNumbersClick(Sender: TObject);
Var
    infile, outfile: TextFile;
    inBuffer,outBuffer: [b]array[/b][[purple]1..65536[/purple]] [b]of[/b] char;
[b]begin[/b]
    AssignFile(Infile, FullFileName);
    reset(infile);
    System.SetTextBuf(infile, inBuffer);
    System.SetTextBuf(outfile, outBuffer);
    AssignFile(outfile, FullFileName);
    rewrite(outfile);

    writeln(outfile, [teal]'(Tests InsertBlockNumbers.)'[/teal]);
        [b]while[/b] [b]not[/b] eof(infile) [b]do[/b]
            [b]begin[/b]
              [navy][i]//Do Something
[/i][/navy]            [b]end[/b];
    CloseFile(infile);
    CloseFile(outfile);
[b]end[/b];

 

[towerbase]

I don't really understand your question because your sample code doesn't make any mention of TRichEdit but the best way to populate a TRichEdit component and save the (changed) data in a TRichEdit is to use the LoadFromFile and SaveToFile methods of TRichEdit.

Loading data into TRichEdit is as simple as

richedit.LoadFromFile( 'filename.txt' );

and saving to the file is as simple as

richedit.SaveToFile( 'filename.txt' );

Those two methods load and save the entire file in one step so there are usually no performance problems.

Andrew
Hampshire, UK

 

[andjem1]

Thanks for the reply. Opening and saving the file is not my problem. Once the file is opened I then want to
process the file but without accessing the richedit (like the routine below ), in other words, process the file
'behind the scenes' and update the richedit when finished.
Rereading my post its not at all clear, I'll take a better look and try to pin down what I need.

Thanks

Andrew
Hertfordshire.

//I can use a procedure something like this; but it's slow when dealing with large files..

[b]begin[/b]
  Linecount:= Richedit1.Lines.Count;
  BlockStart := [purple]0[/purple];
  BlockInc := [purple]10[/purple];
 [b]for[/b] i := [purple]0[/purple] [b]to[/b] Richedit1.Lines.Count-[purple]1[/purple] [b]do[/b]
    [b]begin[/b]
     inc(BlockStart,BlockInc);
     Line := Richedit1.Lines[i]; [navy][i]// load current line
[/i][/navy]     NewLine := [teal]'N'[/teal] + IntToStr(BlockStart) + Line;
     Richedit1.Lines[i] := NewLine; [navy][i]// write converted line back into richedit
[/i][/navy]    [b]end[/b];

 

[towerbase]

The killer, performance wise, is using TRichEdit. Can you use TMemo instead?

If yes, then copy Memo1.text to the text property of a TStringList. Prefix each line with the 'N' and BlockStart and then copy the text back to the Memo1.text



Andrew
Hampshire, UK

 

[Glenn9999]

Referring to this prior thread will be useful to those who are participating in this thread.

thread102-1470823

That being said, consistent with the answer you were given in that thread, RichEdit is your performance problem. Which means, having anything to do with it in the application that you are working with and the data that you are working with is going to present problems.

For the data that you are working with (text file, small line lengths, large number of lines), nothing of that nature will work without a performance problem. This includes the TMemo suggestion. This includes Notepad, Wordpad, Word, OO Writer...well anything. As was noted in the quoted thread, for this reason, LoadFromFile is inordinately slow, and all these options load files of the size discussed in the thread slowly.

As was described in the referenced thread, most data processing programs that handle large numbers of records do not make use of any editing facilities for this very reason. It would be time prohibitive to check so many records (I assume the 20000 record figure in the other thread is a baseline for your users?, or is the actual average number not high enough to be a concern in practice?).

Otherwise, do your users enter this data in your application? If so, it would be prudent to spread things out in different controls to enable data editing and control.

So what to do? The best thing would be to do as suggested in the other thread and drop the editing requirement. Past that, if the editing function is an absolute requirement, you have one of two options to try:

1) Hack TMemo to limit the line lengths to about 25 characters or whatever the proper length limit is for each line.
2) Map the file into a binary file and load segments of it up into the TMemo, and set buttons so the user can page through it. Then process the binary file into text when you perform your data processing against it.

Measurement is not management.

 

[Glenn9999]

I got to playing around some with TRichEdit. You learn enough if you think to look into things (on 20000 lines):

The LoadFromFile method took 110 ms
The SaveToFile method took 16 ms.
Using LoadFromFile and altering each line (string := string + 'Altered.' took 80180 ms.

The third line would be consistent with what was seen in the other thread.

So it seems you could also:

3) Put the file into the TRichEdit using LoadFromFile, when processing use SaveToFile to put it to a temp file, and then process against the temp file to get your final product.

Though you have the problem of what to do with the temp file, as well as cleaning it up in the event of a problem in your run.

And yes, in a lot of ways it sounds truly absurd, but it seems TRichEdit is messed up if you try to process against the strings within it.

Measurement is not management.

 

[andjem1]

Hi Glenn999

Thanks to you both for your ideas and explanations. It seems that I have asked a similar question as before although that was not my intention. I am trying to expand the functionality of my app by gaining from the advantages outlined in the other thread.
I would like to have added a number of other conversion routines without the need to, as I see it, reopen and save the file.
Realistically it looks as though I will have to scale down my expectations for this app for the time being, I can still use normal text editing though, which is still useful.
I stuck with the TRichedit because I have also been working on syntax highlighting, which I'm glad to say works like a charm tested on a 2 million line program.

Are 20000 lines really found? defiantly, though its probably the upper end of the scale for the type of operation that would be converted in this way. For some machining operations the files can regularly top 100,000 the largest file I have used was about 1.1million lines. Some can go larger.
"Do your users enter this data in your application?" no, the files would be created in a CAM application and my app converts the Gcode to a different type of machining which may not otherwise be available.

I had also started think (believe it or not) about your third suggestion, using a temporary file to process against. In my simple world it seems the best way for me to go.

So thanks again,

Andrew.

 

[dhaskins]

Why not just read your data into a TStringList and dump the contents onto your RichEdit?

FullFileData : TStrings;
FullFileData := TStringList.Create;
FullFileData.LoadFromFile(WorkingFilename);

I use this method in a program that parses out 100mb data regularly (into subfields in a treeview) and it handles that in just a few seconds. My code also splits the data into chunks so its possible to view on low RAM systems.

With the data like this you can manipulate the data in the StringList then just update the RichEdit. Would that work for you?