Pchar into String

[Tilen]

Hi

can someone help me with this conversion please. Or give me advise on doing things differently.

My program reads a text file and the process it. At the moment I use TStringList.LoadFromFile function, to load a text file. Then process it.

This is not good for huge text files.

Now I got an example how to read block by block :

repeat
blockread(srcFile, Buffer^, sizeof(Buffer^), bytesRead);
if BytesRead = sizeof(Buffer^) then
seek(srcFile, filepos(srcFile)-10);
Inc(numReads);
import.Add(pchar(buffer));
until (bytesRead = 0);

Here you can see I load every buffer, size is 64K, to import, which is TSTringList.

Now I get into import for each buffer, 64K of lines, inot each import line. So

import[0]=1st 64K of lines...
import[1]=2nd 64K of lines...

I would like to have

import[0]=1st line from file
import[1]=2nd line form file
...

So I need to convert buffer into each line and get it inot import.

I would like to use

import.AddString(buffer)

but it doesn't work... I try

import.AddString(PChar(buffer)) ... still doesn't work.
(err: Incompatible type ... String and PChar ...)

So I need to convert Pchar into String.

Can anyone help me with this one.

Thanx

 

[Spent]

Here is an example of TByteDynArray, As I can remember I used it on PChar too

var buffer: string;
    a: TByteDynArray;
    b:PChar;

..........
begin
  s:= 'TESTING';
  SetLength(a, Length(s));
  Move(s[1],a, Length(s));
  // should work with PChar to
  Move(s[1],b, Length(s));

Another solution might be if you use
TStringList.Strings[] array to get the Lines of the file just like in the TRichEdit Component

Good Luck

Spent
mail:teknet@mail.orbitel.bg

 

[sggaunt]

I am not sure why you don't wat to use the stringlist loadfromfile, but there is also a Loadfromstream method which may be more efficent for large amounts of data.

You do know that your blockread will not differentiate any end of string markers? and strings could be split up.

What datatypes are 'buffer' and 'import'?




Steve: Delphi a feersum engin indeed.

 

[Tilen]

Hi

import = TStringList
MaxBuffersize = 1024*63;
buffer = array[1..MaxBufferSize] of char;

Would TByteDynArray example be fast enough? I'm importing 500MB+ files, with 10mio+ lines of text.

At the moment I use import.LoadFromFile, and then I process line by line from import.

For files 500MB+ LoadFromFile is not good since my process of the file also takes a lot of memory, and I get OutOfMemory error.

So I decided to read block by block and process what I load from file and then read again and process further, so I don't need to load the whole file into memory and then process it.

If I read line by line with Readln ... 10Mio+ lines... no way

So, now I need to conver the buffer into lines of text, that's why I wanted to use import.AssStrings( ... );

How would I use LoadFromStream?

 

[sggaunt]

Is it the case that your file contains terminated strings
of varying lenghts?

An array of char is not neccasarily a string, and pchar (pointer to char) is not really the same as a string, so you cannot pass either to the tstrings.add method.

Look in the Help file for 'Using streams to read and write data', but you may have a problem if you dont know the lenghts of the strings in your file. The same problem as with reading into fixed length data blocks. readln will work because it is looking for the string terminations.


This is from the Delphi help (not much use I know)

procedure caststring;
var
  fs: TFileStream;
const
  s: string = 'Hello';
begin
  fs := TFileStream.Create('temp.txt', fmCreate or fmOpenWrite);
  fs.Write(s, Length(s));	// this will give you garbage
  fs.Write(PChar(s)^, Length(s));	// this is the correct way
end;

Steve: Delphi a feersum engin indeed.

 

[Tilen]

Hi

I just got it:

ExtractStrings([#13],[' '],pchar(buffer),tempstr);

tempstr = TStringList

So, I get each line to tempstr and I can process them one by one.

I just need to have a logic, to connect broken last lines...

I think this should keep me bussy for a while

thanx a lot guys