Puzzling first char in streamed file 2

[sggaunt]

I am implementing streaming data straight (after ASCII conversion) to file.
But I find when checking the file with notepad that there is an inexplicable first character in every line.
It seems to be related to the number of characters in each CR/LF terminated line.

there are 844 line in the file:-

If the line is 111 chars in length (including the CR & LF Chars) then there is a lower case 'o' in the first position
If the Line is 112 chars its a 'p'
113 a 'q' and the final line is 114 chars and starts with 'r'

Temp := ShortString(TempString)+#13#10;
Fs.writebuffer(Temp, length(Tempstring) + 2);

Where Temp is a short string of max 120 chars, I have found that normal strings cannot be streamed successfully
Tempstring is a normal string.
Fs is a TFilestream object.
Temp looks to be as expected at the point of writing.



Steve: N.M.N.F.
If something is popular, it must be wrong: Mark Twain
That's just perfectly normal Paranoia everyone in the universe has that: Slartibartfast

 

[Prattaratt]

In a short string, the leading byte (i.e. Temp[0]) is a length byte, denoting the number of characters in the string. This is why it cannot be more than 254 chars long. In other words, length(Temp) = Temp[0]; see the following from Delphi's online help: Delphi Short String

 

[Prattaratt]

I meant to type 255 characters, not 254.

 

[sggaunt]

Cheers for that a simple answer, thought there might be

Steve: N.M.N.F.
If something is popular, it must be wrong: Mark Twain
That's just perfectly normal Paranoia everyone in the universe has that: Slartibartfast

 

[majlumbo]

I have found that normal strings cannot be streamed successfully

That is not true, you can stream a string, and read it back.

To write a string to a stream, you first write the length of the string to the stream, then you load the string into the stream, reading is just the opposite, first read how long the string is, use setlength to set the length of the string you will read it into, and then read that many characters.

First assuming you have a stream created

var
   MS: TMemoryStream;
begin
   MS := TMemoryStream.Create
   ...

then you can write a string to it like so

var
   MyString1, MyString2: String;
   LengthOfString: Integer;
begin
   MyString1 := 'xxxxx';
   MyString2 := 'yyyyy';
   LengthOfString := LengthOf(MyString1);//get the length of the string
   MS.WriteBuffer(LengthOfString, SizeOf(LengthOfString);//write the length of the string to the stream
   if LengthOfString > 0 then
      MS.WriteBuffer(Pointer(MyString1)^, ByteLength(MyString1));//now write the actual string
   //repeat for MyString2
   LengthOfString := LengthOf(MyString2);
   MS.WriteBuffer(LengthOfString, SizeOf(LengthOfString);
   if LengthOfString > 0 then
      MS.WriteBuffer(Pointer(MyString2)^, ByteLength(MyString2));//use bytelength rather than length works for both pre/post unicode Delphi
end;

now to read it back, you need to know the structure of the items in your stream (MyString1 comes before MyString2)...

var
   LengthOfString: Integer;
   MyString1, MyString2: String;
begin
   MS.ReadBuffer(LengthOfString, SizeOf(LengthOfString));//get the length of the following string
   SetLength(MyString1, LengthOfString); //set the length of the string you are reading into
   if LengthOfString > 0 then
      MS.ReadBuffer(Pointer(MyString1)^, ByteLength(MyString1);//read correct No. of bytes into the string

   //repeat for mystring2

   MS.ReadBuffer(LengthOfString, SizeOf(LengthOfString));//get the length of the following string
   SetLength(MyString2, LengthOfString);
   if LengthOfString > 0 then
      MS.ReadBuffer(Pointer(MyString2)^, ByteLength(MyString2);
end;

 

[whosrdaddy]

2 things:

@sggaunt: why on earth would one use Shortstring?
@majlumbo: When working with strings, TStringStream seems more appropriate

/Daddy

-----------------------------------------------------
Helping people is my job...

 

[majlumbo]

@whosrdaddy While I agree in general, using a StringStream is not appropriate if you need to save multiple strings, or multiple data types to the same stream. The method I showed allows any type of data to be saved to the same stream, (although non-string types you would not need to precede with the byte length to read/write the data). The only downside is that you need to know the sequence of data items added, so you can read them in the same order as they were written in.

 

[Prattaratt]

The shortstring is the implementation of the original Pascal string type; the OP may be doing something that he wants to be compatible with other Pascal compilers. I generally don't ask why somebody is doing something a certain way unless it is obviously a wrong or dangerous way of doing it, I just assume they have their reasons to be doing something that way.

 

[sggaunt]

In this case, I need something quick and dirty. (quick in terms of development and speed of operation).
I Have library functions similar to the ones suggested by majlumbo, but they didn't work and I didn't have the time to mess about with them.
Pressure is off to some extent now, so I will be looking at above stuff.





Steve: N.M.N.F.
If something is popular, it must be wrong: Mark Twain
That's just perfectly normal Paranoia everyone in the universe has that: Slartibartfast

 

[Prattaratt]

In that case, TStringStream and TSTringList are both good candidates, and they have the advantage of using the default String type which is Unicode in current versions going back to 2009. TStringList has streaming and file functions built in; you just call TStringlist.LoadFromStream and SaveToStream for streaming, or LoadFromFile and SaveToFile for files. Same goes for TStringStream. The advantage of TStringList is that you can retrieve individual lines using it's Strings property.

 

[sggaunt]

Interesting stuff this.
Now the way I understand the concept of streaming, is that when outputting say video the data is pulled out of a buffer as the user watches it, and as soon as the data has been 'consumed' it can be discarded. If the user 'winds back' before the relatively small saved buffer, then the data must be re-streamed (For simple file download and save we don't need to be able to do that).
And the reason why we might want to stream is that we don't have to wait for the entire file to be downloaded, and we don't have to hold the entire file in RAM.

In fact I use the StringList.SaveToFile method in the existing program, incoming data is stored in a stringlist.
When download is complete (up to 16Mb) and the user needs to save this data, Save to file is called.

I don't understood the reason for the Stringlist.SaveToStream Method, as (unless I misunderstand this) it does not allow streaming out at the same time as the list is being populated, so its not true streaming?
To quote the help SaveToStream does the same thing as SaveToFile except the application must create and destroy the File Stream.




Steve: N.M.N.F.
If something is popular, it must be wrong: Mark Twain
That's just perfectly normal Paranoia everyone in the universe has that: Slartibartfast

 

[majlumbo]

@sggaunt If you have a 25 Gig text file (yes I've actually seen them that big), you won't be able to open such a file in a TStringList. As far as I know, there's no option to read just a portion of a file using a StringList. For this type of case, you would use a TFileStream.

Buffering is a completely different matter. Natively, Delphi Streams do not buffer (other that what the OS may provide). If you want a buffered version of a FileStream, the you can look at using the code here.

 

[Prattaratt]

A stream in Delphi and C++ is just a more generic mechanism for transferring information. In Object oriented languages like Delphi and C++, streams allow you to send and receive information from a device, whether it be a keyboard, Display device, File system or IP Socket without having to worry about how to go about sending/receiving it. You just call the Write procedure for the Stream, and no matter what kind of stream it is, the data is sent. It is NOT the same concept as streaming a video from the internet, which is a mechanism where you are using the data received earlier while later data is still being downloaded, as opposed to downloading the entire video before viewing it.