SetLength without #0 initialization

[DavidB3]

Hi.

I'm trying to make this function faster:

//Global variables
Positions: array of Integer;
ItemSize: array of Byte;
AllStrings: AnsiString;
FPosStep = 32;
FPosStepBinPwr = 5;


function TShortStringArray.GetItem(const Idx: Integer): AnsiString;
var
   i, d, Position: Integer;
begin
   if (Idx < 0) or (Idx > (FCount - 1)) then
      raise Exception.Create('Index out of bounds!');
   d := Idx shr FPosStepBinPwr;
   if ((Idx - (d shl FPosStepBinPwr)) >= (FPosStep shr 1)) and (d < High(Positions)) then
   begin
      Position := Positions[d + 1];
      i := (d + 1) shl FPosStepBinPwr - 1;
      while i >= Idx do
      begin
         Dec(Position, ItemSize[i]);
         Dec(i);
      end;
   end
   else
   begin
      Position := Positions[d];
      i := d shl FPosStepBinPwr;
      while i < Idx do
      begin
         Inc(Position, ItemSize[i]);
         Inc(i);
      end;
   end;
   if ItemSize[Idx] > 0 then
   begin
      SetLength(Result, ItemSize[Idx]);
      Move(AllStrings[Position], Result[1], ItemSize[Idx]);
   end
   else
      Result := '';
end;

From the total time SetLength takes 86.1% and Move 13.6%.
On internet some sites are saying SetLength does not initialize with #0, other say it does (using FillChar).
I tested (thoroughly) and in 7/XE5 SetLength always initialize with #0.
But in this code the initialization is not required. It's just wasting time.
Is there a way to modify SetLength so it would not initialize the string with #0 and to work fine in any Delphi version...?
Or is there another function for this...?

Thank you.

Regards,
David

 

[Glenn9999]

I really have no direct answer, but would like to play with this if possible. Is a more fuller implementation available so I can see what is going on myself?

It is not possible for anyone to acknowledge truth when their salary depends on them not doing it.

 

[DavidB3]

My code will be free to be used in freeware applications.
But right now it's not finished and I want to release it only when it's finished.
I wanted to sent you the current version in a PM but couldn't find a way.

 

[Glenn9999]

Well, AddItem (or whatever puts strings into your "AllStrings") should be enough for me to see what GetItem is doing and maybe help with your question.

It is not possible for anyone to acknowledge truth when their salary depends on them not doing it.

 

[DavidB3]

Well, I use many: SetItem, InsertItem, AppendItem, InsertItems, AppendItems, Insert/AppendMultipleItems (multiple items in different locations).

Here is SetItem:

procedure TShortStringArray.SetItem(const Idx: Integer; const Value: AnsiString);
var
   i, d, Position, lv, lDiff, h: Integer;
begin
   if (Idx < 0) or (Idx>(FCount - 1)) then
      raise Exception.Create('Index out of bounds!');
   d := Idx shr FPosStepBinPwr;
   if ((Idx - (d shl FPosStepBinPwr))>=(FPosStep shr 1)) and (d < High(Positions)) then
   begin
      Position := Positions[d + 1];
      i := (d + 1) shl FPosStepBinPwr - 1;
      while i >= Idx do
      begin
         Dec(Position, ItemSize[i]);
         Dec(i);
      end;
   end
   else
   begin
      Position := Positions[d];
      i := d shl FPosStepBinPwr;
      while i < Idx do
      begin
         Inc(Position, ItemSize[i]);
         Inc(i);
      end;
   end;
   lv := Min(255, Length(Value));
   h := High(Positions);
   if lv < ItemSize[Idx] then
   begin
      lDiff := ItemSize[Idx] - lv;
      Move(AllStrings[Position + ItemSize[Idx]], AllStrings[Position + lv], FLength - Position - ItemSize[Idx] + 1);
      Dec(FLength, lDiff);
      if FIsDynamic then
      begin
         if (FLength > FStringsCapacity) or (FLength < Floor(0.8 * FStringsCapacity)) then
         begin
            FStringsCapacity := Max(100, FLength);
            SetLength(AllStrings, FStringsCapacity);
         end;
      end
      else
         SetLength(AllStrings, FLength);
      ItemSize[Idx] := lv;
      for i := Idx shr FPosStepBinPwr + 1 to h do
         Dec(Positions[i], lDiff);
   end
   else if lv > ItemSize[Idx] then
   begin
      lDiff := lv - ItemSize[Idx];
      Inc(FLength, lDiff);
      if FIsDynamic then
      begin
         if FStringsCapacity < FLength then
         begin
            FStringsCapacity := Ceil(1.1 * FLength);
            SetLength(AllStrings, FStringsCapacity);
         end;
      end
      else
         SetLength(AllStrings, FLength);
      Move(AllStrings[Position + ItemSize[Idx]], AllStrings[Position + lv], FLength - Position - ItemSize[Idx] + 1);
      ItemSize[Idx] := lv;
      for i := Idx shr FPosStepBinPwr + 1 to h do
         Inc(Positions[i], lDiff);
   end;
   if lv > 0 then
      Move(Value[1], AllStrings[Position], lv);
end;

 

[sggaunt]

Generally I expect the answer to this will involve getting rid of all the built in functions you are using, (they contain a lot of baggage code that you don't need for a specific task), and making heavy use of pointers to access the strings. Perhaps even some inline Assembler.
These are things that I avoid using myself (horses for courses). But there are/is at least one forum member who can help with this. So watch this space as they say.

Steve: N.M.N.F.
If something is popular, it must be wrong: Mark Twain
That's just perfectly normal Paranoia everyone in the universe has that: Slartibartfast

 

[Glenn9999]

I'd probably do as sggaunt says with this project in general if I was working on it, even just using a big block of data instead of using string structures, especially since they are very costly in general. You'd be surprised at the rat's nest that's there in the source if you trace out some of the functions. Now that I saw the other thing and looked at it a bit more, that's probably what I would suggest: find a way to not call "SetLength" at all. Speaking of which (I didn't catch this before), since you use an AnsiString, the #0 is part of the package since an AnsiString is a block of characters terminated by a #0.

It is not possible for anyone to acknowledge truth when their salary depends on them not doing it.

 

[Glenn9999]

Or even better, descend TShortStringArray from TList and use TList as a basis for your storage, especially since it will cover most of what we've been talking about.

It is not possible for anyone to acknowledge truth when their salary depends on them not doing it.

 

[DavidB3]

Not use SetLength at all it's a good idea, but the key question is "how"? I know that AnsiString is delimited by #0 (since is in fact a PAnsiChar) but that doesn't help me a lot here.

I can't use TList, in fact I am doing my component(s) as a replacement for T(String)List. So far are taking less Ram, fragments the Ram less, are faster when you add/delete items and are more "flexible".
The only problem is the read speed which is slightly lower.
If you show me a way to sent you the entire code you'll understand.