Atozed Forums
TIdMessage.LoadFromStream: Missing fields - Printable Version

+- Atozed Forums (https://www.atozed.com/forums)
+-- Forum: Indy (https://www.atozed.com/forums/forum-8.html)
+--- Forum: Indy General Discussion (https://www.atozed.com/forums/forum-9.html)
+--- Thread: TIdMessage.LoadFromStream: Missing fields (/thread-3062.html)

Pages: 1 2

TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-06-2023

I use the IMAP system to dowload an email.
MSG: TIdMessage;
IMAP: TIdIMAP4;

I then call  IMAP.UIDRetrieve(IntToStr(IMAP.MailBox.SearchResult[i]), MSG);
I can now read all fields like From, Subject etc. fine.

Then the entire email is loaded into a Stream using
IMAP.UIDRetrieveNoDecodeToStream(IntToStr(IMAP.MailBox.SearchResult[i]), AStream);

This Stream is then saved to file, as an .eml file.
I can load this file with an email program fine, all data complete.

The same Stream is internally kept.
But when I afterwards use an IdMessage.LoadFromStream, all fields like From, Subject, etc. are all empty.
The LoadFromStream does not seem to initialize the fields.

It seems only the IMAP does this, during the call to IMAP.UIDRetrieve.

How do I get my data back in the TIdMessage after a LoadFromStream?

RE: TIdMessage.LoadFromStream: Missing fields - rlebeau - 03-07-2023

(03-06-2023, 10:29 PM)BartKindt Wrote: I then call  IMAP.UIDRetrieve(IntToStr(IMAP.MailBox.SearchResult[i]), MSG);
I can now read all fields like From, Subject etc. fine.

Internally, TIdIMAP4.UIDRetrieve() downloads the raw bytes of the email into a TMemoryStream, and then calls TIdMessage.LoadFromStream() with the AUsesDotTransparency parameter set to False (its default is True instead).

(03-06-2023, 10:29 PM)BartKindt Wrote: Then the entire email is loaded into a Stream using
IMAP.UIDRetrieveNoDecodeToStream(IntToStr(IMAP.MailBox.SearchResult[i]), AStream);

Internally, TIdIMAP4.UIDRetrieveNoDecodeToStream() creates a temporary TIdMessage and then downloads the email using the same logic as TIdIMAP4.UIDRetrieve(), except that it sets both TIdMessage.NoDecode and TIdMessage.NoEncode to True. Then, it calls TIdMessage.SaveToStream() with the AUsesDotTransparency parameter set to True.

You really need AUsesDotTransparency to be set to False instead in this situation.  There is a TODO comment about this in the source code for TIdIMAP4.UIDRetrieveNoDecodeToStream() (and all of the other TIdIMAP4.(UID)RetrieveNoDecodeTo...() methods):

Code:
{RLebeau 12/09/2012: NOT currently using the same workaround here that
is being used in AppendMsg() to avoid SMTP dot transparent output from
TIdMessage.SaveToStream().  The reason for this is because I don't
know how this method is being used and I don't want to break anything
that may be depending on that transparent output being generated...}
...
{TODO: add an optional parameter to specify whether dot transparency
should be used or not, and then pass that to SaveToStream(). Or better,
just deprecate this method and implement a replacement that downloads
the message directly to the file without dot transparency, since it
has no meaning in IMAP. InternalRetrieve() uses an internal stream
anyway to receive the data, so let's just cut out TIdMessage here...}

(03-06-2023, 10:29 PM)BartKindt Wrote: This Stream is then saved to file, as an .eml file.
I can load this file with an email program fine, all data complete.

Most other programs are not expecting an .EML file to have its data escaped with dot-transparency.  That is a detail of the POP3 and SMTP protocols only, IMAP doesn't use dot-transparency at all.  But TIdMessage does by default, so watch out for that!

(03-06-2023, 10:29 PM)BartKindt Wrote: The same Stream is internally kept.
But when I afterwards use an IdMessage.LoadFromStream, all fields like From, Subject, etc. are all empty.
The LoadFromStream does not seem to initialize the fields.

Are you resetting the stream's Position back to 0 before you load the stream?  TIdMessage.LoadFromStream() will not do that for you.

Also, make sure that you always call TIdMessage.LoadFromStream() using the same AUsesDotTransparency parameter value that was used when TIdMessage.SaveToStream() was called.

(03-06-2023, 10:29 PM)BartKindt Wrote: It seems only the IMAP does this, during the call to IMAP.UIDRetrieve.

TIdIMAP4.UIDRetrieve() and TIdIMAP4.UIDRetrieveNoDecodeToStream() are using TIdMessage in different ways.

RE: TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-07-2023

I have no access to the AUsesDotTransparency. I also don't know what it means.

But:

- I load the TIdIMAP4.UIDRetrieveNoDecodeToStream, then save the resulting TIdMessage.SaveToFile();
- I create a *new* TIdMessage, and reload from File.
There is no valid data in any of the fields.

What do I have to do in the TIdMessage itself, to actually populate all the fields?
Even if I use 'IdMessage.LoadFromFile(xxx,true) // Headers only
then there are NO headers. The Count is 0. All other fields are empty.

There must be some command inside the IdMessage itself to populate all fields, after the loading of the Stream or File???

RE: TIdMessage.LoadFromStream: Missing fields - rlebeau - 03-07-2023

(03-07-2023, 03:16 AM)BartKindt Wrote: I have no access to the AUsesDotTransparency.

Add the IdMessageHelper unit to your uses clause.  Right now, a class helper is being used to overload the TIdMessage.LoadFrom...() and TIdMessage.SaveTo...() methods with an additional AUsesDotTransparency parameter:

Code:
type
  TIdMessageHelper = class helper for TIdMessage
  public
    procedure LoadFromFile(const AFileName: string; const AHeadersOnly: Boolean; const AUsesDotTransparency: Boolean); overload;
    procedure LoadFromStream(AStream: TStream; const AHeadersOnly: Boolean; const AUsesDotTransparency: Boolean); overload;
    procedure SaveToFile(const AFileName: string; const AHeadersOnly: Boolean; const AUseDotTransparency: Boolean); overload;
    procedure SaveToStream(AStream: TStream; const AHeadersOnly: Boolean; const AUseDotTransparency: Boolean); overload;
  end;

(03-07-2023, 03:16 AM)BartKindt Wrote: I also don't know what it means.

The way that the POP3 and SMTP protocols are structured (but not the IMAP protocol), email data needs to be escaped when transmitted over the socket connection.  Specifically, that escaping involves adding an extra period ('.') character in front of any line of text that already begins with a period, because the email data is terminated by sending a period ('.') character on a line by itself.  The TIdMessage.SaveTo...() methods save only escaped emails, and the TIdMessage.LoadFrom...() methods load only escaped emails, regardless of whether the emails are being used with POP3/SMTP or not (see https://github.com/IndySockets/Indy/issues/135).  The TIdMessageHelper class helper was added to workaround that issue without completely redesigning TIdMessage's internals.

When AUsesDotTransparency is True, the escaping is saved/parsed as before.

When AUsesDotTransparency is False, the escaping is not saved/parsed.

(03-07-2023, 03:16 AM)BartKindt Wrote: I load the TIdIMAP4.UIDRetrieveNoDecodeToStream, then save the resulting TIdMessage.SaveToFile();

Since TIdIMAP4.UIDRetrieveNoDecodeToStream() uses TIdMessage.SaveToStream(), the stream data will be dot-escaped.  If you are simply calling TIdMessage.LoadFromStream() and then TIdMessage.SaveToFile(), the stream will be loaded with dot-escaping parsed, and then the file will be saved with dot-escaping applied.

(03-07-2023, 03:16 AM)BartKindt Wrote: I create a *new* TIdMessage, and reload from File.

If you are simply calling TIdMessage.LoadFromFile(), then it will expect a dot-escaped file (which you have), and so the TIdMessage would end up holding the original un-escaped data.

(03-07-2023, 03:16 AM)BartKindt Wrote: There is no valid data in any of the fields.

It should be. I would need to see the raw stream/file data.

(03-07-2023, 03:16 AM)BartKindt Wrote: What do I have to do in the TIdMessage itself, to actually populate all the fields?

Given everything you have described, you should NOT have to do anything extra.

However, I would suggest that you call TIdMessage.SaveToFile() and TIdMessage.LoadFromFile() with the AUseDotTransparency parameter set to False for both, since .EML files don't need to be escaped.

(03-07-2023, 03:16 AM)BartKindt Wrote: Even if I use 'IdMessage.LoadFromFile(xxx,true) // Headers only
then there are NO headers. The Count is 0.

The ONLY way that should happen is if either:
  • the file is completely empty to begin with.
  • the file is (treated as) dot-escaped, and there is a sole '.' line before the headers.


RE: TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-16-2023

- I load a IdMessage via the IMAP system, and all data is complete (headers, body, etc.).
- I save the IdMessage both to File and to Stream

- I create a new IdMessage, and do a LoadFrom File: The message shows empty.

- I Create a new IdMessage, and do a LoadFromSteam, and it is also empty.



The raw file (as saved to disk) is clean, and I can open it fine in the Thunderbird email client.

The only dot in the file is at the end.



Somehow the IdMessageClient.ProcessMessage is not processing anything.



In the latest compiler, I cannot edit the Indy files at all; When I copy the various units into my own directory, and try to compile I get errors about some other units 'not compiled with the save version'.



What else can I do?
I attach the raw file as a TXT file.

RE: TIdMessage.LoadFromStream: Missing fields - rlebeau - 03-17-2023

(03-16-2023, 06:54 PM)BartKindt Wrote: - I load a IdMessage via the IMAP system, and all data is complete (headers, body, etc.).

The TIdIMAP4.UIDRetrieveNoDecode...() methods don't actually parse an email's body, they just store the raw body text as-is. The non-NoDecode TIdIMAP4.UIDRetrieve...() methods actually parse an email body.

(03-16-2023, 06:54 PM)BartKindt Wrote: Somehow the IdMessageClient.ProcessMessage is not processing anything.

If that were true, the decoding TIdIMAP4.UIDRetrieve...() methods would be suffering from the same issue. But based on your description, they are not. So something else is going on.

(03-16-2023, 06:54 PM)BartKindt Wrote: In the latest compiler, I cannot edit the Indy files at all;

Why are you trying to edit them? What are you editing, exactly?

(03-16-2023, 06:54 PM)BartKindt Wrote: When I copy the various units into my own directory, and try to compile I get errors about some other units 'not compiled with the save version'.

Then your project must be using other things that are linked to the version of Indy that shipped with the IDE.

(03-16-2023, 06:54 PM)BartKindt Wrote: I attach the raw file as a TXT file.

The file looks fine to me. TIdMessage.LoadFromFile() should not be having any issues loading it. So, how exactly are you validating that it does have issue?

Even if TIdMessage.LoadFromFile() can't parse the email body, it should be parsing the email headers at least, but you claim that all of the TIdMessage properties are blank, and I find that very hard to believe. For instance, that would imply that the very 1st call to IOHandler.ReadLnRFC() inside of TIdMessageClient.ReceiveHeader() is either returning a blank line or setting LMsgEnd=True, either way breaking the reading loop. The TXT file you have shown clearly does not begin with a blank line or a message terminator.

RE: TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-17-2023

I will have to find a way to copy the indy files accros, to that I can edit them and make changes to see where it goes wrong.
At the moment I have no idea where it goes wrong in the ProcessMessage, but it does look like it has something to do with the code here:
Code:
repeat
      Result := IOHandler.ReadLnRFC(LMsgEnd);
      // Exchange Bug: Exchange sometimes returns . when getting a message instead of
      // '' then a . - That is there is no seperation between the header and the message for an
      // empty message.
      if ((Length(AAltTerm) = 0) and LMsgEnd) or  {do not localize}
         ({APR: why? (Length(AAltTerm) > 0) and }(Result = AAltTerm)) then begin
        Break;
      end else if Result <> '' then begin
        AMsg.Headers.Append(Result);
      end;
    until False;
    AMsg.ProcessHeaders;

As said, even loading directly from the Stream I just saved, the new copy shows no headers and no body after loading it.
When copying Indy files accross, I get these errors:

[dcc32 Fatal Error] IdSMTPBase.pas(603): F2051 Unit IdSMTP was compiled with a different version of IdSMTPBase.TIdSMTPBase

This keeps on going, until I probably have all Indy files copied over...
Is there any way to circumvent this?

RE: TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-18-2023

I tried to re-compile the Indy10 included in the latest RAD Studio 11.3
I got an error:
[Fatal Error] Cannot compile package 'IndySystem280' which is currently required by RAD Studio 11.

I have now downloaded the entire Indy10 from Github.
I load the file: Indy280.groupproj
Loads normally.
But I cannot compile it at all. I immediately get the compiler error:
[Fatal Error] Cannot compile package 'IndySystem280' which is currently required by RAD Studio 11.

I don't understand what is going on. I never had problems before compiling a downloaded version of Indy10.

RE: TIdMessage.LoadFromStream: Missing fields - BartKindt - 03-19-2023

Okay, after whole dramas where I lost all Embarcadero Indy10 files, and had to re-install the entire IDE, I now managed to have a copy of Indy10 which I can actually edit.

Remy,

The problem seems to be that a record called TIdIOHandler.FInputBuffer is empty, after the Stream is loaded.

Please see the following pieces of code ( view from top to bottom)
Code:
procedure TIdMessageClient.ProcessMessage(AMsg: TIdMessage; AStream: TStream; AHeaderOnly: Boolean = False);
var
  LIOHandler: TIdIOHandlerStreamMsg;
begin
  Log('IdMessageClient.ProcessMessage (Stream): Stream.Size='+AStream.Size.ToString);
  LIOHandler := TIdIOHandlerStreamMsg.Create(nil, AStream); // Here the Stream is copied into the "FReceiveStream" and it is complete.
  try
    LIOHandler.FreeStreams := False;
    LIOHandler.MaxLineLength := MaxInt;
    IOHandler := LIOHandler;
    // LIOHandler.
    try
      IOHandler.Open; // BART: Here it calls "FInputBuffer.Clear";
      ProcessMessage(AMsg, AHeaderOnly);
---

Procedure TIdMessageClient.ProcessMessage(AMsg: TIdMessage; AHeaderOnly: Boolean = False);
begin
  if IOHandler <> nil then begin
  Log('IdMessageClient.ProcessMessage...');
    //Don't call ReceiveBody if the message ended at the end of the headers
    //(ReceiveHeader() would have returned '.' in that case)...
    BeginWork(wmRead);
    try
      if ReceiveHeader(AMsg) = '' then begin
---

Function TIdMessageClient.ReceiveHeader(AMsg: TIdMessage; const AAltTerm: string = ''): string;
var
  LMsgEnd: Boolean;
begin
  BeginWork(wmRead);
  try
    repeat
      // BART
      // Result := IOHandler.ReadLnRFC(LMsgEnd);
      result := IOHandler.ReadLn; // Bart: I called ReadLn direct for testing, but not the problem.

---

And now: Where is the FInputBuffer data loaded? I may be blind but I do not see it.
The Stream is loaded into the IoHandler; But how is the FInputbuffer loaded??


Code:
function TIdIOHandler.ReadLn(ATerminator: string; ATimeout: Integer = IdTimeoutDefault;
  AMaxLineLength: Integer = -1; AByteEncoding: IIdTextEncoding = nil
  {$IFDEF STRING_IS_ANSI}; ADestEncoding: IIdTextEncoding = nil{$ENDIF}
  ): string;
var
  LInputBufferSize: Integer;
  LStartPos: Integer;
  LTermPos: Integer;
  LReadLnStartTime: TIdTicks;
  LTerm, LResult: TIdBytes;
begin
  AByteEncoding := iif(AByteEncoding, FDefStringEncoding);
  {$IFDEF STRING_IS_ANSI}
  ADestEncoding := iif(ADestEncoding, FDefAnsiEncoding, encOSDefault);
  {$ENDIF}
  if AMaxLineLength < 0 then begin
    AMaxLineLength := MaxLineLength;
  end;
  // User may pass '' if they need to pass arguments beyond the first.
  if ATerminator = '' then begin
    ATerminator := LF;
  end;
  // snip comments

  LTerm := ToBytes(ATerminator, AByteEncoding
    {$IFDEF STRING_IS_ANSI}, ADestEncoding{$ENDIF}
    );
  FReadLnSplit := False;
  FReadLnTimedOut := False;
  LTermPos := -1;
  LStartPos := 0;
  LReadLnStartTime := Ticks64;
  repeat
    LInputBufferSize := FInputBuffer.Size;
    if LInputBufferSize > 0 then begin
      if LStartPos < LInputBufferSize then begin
        LTermPos := FInputBuffer.IndexOf(LTerm, LStartPos);
      end else begin
        LTermPos := -1;
      end;
      LStartPos := IndyMax(LInputBufferSize-(Length(LTerm)-1), 0);
    end;
    // if the line length is limited and terminator is found after the limit or not found and the limit is exceeded
    if (AMaxLineLength > 0) and ((LTermPos > AMaxLineLength) or ((LTermPos = -1) and (LStartPos > AMaxLineLength))) then begin
      if MaxLineAction = maException then begin
        raise EIdReadLnMaxLineLengthExceeded.Create(RSReadLnMaxLineLengthExceeded);
      end;
     
      FReadLnSplit := True;
      Result := FInputBuffer.ExtractToString(AMaxLineLength, AByteEncoding
        {$IFDEF STRING_IS_ANSI}, ADestEncoding{$ENDIF}
        );
      Exit;
    end
    // ReadFromSource blocks - do not call unless we need to
    else if LTermPos = -1 then begin
      // ReadLn needs to call this as data may exist in the buffer, but no EOL yet disconnected
      CheckForDisconnect(True, True);
      // Can only return -1 if timeout
      FReadLnTimedOut := ReadFromSource(True, ATimeout, False) = -1;
      if (not FReadLnTimedOut) and (ATimeout >= 0) then begin
        if GetElapsedTicks(LReadLnStartTime) >= UInt32(ATimeout) then begin
          FReadLnTimedOut := True;
        end;
      end;
      if FReadLnTimedOut then begin
        Result := '';
        Exit;
      end;
    end;
  until LTermPos > -1;
  // Extract actual data
  // snip comments
  Result := FInputBuffer.Extract(LTermPos + Length(ATerminator), AEncoding);
  LTermPos := IndyMin(LTermPos, Length(Result));
  if (ATerminator = LF) and (LTermPos > 0) then begin
    if Result[LTermPos] = CR then begin
      Dec(LTermPos);
    end;
  end;
  SetLength(Result, LTermPos);
  }
  Log('IdIOHandler.ReadLn: FInputBuffer.Size='+FInputBuffer.Size.ToString); // Its size = 5.
  FInputBuffer.ExtractToBytes(LResult, LTermPos + Length(LTerm));
  Log('IdIOHandler.ReadLn: length(LResult)='+length(LResult).ToString);
  if (ATerminator = LF) and (LTermPos > 0) then begin
    Log('IdIOHandler.ReadLn: Line okay');
    if LResult[LTermPos-1] = Ord(CR) then begin
      Dec(LTermPos);
    end;
  end;
  Result := BytesToString(LResult, 0, LTermPos, AByteEncoding
    {$IFDEF STRING_IS_ANSI}, ADestEncoding{$ENDIF}
    );
  Log('IdIOHandler.ReadLn: Result='+result); // Its empty because the FInputBuffer is empty.
end;



The FInputBuffer is only 5 bytes big. The loaded Stream is the original email file.
My Debug  Log shows this:

[12:18:34/555] D3: IdMessageClient.ProcessMessage (Stream): Stream.Size=107825
[12:18:34/555] D3: IdIOHandlerStreamMsg.Create: AReceiveStream.Size=107825
[12:18:34/555] D3: IdMessageClient.ProcessMessage...
[12:18:34/571] D3: IdIOHandler.ReadLn: FInputBuffer.Size=5
[12:18:34/571] D3: IdIOHandler.ReadLn: length(LResult)=2
[12:18:34/586] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/586] D3: IdIOHandler.ReadLn: Result=
[12:18:34/602] D3: IdMessageClient.ReceiveHeader: 1
[12:18:34/602] D3: IdMessageClient.ProcessMessage: ReceiveHeader=""
[12:18:34/624] D3: IdIOHandler.ReadLn: FInputBuffer.Size=3
[12:18:34/624] D3: IdIOHandler.ReadLn: length(LResult)=3
[12:18:34/640] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/640] D3: IdIOHandler.ReadLn: Result=.
[12:18:34/655] D3: IdMessageClient.ProcessMessage: Done. AMsg.Headers.Count=0

RE: TIdMessage.LoadFromStream: Missing fields - rlebeau - 03-20-2023

(03-19-2023, 06:16 AM)BartKindt Wrote: // BART
// Result := IOHandler.ReadLnRFC(LMsgEnd);
result := IOHandler.ReadLn; // Bart: I called ReadLn direct for testing, but not the problem.

TIdIOHandler.ReadLnRFC() simply calls TIdIOHandler.ReadLn(), and then sets LMsgEnd=True if the line is '.', otherwise it sets LMsgEnd=False and strips off the 1st '.' if the line begins with '..'.  This is the decoding process for handling dot-transparency in emails.

(03-19-2023, 06:16 AM)BartKindt Wrote: And now: Where is the FInputBuffer data loaded? I may be blind but I do not see it.
The Stream is loaded into the IoHandler; But how is the FInputbuffer loaded??

Whenever a TIdIOHandler reading method, such as ReadLn(), needs bytes that are not already buffered in the InputBuffer, TIdIOHandler.ReadFromSource() is called to read whatever bytes are available from its data source and put them into the InputBuffer, repeating as needed until the read operation can finish its work.

TIdIOHandler.ReadFromSource() calls TIdIOHandler.ReadDataFromSource(), which TIdIOHandlerStreamMsg overrides to read from its FReceiveStream:

Code:
function TIdIOHandlerStreamMsg.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
var
  LTerminator: String;
begin
  if not FTerminatorWasRead then
  begin
    Result := inherited ReadDataFromSource(VBuffer);

Code:
function TIdIOHandlerStream.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
begin
  // We dont want to read the whole stream in at a time. If its a big
  // file will consume way too much memory by loading it all at once.
  // So lets read it in chunks.
  if Assigned(FReceiveStream) then begin
    Result := IndyMin(32 * 1024, Length(VBuffer));
    if Result > 0 then begin
      Result := TIdStreamHelper.ReadBytes(FReceiveStream, VBuffer, Result);
    end;
  end else begin
    Result := 0;
  end;
end;

Code:
class function TIdStreamHelperVCL.ReadBytes(const AStream: TStream; var VBytes: TIdBytes;
  const ACount, AOffset: Integer): Integer;
var
  LActual: Integer;
begin
  Assert(AStream<>nil);
  Result := 0;

  if VBytes = nil then begin
    SetLength(VBytes, 0);
  end;
  //check that offset<length(buffer)? offset+count?
  //is there a need for this to be called with an offset into a nil buffer?

  LActual := ACount;
  if LActual < 0 then begin
    LActual := AStream.Size - AStream.Position;
  end;

  //this prevents eg reading 0 bytes at Offset=10 from allocating memory
  if LActual = 0 then begin
    Exit;
  end;

  if Length(VBytes) < (AOffset+LActual) then begin
    SetLength(VBytes, AOffset+LActual);
  end;

  Assert(VBytes<>nil);
  Result := AStream.Read(VBytes[AOffset], LActual);
end;

So, in this case, TIdMessage.LoadFromStream() creates a TIdMessageClient with a TIdIOHandlerStreamMsg as its IOHandler, which reads data from the input TStream. Thus, when TIdMessageClient.ReceiveHeader() calls IOHandler.ReadLn(), it will keep reading raw bytes from the TStream until a LF line break is read and buffered in the InputBuffer, then it will extract the whole line from the InputBuffer.

(03-19-2023, 06:16 AM)BartKindt Wrote: The FInputBuffer is only 5 bytes big.

If TIdIOHandlerStreamMsg.ReadDataFromSource() fails to read from the TStream, ie end-of-stream is reached, it will return a final email terminator (<CRLF>.<CRLF>), just in case, which will force TIdIOHandler.ReadLnRFC() to return LMsgEnd=True if it hasn't already:

Code:
function TIdIOHandlerStreamMsg.ReadDataFromSource(var VBuffer: TIdBytes): Integer;
var
  LTerminator: String;
begin
  if not FTerminatorWasRead then
  begin
    Result := inherited ReadDataFromSource(VBuffer);
    if Result > 0 then begin
      FLastByteRecv := VBuffer[Result-1];
      Exit;
    end;
    // determine whether the stream ended with a line
    // break, adding an extra CR and/or LF if needed...
    if (FLastByteRecv = Ord(LF)) then begin
      // don't add an extra line break
      LTerminator := '.' + EOL;
    end else if (FLastByteRecv = Ord(CR)) then begin
      // add extra LF
      LTerminator := LF + '.' + EOL;
    end else begin
      // add extra CRLF
      LTerminator := EOL + '.' + EOL;
    end;
    FTerminatorWasRead := True;
    // in theory, CopyTIdString() will write the string
    // into the byte array using 1-byte characters even
    // under DotNet where strings are usually Unicode
    // instead of ASCII...
    CopyTIdString(LTerminator, VBuffer, 0);
    Result := Length(LTerminator);
  end else begin
    Result := 0;
  end;
end;

(03-19-2023, 06:16 AM)BartKindt Wrote: [12:18:34/555] D3: IdMessageClient.ProcessMessage...
[12:18:34/571] D3: IdIOHandler.ReadLn: FInputBuffer.Size=5
[12:18:34/571] D3: IdIOHandler.ReadLn: length(LResult)=2
[12:18:34/586] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/586] D3: IdIOHandler.ReadLn: Result=

This implies that the very 1st call to IOHandler.ReadLn() in TIdMessageClient.ReceiveHeader() is failing to read any bytes at all from the TStream, causing TIdIOHanderStreamMsg to return its generated terminator (the 5 bytes you are logging), which ReadLn() will then extract from the InputBuffer up to the 1st CRLF (the 2 bytes you are logging) and then strip off the CRLF from the Result (the blank string you are logging).  TIdMessageClient.ReceiveHeader() will then interpret that as end-of-headers, causing TIdMessageClient.ReceiveHeaders() to exit so TIdMessageClient.ProcessMessage() will then call TIdMessageClient.ReceiveBody() next.

However, it is concerning that the input TStream would fail to read any bytes at all, when it clearly has bytes available, according to your logging of its Size.  Any such failure would be happening outside of Indy's control.

Which goes right back to my original question to you 2 weeks ago - when you save a TIdMessage to a TStream and then load it back, are you remembering to reset the TStream.Position back to 0 in between?  If you are loading from a file instead, then that is a non-issue, since file streams always start at Position 0.

You will have to debug the TStream itself to find out what is going on with it.

(03-19-2023, 06:16 AM)BartKindt Wrote: [12:18:34/624] D3: IdIOHandler.ReadLn: FInputBuffer.Size=3
[12:18:34/624] D3: IdIOHandler.ReadLn: length(LResult)=3
[12:18:34/640] D3: IdIOHandler.ReadLn: Line okay
[12:18:34/640] D3: IdIOHandler.ReadLn: Result=.

This is from TIdMessage.ReadBody() calling IOHandler.ReadLn(), extracting the 2nd half of TIdIOHandlerStreamMsg's generated terminator from the InputBuffer, which TIdMessageClient.ReceiveBody() will then interpret as end-of-message.