Atozed Forums
Porting old Delphi2007 application using TServerSocket.... - Printable Version

+- Atozed Forums (https://www.atozed.com/forums)
+-- Forum: Indy (https://www.atozed.com/forums/forum-8.html)
+--- Forum: Indy General Discussion (https://www.atozed.com/forums/forum-9.html)
+--- Thread: Porting old Delphi2007 application using TServerSocket.... (/thread-1778.html)

Pages: 1 2 3

RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 09-29-2020

I have started debugging tests now with the Linux port of the server and the client running on Windows but ported to FPC/Lazarus.
On the client side I use the newly created TTcpClientComm object based on TIdTcpClient and I have discovered some issues when dealing with larger amounts of data...
Here is the declaration of the object plus the thread implementation:

Code:
  TReadingThread = class(TThread)
  protected
    FRxData: TIdBytes;
    FOwner: TTcpClientComm;
      procedure Execute; override;
    procedure DoOnRxData;
  public
      constructor Create(Owner: TTcpClientComm);
  end;

  { TTcpClientComm }

  TTcpClientComm = class(TObject)
  private
    FConn: TIdTCPClient;
    FReadThread: TReadingThread;
    FOnRxData: TCommRxEvent;
    FOnLogMsgEvent: TLogMessageEvent;
    FLastError: string;
    FConnected: boolean;
    FHost: string;
    FPort: word;
    procedure OnStatusChange(ASender: TObject; const AStatus: TIdStatus; const AStatusText: string);
    procedure LogEventData(Msg: Ansistring);
  public
    constructor Create;
    destructor  Destroy; override;
    property OnRxData: TCommRxEvent read FOnRxData write FOnRxData;
    property OnLogMsgEvent: TLogMessageEvent read FOnLogMsgEvent write FOnLogMsgEvent;
    property LastError: string read FLastError;
    property Connected: boolean read FConnected;
    property Host: string read FHost;
    property Port: word read FPort;
    procedure Connect(AHost: string; APort: word);
    procedure Disconnect;
    function  WriteData(const Data: string): boolean; overload;
    function  WriteData(const Data: TIdBytes): boolean; overload;
  end;

{ TReadingThread }

procedure TReadingThread.Execute;
begin
    while not Terminated do
    begin
    if FOwner.Connected then  //Do not use FConn.Connected here because that triggers a read operation on the socket...
    begin
      FOwner.FConn.IOHandler.ReadBytes(FRxData, -1, False); //Get all available data
      if Assigned(FOwner.FOnRxData) then
        Synchronize(DoOnRxData);
      end;
  end;
end;

procedure TReadingThread.DoOnRxData;
begin
  if Assigned(FOwner.FOnRxData) and (Length(FRxData) > 0) then
    FOwner.FOnRxData(Self, FRxData);
  SetLength(FRxData, 0);
end;

constructor TReadingThread.Create(Owner: TTcpClientComm);
begin
  FOwner := Owner;
  inherited Create(False);
end;

As you can see I am setting the FRxData container to zero length after it has been submitted to the event handler.
I did this to ensure that the received data are not processed twice.
In the client the event procedure looks like this:

Code:
procedure TSSRemoteClient.OnRxData(Sender: TObject; const Data: TIdBytes);
{This event is where data from the TCP/IP connection to the server are received.
Move data into a string and process as needed.}
var
  RxData: string;
  len: integer;
begin
  len := Length(Data);
  SetLength(RxData, len);
  Move(Data[0], RxData[1], len);
  HandleRxCommand(RxData);
end;
The reason the data is moved into a string is that the bulk of the existing processing (which has worked OK for many years) is using the string type as a container for the data. And in this case all payload data are textual even transfered files because they are hex encoded before transfer (doubles the byte count but makes it possible to deal with as text)

Now it seems like I am losing some received data when the amount is large like when the content of a logfile is transferred.

Is this possibly the reason?
How can I safeguard against data loss?
During synchronize, I thought that the read thread is paused until the event procedure returns, is that correct?

I can compare the operations of this new ported client with the old Delphi client and I see the problems this way.

RE: Porting old Delphi2007 application using TServerSocket.... - rlebeau - 09-29-2020

(09-29-2020, 11:20 AM)BosseB Wrote: The reason the data is moved into a string is that the bulk of the existing processing (which has worked OK for many years) is using the string type as a container for the data.

Then why not do the conversion to string inside the reading thread, rather than in the event handler?

(09-29-2020, 11:20 AM)BosseB Wrote: And in this case all payload data are textual even transfered files because they are hex encoded before transfer (doubles the byte count but makes it possible to deal with as text)

That is just a bad waste of bandwidth.

(09-29-2020, 11:20 AM)BosseB Wrote: Now it seems like I am losing some received data when the amount is large like when the content of a logfile is transferred.

You are not framing your data in any way (like with STX/ETX), so it is possible, even likely, that your event handler is dealing with partial/incomplete messages to being with. Remember, TCP is a byte stream, it has no concept of application-level messages, so you MUST frame your messages in such a way that the receiver knows where a message ends and the next message begins. I strongly suggest you deal with the framing in your reading thread, and pass only complete messages to your event handler.

(09-29-2020, 11:20 AM)BosseB Wrote: Is this possibly the reason?

No. The size of the data does not matter. TCP is just a stream, it doesn't care how large each message actually is. That is for the application layer to deal with.

(09-29-2020, 11:20 AM)BosseB Wrote: How can I safeguard against data loss?

Frame your data properly.

(09-29-2020, 11:20 AM)BosseB Wrote: During synchronize, I thought that the read thread is paused until the event procedure returns, is that correct?

Yes. But that doesn't mean your event handler is receiving a completed data message to begin with. It may be receiving only part of a message, or even pieces of multiple messages stringed together. It is your responsibility to separate the messages properly, TCP (and Indy) won't do that for you.

RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 09-29-2020

I used TIdBytes in the thread because this is purportedly better than string.

The old code expects a stream of events each supplying a string type chunk.
The processing part of the event handler adds the new chunk to a client side string buffer and then checks if the ETX has arrived. So there is a packeting mechanism there.
If it has it also signals that there is a message to read for the main code.

If I instead make the read thread wait for a complete message ending with ETX, before firing off the event, then I guess all of the data (potentially rather big) will be supplied in one call. And the ETX will not be part of it, right?
So I could add ETX to the end so that the existing code will get it like before.

Question:
If I do a read with delimiting char in the thread (ReadLn), then I suspect that if the transfer takes some time the timeout will fire in the thread. How can it know that this happened and not send off the data but instead do another read call?
I cannot find a version of Read that uses a terminator and TIdBytes as an expanding buffer...
Or does ReadLn(ETX) not return any data if the terminator has not been seen?

Will this work (with FRxData being a string instead of TIdBytes):

Code:
procedure TReadingThread.Execute;
begin
  while not Terminated do
  begin
    if FOwner.Connected then  //Do not use FConn.Connected here
    begin
      FRxData := FOwner.FConn.IOHandler.ReadLn(ETX); //was ReadBytes(FRxData, -1, False); //Get all available data
      if Assigned(FOwner.FOnRxData) and (Length(FRxData) > 0) then
        Synchronize(DoOnRxData);
    end;
  end;
end;

procedure TReadingThread.DoOnRxData;
begin
  if Assigned(FOwner.FOnRxData) then
    FOwner.FOnRxData(Self, FRxData + ETX);
end;

EDIT:
Well, it turns out that using the ETX in socket readln makes the system throw up an exception from within Indy10 with this message:
Quote:Debugger exception notification
Project <name> raised exception class 'EIdReadLnMaxLineLengthExceeded' with message
Max line length exceeded.

in file '.\Core\IdIOHandler.pas' at line 1524

It happens when a logfile was transfered.
It is pretty big (currently 7.6 Mbytes) and was the reason I asked about the maximum size of transfers before...
What is the maximum line length for Indy10?

EDIT2:
Turns out that in Core/IdIOHandler.pas this is specified:
Code:
const
  GRecvBufferSizeDefault = 32 * 1024;
  GSendBufferSizeDefault = 32 * 1024;
  IdMaxLineLengthDefault = 16 * 1024;
  // S.G. 6/4/2004: Maximum number of lines captured
  // S.G. 6/4/2004: Default to "unlimited"
  Id_IOHandler_MaxCapturedLines = -1;
Which means that the tx and rx buffers are each 32K and the max line length for ReadLn is 16K.
How can one then transfer data that is megabytes in size using an STX/ETX protocol?

RE: Porting old Delphi2007 application using TServerSocket.... - rlebeau - 09-29-2020

(09-29-2020, 06:04 PM)BosseB Wrote: I used TIdBytes in the thread because this is purportedly better than string.

For binary data, certainly.

(09-29-2020, 06:04 PM)BosseB Wrote: The old code expects a stream of events each supplying a string type chunk.
The processing part of the event handler adds the new chunk to a client side string buffer and then checks if the ETX has arrived. So there is a packeting mechanism there.

OK, in that case the problem you mentioned earlier (which you didn't actually go into details about) is related to something else.

(09-29-2020, 06:04 PM)BosseB Wrote: If I instead make the read thread wait for a complete message ending with ETX, before firing off the event, then I guess all of the data (potentially rather big) will be supplied in one call. And the ETX will not be part of it, right?

We have already gone over how to deal with STX/ETX handling with Indy.  I'm not going to get back into that.  Go re-read our earlier comments.

(09-29-2020, 06:04 PM)BosseB Wrote: If I do a read with delimiting char in the thread (ReadLn), then I suspect that if the transfer takes some time the timeout will fire in the thread.

The timeout is applied on a per-byte basis, not for the entire function call.  But yes, if the time between individual bytes exceeds the timeout then the whole read operation will fail and exit.

(09-29-2020, 06:04 PM)BosseB Wrote: How can it know that this happened and not send off the data but instead do another read call?

If TIdIOHandler.ReadLn() fails due to timeout, it will return a blank string, and set IOHandler.ReadLnTimedOut=True.  If TIdIOHandler.ReadBytes() fails due to timeout, it will raise an exception.  Either condition should cause you to skip sending data to your event handler.

(09-29-2020, 06:04 PM)BosseB Wrote: I cannot find a version of Read that uses a terminator and TIdBytes as an expanding buffer...

There is currently no version of TIdIOHandler.ReadLn() available for bytes, only for strings.  But it is not hard to write such logic manually in your own code, eg:

Code:
procedure TReadingThread.Execute;
var
  LInputBufferSize,
  LStartPos,
  LTermPos: Integer;
begin
  LStartPos := 0;

  while not Terminated do
  begin
    if FOwner.Connected then  //Do not use FConn.Connected here
    begin
      LInputBufferSize := FOwner.FConn.IOHandler.InputBuffer.Size;

      if (LInputBufferSize > 0) and (LStartPos < LInputBufferSize) then begin
        LTermPos := FOwner.FConn.IOHandler.InputBuffer.IndexOf($03{ETX}, LStartPos);
      end else begin
        LTermPos := -1;
      end;

      if LTermPos = -1 then begin
        if not FOwner.FConn.IOHandler.CheckForDataOnSource(YourTimeoutHere) then
          Exit; // timeout exceeded...
        FOwner.FConn.IOHandler.CheckForDisconnect;
        LStartPos := LInputBufferSize;
        Continue;
      end;

      FOwner.FConn.IOHandler.ReadBytes(FRxData, LTermPos + 1, False);
      LStartPos := 0;

      if Assigned(FOwner.FOnRxData) then
        Synchronize(DoOnRxData);

      SetLength(FRxData, 0);
    end;
  end;
end;

(09-29-2020, 06:04 PM)BosseB Wrote: Or does ReadLn(ETX) not return any data if the terminator has not been seen?

Correct.

(09-29-2020, 06:04 PM)BosseB Wrote: Will this work (with FRxData being a string instead of TIdBytes):

If the data is delimited with ETX, then yes.

(09-29-2020, 06:04 PM)BosseB Wrote: EDIT:
Well, it turns out that using the ETX in socket readln makes the system throw up an exception from within Indy10 with this message:
Quote:Debugger exception notification
Project <name> raised exception class 'EIdReadLnMaxLineLengthExceeded' with message
Max line length exceeded.

in file '.\Core\IdIOHandler.pas' at line 1524

It happens when a logfile was transfered.
It is pretty big (currently 7.6 Mbytes) and was the reason I asked about the maximum size of transfers before...
What is the maximum line length for Indy10?

EDIT2:
Turns out that in Core/IdIOHandler.pas this is specified:
Code:
const
  GRecvBufferSizeDefault = 32 * 1024;
  GSendBufferSizeDefault = 32 * 1024;
  IdMaxLineLengthDefault = 16 * 1024;
  // S.G. 6/4/2004: Maximum number of lines captured
  // S.G. 6/4/2004: Default to "unlimited"
  Id_IOHandler_MaxCapturedLines = -1;
Which means that the tx and rx buffers are each 32K and the max line length for ReadLn is 16K.

Those are just the defaults.  You can change them at runtime as needed, via the IOHandler's public RecvBufferSize, SendBufferSize, and MaxLineLength properties. Or, in the case of MaxLineLength, TIdIOHandler.ReadLn() has an optional AMaxLineLength input parameter that supercedes the TIdIOHandler.MaxLineLength property if set.

(09-29-2020, 06:04 PM)BosseB Wrote: How can one then transfer data that is megabytes in size using an STX/ETX protocol?

You could set MaxLineLength=0 or MaxLineLength=MaxInt, which will then allow you to receive such large strings as far as available memory will allow.  Or, you can set the IOHandler's MaxLineAction=maSplit and deal with the possibility that ReadLn() will then return partial strings when the MaxLineLength is exceeded (which FYI, there is a separate TIdIOHandler.ReadLnSplit() method for that very purpose).

But, I would not recommend using strings for such large data to begin with.

RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 10-01-2020

Just to close this thread with final thoughts if someone stumbles across it in the future...

It all seems to work now to use several different approaches, so I decided to use this where I have a TIdBytes buffer and read all available data every pass in the Execute procedure.
But I am using an event handler where the transfer container is a string, partial code below.

Note that I have both set the event handler Data container as const as shown below and without const. I see no difference either way.


Code:
type
  TCommRxEvent = procedure (Sender: TObject; const Data: string) of object;

  TTcpClientComm = class;

  { TReadingThread }

  TReadingThread = class(TThread)
  protected
    FRxData: TIdBytes;
    FOwner: TTcpClientComm;
  procedure Execute; override;
    procedure DoOnRxData;
  public
    constructor Create(Owner: TTcpClientComm);
  end;


  { TTcpClientComm }

  TTcpClientComm = class(TObject)
  private
    FConn: TIdTCPClient;
    FReadThread: TReadingThread;
    FOnRxData: TCommRxEvent;
    FOnLogMsgEvent: TLogMessageEvent;
    FLastError: string;
    FConnected: boolean;
    FHost: string;
    FPort: word;
    procedure OnStatusChange(ASender: TObject; const AStatus: TIdStatus; const AStatusText: string);
    procedure LogEventData(Msg: Ansistring);
  public
    constructor Create;
    destructor  Destroy; override;
    property OnRxData: TCommRxEvent read FOnRxData write FOnRxData;
    property OnLogMsgEvent: TLogMessageEvent read FOnLogMsgEvent write FOnLogMsgEvent;
    property LastError: string read FLastError;
    property Connected: boolean read FConnected;
    property Host: string read FHost;
    property Port: word read FPort;
    procedure Connect(AHost: string; APort: word);
    procedure Disconnect;
    function  WriteData(const Data: string): boolean; overload;
    function  WriteData(const Data: TIdBytes): boolean; overload;
  end;

implementation
const
    ETX = #3;

constructor TTcpClientComm.Create;
begin
  FConnected := false;
  FHost := '';
  FPort := 0;
  FConn := TIdTCPClient.Create(NIL);
  FConn.OnStatus := OnStatusChange;
  FConn.ConnectTimeout := 5000; //5 second connect timeout
  FConn.ReadTimeout := 100; //Used in the read thread to not block
end;

procedure TTcpClientComm.Connect(AHost: string; APort: word);
begin
  try
    FLastError := '';
    FConn.Connect(AHost, APort); //Create read thread in OnStatus.hsConnected
    FHost := Host;
    FPort := APort;
  except
    on E: Exception do
    begin
      FLastError := E.Message;
    end;
  end;
end;

procedure TTcpClientComm.OnStatusChange(ASender: TObject; const AStatus: TIdStatus; const AStatusText: string);
{This procedure parses the socket status change messages}
begin
  case AStatus of
    hsResolving: ;
    hsConnecting: ;
    hsConnected:
      begin
        FReadThread := TReadingThread.Create(Self);
        FConnected := true;
        LogEventData('Connected to server ' + FHost + ':' + IntToStr(FPort));
      end;
    hsDisconnecting: FConnected := false;
    hsDisconnected:
      begin
        FConnected := false;
        if FReadThread <> NIL then
          FReadThread.Terminate;
        if FReadThread <> NIL then
        begin
          FReadThread.WaitFor;
          FreeAndNil(FReadThread);
        end;
        LogEventData('Disconnected from server ' + FHost);
      end;
    hsStatusText:
      ;
  end;
end;

{ TReadingThread }

procedure TReadingThread.Execute;
begin
while not Terminated do
begin
    if FOwner.Connected then  //Do not use FConn.Connected here because that triggers a read operation on the socket...
    begin
      try
        FOwner.FConn.IOHandler.ReadBytes(FRxData, -1, False); //Get all available data
      except
        //Do nothing, timeout happened
      end;
      if (Length(FRxData) > 0) and Assigned(FOwner.FOnRxData) then
        Synchronize(DoOnRxData);
  end;
  end;
end;

procedure TReadingThread.DoOnRxData;
begin
  if Assigned(FOwner.FOnRxData) then
    FOwner.FOnRxData(Self, BytesToStringRaw(FRxData));
  SetLength(FRxData, 0);
end;

constructor TReadingThread.Create(Owner: TTcpClientComm);
begin
  FOwner := Owner;
  inherited Create(False);
end;