Reading returned data via TIdTcpClient?

[BosseB]

I am always struggling with this whenever I try to throw together a TCP client application...
I can easily send commands to the server (in this case a WiFi IoT module.
I know it responds to the command but for the life of me I cannot get the reply.
The platform I use is 64 bit FreePascal 3.0.4 + Lazarus 1.8.0 with the IndyLaz package installed


THe function that fails to work as I hoped looks like this (FComm is a TIdTcpClient object):

Code:

function TConfigCommHandler.GetItem(Cmd: TIdBytes; var Data: TIdBytes): boolean;
begin
  Result := false;
  SetLength(Cmd, Length(Cmd) + 1);
  Cmd[Length(Cmd)-1] := ETX;
  FComm.IOHandler.ReadTimeout := 2000;
  try
    FComm.IOHandler.Write(Cmd);
    SetLength(Data, 0);
    FComm.IOHandler.ReadBytes(Data, -1, false); //<== Is this way correct?
    Result := Length(Data) > 0;
    if Result then
    begin
      if Data[Length(Data)-1] <> ETX then //Received packet should end in ETX...
      begin
        FLastError := 'Terminating ETX missing!';
        Result := false;
      end
      else
        SetLength(Data, Length(Data) -1); //Remove trailing ETX
    end
    else
      FLastError := 'No data received within timeout!';
  except
    on E: Exception do
    begin
      Result := false;
      FLastError := 'Exception: ' + E.Message;
      Exit;
    end;
  end;
end;

For example what happens if this ReadBytes function does not get all of the incoming bytes?
I try to detect it via the ETX check but since the data is only at maximum some 25-40 bytes (not possible to know beforehand) I thought it highly unlikely it would be split between packets...

---

I found the cause regarding this specific problem:
The server did not add the trailing ETX to replies, so all of them were thrown away.
I modified the server code to always add the ETX to all outgoing messages.
But I have a follow-up question:
How can the PC application be made such that it will catch and display all incoming data without first sending a command?
For example spontaneous messages from the device are also interesting...

[rlebeau]

I can easily send commands to the server (in this case a WiFi IoT module.
I know it responds to the command but for the life of me I cannot get the reply.

What does the response actually look like?  If you can't read a response, it is usually because you are not reading it correctly.

Code:

FComm.IOHandler.ReadBytes(Data, -1, false); //<== Is this way correct?

When you call TIdIOHandler.ReadBytes() with AByteCount=-1, you are asking it to return whatever arbitrary bytes are currently available at that moment, waiting up to the ReadTimeout for new bytes to arrive.

Prior to SVN rev 5372 (Oct 6 2016), ReadBytes() with AByteCount=-1 would ALWAYS read from the socket, even if there were already bytes left over in the InputBuffer from a previous read.  From rev 5372 onwards, ReadBytes() with AByteCount=-1 reads from the socket ONLY if the InputBuffer is empty.

Code:

if Data[Length(Data)-1] <> ETX then //Received packet should end in ETX...

If the response data is ALWAYS terminated by an ETX, then I would not suggest using ReadBytes() with AByteCount=-1 at all.

If the data is textual in nature, you can use TIdIOHandler.ReadLn() or TIdIOHandler.WaitFor().  Both will allow you to specify ETX as a terminator, and will wait for the ETX to arrive before then returning everything that precedes it.

If the data is binary in nature, TIdIOHandler doesn't have similar methods for bytes, but it would not be hard to implement manually in your code, eg:

Code:

function TConfigCommHandler.GetItem(const Cmd: TIdBytes; var Data: TIdBytes): boolean;
var
 Idx: Integer;
begin
 Result := false;
 SetLength(Data, 0);
 try
   FComm.IOHandler.WriteBufferOpen;
   try
     FComm.IOHandler.Write(Cmd);
     FComm.IOHandler.Write(Byte(ETX));
     FComm.IOHandler.WriteBufferClose;
   except
     FComm.IOHandler.WriteBufferCancel;
     raise;
   end;
   // Received packet should end in ETX...
   Idx := FComm.IOHandler.InputBuffer.IndexOf(Byte(ETX));
   while Idx = -1 do
   begin
     Idx := FComm.IOHandler.InputBuffer.Size;
     if not FConn.IOHandler.CheckForDataOnSource(2000) then
     begin
       FLastError := 'No data received within timeout!';
       Exit;
     end;
     FConn.IOHandler.CheckForDisconnect(True, True);
     Idx := FComm.IOHandler.InputBuffer.IndexOf(Byte(ETX), Idx);
   end;
   FComm.IOHandler.ReadBytes(Data, Idx, False);
   FComm.IOHandler.Discard(1); //trailing ETX
 except
   on E: Exception do
   begin
     FLastError := 'Exception: ' + E.Message;
     Exit;
   end;
 end;
 Result := true;
end;

For example what happens if this ReadBytes function does not get all of the incoming bytes?

It will return what it can, and you will have to call it again to read more bytes.

I try to detect it via the ETX check but since the data is only at maximum some 25-40 bytes (not possible to know beforehand) I thought it highly unlikely it would be split between packets...

With TCP, it very well can be.  Just because messages are small doesn't mean they can't be split up into multiple network packets.

I found the cause regarding this specific problem:
The server did not add the trailing ETX to replies, so all of them were thrown away.
I modified the server code to always add the ETX to all outgoing messages.

Good.

How can the PC application be made such that it will catch and display all incoming data without first sending a command?

You would need to run a continuous reading loop, such as with a timer or worker thread, and then display whatever you receive.  In that kind of situation, using ReadBytes() with AByteCount=-1 would make more sense.  Unless the messages have a structure to them, then it is best to read the structure (like above, reading until ETX is reached), and then display only complete messages.

For example spontaneous messages from the device are also interesting...

Dealing with unsolicited messages is always fun Then you have the added burden of detecting when an incoming message is a reply to a previous command vs an unsolicited message.  And almost always requires you to code your app to work asynchronously, unlike the code above which is synchronous.  When you send a command, you would not be able to wait for the reply, you would have to exit and move on, and let your separate reading loop receive the eventual reply and decide what to do with it, in addition to any unsolicited messages it may also receive.