![]() |
Porting old Delphi2007 application using TServerSocket.... - Printable Version +- Atozed Forums (https://www.atozed.com/forums) +-- Forum: Indy (https://www.atozed.com/forums/forum-8.html) +--- Forum: Indy General Discussion (https://www.atozed.com/forums/forum-9.html) +--- Thread: Porting old Delphi2007 application using TServerSocket.... (/thread-1778.html) |
RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 09-29-2020 I have started debugging tests now with the Linux port of the server and the client running on Windows but ported to FPC/Lazarus. On the client side I use the newly created TTcpClientComm object based on TIdTcpClient and I have discovered some issues when dealing with larger amounts of data... Here is the declaration of the object plus the thread implementation: Code: TReadingThread = class(TThread) As you can see I am setting the FRxData container to zero length after it has been submitted to the event handler. I did this to ensure that the received data are not processed twice. In the client the event procedure looks like this: Code: procedure TSSRemoteClient.OnRxData(Sender: TObject; const Data: TIdBytes); Now it seems like I am losing some received data when the amount is large like when the content of a logfile is transferred. Is this possibly the reason? How can I safeguard against data loss? During synchronize, I thought that the read thread is paused until the event procedure returns, is that correct? I can compare the operations of this new ported client with the old Delphi client and I see the problems this way. RE: Porting old Delphi2007 application using TServerSocket.... - rlebeau - 09-29-2020 (09-29-2020, 11:20 AM)BosseB Wrote: The reason the data is moved into a string is that the bulk of the existing processing (which has worked OK for many years) is using the string type as a container for the data. Then why not do the conversion to string inside the reading thread, rather than in the event handler? (09-29-2020, 11:20 AM)BosseB Wrote: And in this case all payload data are textual even transfered files because they are hex encoded before transfer (doubles the byte count but makes it possible to deal with as text) That is just a bad waste of bandwidth. (09-29-2020, 11:20 AM)BosseB Wrote: Now it seems like I am losing some received data when the amount is large like when the content of a logfile is transferred. You are not framing your data in any way (like with STX/ETX), so it is possible, even likely, that your event handler is dealing with partial/incomplete messages to being with. Remember, TCP is a byte stream, it has no concept of application-level messages, so you MUST frame your messages in such a way that the receiver knows where a message ends and the next message begins. I strongly suggest you deal with the framing in your reading thread, and pass only complete messages to your event handler. (09-29-2020, 11:20 AM)BosseB Wrote: Is this possibly the reason? No. The size of the data does not matter. TCP is just a stream, it doesn't care how large each message actually is. That is for the application layer to deal with. (09-29-2020, 11:20 AM)BosseB Wrote: How can I safeguard against data loss? Frame your data properly. (09-29-2020, 11:20 AM)BosseB Wrote: During synchronize, I thought that the read thread is paused until the event procedure returns, is that correct? Yes. But that doesn't mean your event handler is receiving a completed data message to begin with. It may be receiving only part of a message, or even pieces of multiple messages stringed together. It is your responsibility to separate the messages properly, TCP (and Indy) won't do that for you. RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 09-29-2020 I used TIdBytes in the thread because this is purportedly better than string. The old code expects a stream of events each supplying a string type chunk. The processing part of the event handler adds the new chunk to a client side string buffer and then checks if the ETX has arrived. So there is a packeting mechanism there. If it has it also signals that there is a message to read for the main code. If I instead make the read thread wait for a complete message ending with ETX, before firing off the event, then I guess all of the data (potentially rather big) will be supplied in one call. And the ETX will not be part of it, right? So I could add ETX to the end so that the existing code will get it like before. Question: If I do a read with delimiting char in the thread (ReadLn), then I suspect that if the transfer takes some time the timeout will fire in the thread. How can it know that this happened and not send off the data but instead do another read call? I cannot find a version of Read that uses a terminator and TIdBytes as an expanding buffer... Or does ReadLn(ETX) not return any data if the terminator has not been seen? Will this work (with FRxData being a string instead of TIdBytes): Code: procedure TReadingThread.Execute; EDIT: Well, it turns out that using the ETX in socket readln makes the system throw up an exception from within Indy10 with this message: Quote:Debugger exception notification It happens when a logfile was transfered. It is pretty big (currently 7.6 Mbytes) and was the reason I asked about the maximum size of transfers before... What is the maximum line length for Indy10? EDIT2: Turns out that in Core/IdIOHandler.pas this is specified: Code: const How can one then transfer data that is megabytes in size using an STX/ETX protocol? RE: Porting old Delphi2007 application using TServerSocket.... - rlebeau - 09-29-2020 (09-29-2020, 06:04 PM)BosseB Wrote: I used TIdBytes in the thread because this is purportedly better than string. For binary data, certainly. (09-29-2020, 06:04 PM)BosseB Wrote: The old code expects a stream of events each supplying a string type chunk. OK, in that case the problem you mentioned earlier (which you didn't actually go into details about) is related to something else. (09-29-2020, 06:04 PM)BosseB Wrote: If I instead make the read thread wait for a complete message ending with ETX, before firing off the event, then I guess all of the data (potentially rather big) will be supplied in one call. And the ETX will not be part of it, right? We have already gone over how to deal with STX/ETX handling with Indy. I'm not going to get back into that. Go re-read our earlier comments. (09-29-2020, 06:04 PM)BosseB Wrote: If I do a read with delimiting char in the thread (ReadLn), then I suspect that if the transfer takes some time the timeout will fire in the thread. The timeout is applied on a per-byte basis, not for the entire function call. But yes, if the time between individual bytes exceeds the timeout then the whole read operation will fail and exit. (09-29-2020, 06:04 PM)BosseB Wrote: How can it know that this happened and not send off the data but instead do another read call? If TIdIOHandler.ReadLn() fails due to timeout, it will return a blank string, and set IOHandler.ReadLnTimedOut=True. If TIdIOHandler.ReadBytes() fails due to timeout, it will raise an exception. Either condition should cause you to skip sending data to your event handler. (09-29-2020, 06:04 PM)BosseB Wrote: I cannot find a version of Read that uses a terminator and TIdBytes as an expanding buffer... There is currently no version of TIdIOHandler.ReadLn() available for bytes, only for strings. But it is not hard to write such logic manually in your own code, eg: Code: procedure TReadingThread.Execute; (09-29-2020, 06:04 PM)BosseB Wrote: Or does ReadLn(ETX) not return any data if the terminator has not been seen? Correct. (09-29-2020, 06:04 PM)BosseB Wrote: Will this work (with FRxData being a string instead of TIdBytes): If the data is delimited with ETX, then yes. (09-29-2020, 06:04 PM)BosseB Wrote: EDIT: Those are just the defaults. You can change them at runtime as needed, via the IOHandler's public RecvBufferSize, SendBufferSize, and MaxLineLength properties. Or, in the case of MaxLineLength, TIdIOHandler.ReadLn() has an optional AMaxLineLength input parameter that supercedes the TIdIOHandler.MaxLineLength property if set. (09-29-2020, 06:04 PM)BosseB Wrote: How can one then transfer data that is megabytes in size using an STX/ETX protocol? You could set MaxLineLength=0 or MaxLineLength=MaxInt, which will then allow you to receive such large strings as far as available memory will allow. Or, you can set the IOHandler's MaxLineAction=maSplit and deal with the possibility that ReadLn() will then return partial strings when the MaxLineLength is exceeded (which FYI, there is a separate TIdIOHandler.ReadLnSplit() method for that very purpose). But, I would not recommend using strings for such large data to begin with. RE: Porting old Delphi2007 application using TServerSocket.... - BosseB - 10-01-2020 Just to close this thread with final thoughts if someone stumbles across it in the future... It all seems to work now to use several different approaches, so I decided to use this where I have a TIdBytes buffer and read all available data every pass in the Execute procedure. But I am using an event handler where the transfer container is a string, partial code below. Note that I have both set the event handler Data container as const as shown below and without const. I see no difference either way. Code: type |