TIdFTPServer slow-ish

[rlebeau]

Device establishes a new FTP connection - login etc - for every file and disconnects after each, then immediately logs in for the next file.

Why? That is highly inefficient, unless there is a significant delay in between sending each file.

The problem is the delay between lines 74 & 75 - might only be 300ms, but that adds up over 200+ files.
(Using Filezilla for the FTP server is about 3 times faster than using Indy.)

Why is there a TCP retransmission in that time frame? That is suspicious, but I doubt that is Indy's fault directly, since such things are handled by the OS socket stack behind the scenes.

I initially noticed that after the final data packet (line 70 above) Indy wasn't responding with the expected 226 'Closing data connection.' , so I thought (I'm a FTP novice) that perhaps the device wasn't closing the data connection.

Yes, it is. You can see that with the 2 [FIN,ACK] packets, one from the client to the server, and one from the server to the client. So the data connection is fully closed.

However, what I do see is that the client is clearly not waiting to receive the 226 reply before then sending its next commands (RNFR, RNTO, and QUIT). That is a violation of the FTP protocol spec.

But, in any case, TIdFTPServer IS responding to those subsequent commands, which can happen only if DoDataChannelOperation() has either fully exited, or has queued those commands inside of CheckControlConnection() AND then processed that queue after sending the 226 reply. Either way, the 226 reply should be getting sent before any new commands are processed. I don't see any code path in DoDataChannelOperation() that would prevent that from happening without the data connection getting stuck completely, which is clearly not happening.

So I hacked the ReadFromStream procedure in TIdFTPServer.DoDataChannelOperation() to exit if the packet size was under 1k. That resulted in Indy providing the 226 response

The only way that could make any difference is if under the original code, the reading loop was not exiting correctly on disconnect of the data connection. However, if that were the case, then ReadFromStream() would be stuck in an endless loop and the subsequent commands would simply be queued and not processed, but that is clearly not happening since they ARE being processed.

Looking at the code, I wonder if it would be better to simply have ReadFromStream() check FDataChannel.IOHandler.Connected before calling CheckControlConnection() instead of after. Can you try that and see if it helps?

Code:

repeat
  //AContext.FDataChannel.FDataChannel.IOHandler.CheckForDisconnect(False); // <-- I THINK THIS IS REDUNDANT
  AContext.FDataChannel.FDataChannel.IOHandler.ReadStream(LM, DEF_BLOCKSIZE, True);
  if not AContext.FDataChannel.FDataChannel.IOHandler.Connected then Break; // <-- MOVED HERE
  CheckControlConnection(AContext, ACmdQueue);
until False{not AContext.FDataChannel.FDataChannel.IOHandler.Connected};

But there's still a delay before the 226 response (line 81).

Have yo tried stepping into Indy's code with the debugger to see what ReadFromStream() is really doing during the troublesome timeframes? And why sending the 226 reply is not being reached?

After a quick Google of 'FIN ACK' it seems that the device is to blame, as it shouldn't ?? be including the ACK as it has nothing to acknowledge.

Possibly.

So I'm thinking the unexpected ACK is throwing Indy off.

Indy doesn't care about the ACKs directly. That is the job of the OS to handle. Indy only cares if the socket is readable or closed, as reported by the OS.

Went hunting for Indy's FIN (and ACK) handling and came up empty (except for IdRawHeaders.pas and IdRawFunctions.pas which appear to not be referenced). Bit more Googling seems to show they are handled at a lower level than Indy.

Correct. The OS handles them, and then reports the socket status back to Indy (or any other app that is using a socket).

(But if that's the case how come Filezilla works???)

I can't answer that since I don't have Filezilla's source code.