TIdSmtpServer not freeing threads - Printable Version +- Atozed Forums (https://www.atozed.com/forums) +-- Forum: Indy (https://www.atozed.com/forums/forum-8.html) +--- Forum: Indy General Discussion (https://www.atozed.com/forums/forum-9.html) +--- Thread: TIdSmtpServer not freeing threads (/thread-768.html) Pages:
1
2
|
TIdSmtpServer not freeing threads - bluewwol - 10-24-2018 Hi, I have an extremely simple SmtpServer set up to simply receive emails and write them to disk. My problem is that over several hours the number of active threads just keeps growing, leading me to believe they are not being freed correctly. I further have to conclude it is quite likely as a result of something I am doing or not doing that is causing this, however for the life of me I cant seem to identify the problem. I have attached my sources, they do contain an external reference to a text file logging routine that is thread safe (file protected by critical section) that has been used without issue for many years. In my current setup I am binding to 20 IP addresses and they all seem to receive and handle the incomming email as expected. The load on the system is between 1 and 5 messages per minute. after 24 hours there are well over 1,000 threads active, this would not be all the threads created (fewer than messages received) in that period. By my speculation my error should be in my configuration of the component with WwolSmtpServ do begin ReuseSocket := rsTrue; AllowPipelining := True; ServerName := 'WWOL SMTP server'; DefaultPort := 25; ListenQueue := 50; MaxConnections := 0; MaxMsgSize := 0; TerminateWaitTime := 500; UseNagle := True; UseTLS := utNoTLSSupport; end; OR in my component exception handlers procedure TdSmtpServ.WwolSmtpServException(AContext: TIdContext; AException: Exception); begin try if ((AException is EIdConnClosedGracefully) or (AException is EIdNotConnected)) then begin Exit; end else begin if AContext.Connection.Connected then begin Log(Now, 0, 1, AContext.Binding.IP, AContext.Binding.PeerIP, AException.ClassName, AException.message + ' - ' + ' SmtpServException still connected'); AContext.Connection.Disconnect; end else Log(Now, 0, 1, AContext.Binding.IP, AContext.Binding.PeerIP, AException.ClassName, AException.message + ' - ' + ' SmtpServException disconnected'); // AContext.Connection.Disconnect(False); end; except on E: Exception do begin dmMain.TextLog('WwolSmtpServException : ' + E.Message); dmMain.TextLog('WwolSmtpServException : ' + AContext.Binding.IP + ' : ' + AContext.Binding.PeerIP + ' : ' + AException.ClassName + ' : ' + AException.message); end; end; end; procedure TdSmtpServ.WwolSmtpServListenException(AThread: TIdListenerThread; AException: Exception); begin // Log(logModerate, 0, 'TdSmtpServ', 'WwolSmtpServListenException', '', '', AException.Message); Log(Now, 0, -3, AThread.Binding.IP, AThread.Binding.PeerIP, AException.ClassName, AException.message + ' - WwolSmtpServListenException'); // AThread.Connection.Disconnect; end; OR in the component disconnect event procedure TdSmtpServ.WwolSmtpServConnect(AContext: TIdContext); begin Log(Now, 0, -4, AContext.Binding.IP, AContext.Binding.PeerIP, 'WwolSmtpServConnect', AContext.Binding.DisplayName + ' - ' + ' WwolSmtpServConnect'); end; I am hoping I have missed something stupid and some fresh eyes might see the obvious, Thanks for any assistance -Allen RE: TIdSmtpServer not freeing threads - rlebeau - 10-24-2018 (10-24-2018, 05:04 PM)bluewwol Wrote: after 24 hours there are well over 1,000 threads active, this would not be all the threads created (fewer than messages received) in that period. That typically happens when either:
(10-24-2018, 05:04 PM)bluewwol Wrote: By my speculation my error should be in my configuration of the component No. The config is fine. (10-24-2018, 05:04 PM)bluewwol Wrote: OR in my component exception handlers The handlers you have shown are fine in general. Though, you don't need to call Disconnect() in the OnException event, the server will do that for you after the event handler exits. And the PeerIP is not valid in the OnListenException event, as that event is for a listening socket, not a connected client socket. (10-24-2018, 05:04 PM)bluewwol Wrote: OR in the component disconnect event That handler is fine. (10-24-2018, 05:04 PM)bluewwol Wrote: I am hoping I have missed something stupid and some fresh eyes might see the obvious, Nothing obvious that I see. You are just going to have to debug your code to find out what is actually happening. For instance, are you getting an OnDisconnect event triggered for every client that connects? Under normal conditions, you should be. RE: TIdSmtpServer not freeing threads - bluewwol - 10-24-2018 Remy; These seem most likely, I will review my code carefully and make certain I am not leaving anything hanging. As always thanks for your kind assistance, -Allen (10-24-2018, 06:48 PM)rlebea u Wrote: That typically happens when either: RE: TIdSmtpServer not freeing threads - bluewwol - 10-25-2018 Remy; The problem appears practical, not technical. The issue in reality is my server not keeping up with probing attempts from the outside. My initial steps have greatly reduced the impact of the problem. The two changes I have made are to procedure TdSmtpServ.WwolSmtpServMailFrom(ASender: TIdSMTPServerContext; const AAddress: string; AParams: TStrings; var VAction: TIdMailFromReply); begin if Trim(AAddress) = '' then begin VAction := mReject; end else begin VAction := mAccept; end; end; and TerminateWaitTime := 30; These are simplistic and incomplete, however have largely eliminated the current problem. The issue really is not properly understanding how my app is interacting with the world. Thanks again for the assistance. -Allen RE: TIdSmtpServer not freeing threads - bluewwol - 10-26-2018 Remy; I do still have some threads not being freed, but will focus some attention on those in the next few days. Threads associated with VAction := mReject from OnMailFrom event and VAction := rInvalid from OnRcptTo event and threads where the client disconnected are currently freed after five minutes. There are a few threads the remain active well beyond this that I have not yet determined a cause. I had also assumed that the TerminateWaitTime would impact the time interval for these to be cleaned up, however it seems TerminateWaitTime has no impact on this. In the mean time I have adjusted the OnMailFrom event to include a ASender.Connection.Disconnect which swams to be making a big improvement, however is this a foolish solution? Code: procedure TdSmtpServ.WwolSmtpServMailFrom(ASender: TIdSMTPServerContext; const AAddress: string; AParams: TStrings; var VAction: TIdMailFromReply); I have also added a procedure on a 30 minute timer that cleans up other threads (a heavy handed remedy of the symptom) Code: procedure TdSmtpServ.TerminateThreads; While maybe technically inept my app now frees threads immediately after a message is received, after the sender is determent bad, and after a recipient is determined to be bad. After five minutes the threads Indy believes to be complete are cleaned up. After thirty minutes I brute force clean up the remaining threads. Thanks again for the help, RE: TIdSmtpServer not freeing threads - rlebeau - 10-26-2018 (10-26-2018, 12:36 AM)bluewwol Wrote: I do still have some threads not being freed And what are those threads actually doing? And do you get an OnDisconnect event fired for them? (10-26-2018, 12:36 AM)bluewwol Wrote: Threads associated with VAction := mReject from OnMailFrom event and VAction := rInvalid from OnRcptTo event and threads where the client disconnected are currently freed after five minutes. If a client disconnects, its thread should be stopping itself immediately. If it is not, then there is a serious problem with your server. (10-26-2018, 12:36 AM)bluewwol Wrote: I had also assumed that the TerminateWaitTime would impact the time interval for these to be cleaned up, however it seems TerminateWaitTime has no impact on this. No, it does not. In fact, it has NO EFFECT WHATSOEVER in Indy 10 at all. It is a legacy property that is NO LONGER IMPLEMENTED. However, even in Indy 9 and earlier, where it USED to be implemented, it had NO EFFECT on running client threads during normal server usage. The property applied ONLY when the server was being deactivated. The property specifies the total amount of time the server waits for all active threads to terminate. In Indy 8, if a timeout occurred, the deactivation would simply exit immediately. In Indy 9, the deactivation would raise an EIdTerminateThreadTimeout exception instead. In Indy 10, the property is ignored altogether, deactivation waits as long as it takes for all threads to terminate. (10-26-2018, 12:36 AM)bluewwol Wrote: In the mean time I have adjusted the OnMailFrom event to include a ASender.Connection.Disconnect which swams to be making a big improvement, however is this a foolish solution? Yes, it is foolish, not to mention a violation of the SMTP protocol. Just because a client sends a MAIL FROM command with a blank address should not doom that client to an immediate disconnection. In fact, MAIL FROM:<> is a valid command (see RFC 5321 Section 3.6.3 regarding "undeliverable mail" notifications). Now, if a client sends multiple commands that repeatedly fail, that would be a different case to consider. You might also consider using the TIdSMTPServer.OnSPFCheck event to make sure clients are coming from IPs that are even authorized to access your server in the first place. (10-26-2018, 12:36 AM)bluewwol Wrote: I have also added a procedure on a 30 minute timer that cleans up other threads (a heavy handed remedy of the symptom) Very heavy handed. I would opt for a different solution that does not rely on a looping timer, or even accessing the server's Contexts list. First, start with assigning a non-infinite ReadTimeout for each connected client, such as in the OnConnect event. If a client connects and then doesn't send a command for awhile, an exception will be raised that will close the connection. You might also go as far as using AContext.Connection.Socket.SetSockOpt() to set a non-infinite SO_SNDTIMEO writing timeout, too (Indy currently does not have any kind of WriteTimeout property), to handle cases where a client connects and does not read anything the server sends, causing a thread blockage if the client's inbound buffer fills up. Beyond that, TIdSMTPServer is derived from TIdCmdTCPServer, which has virtual DoExecute() and ReadCommandLine() methods. I would override one of those to handle the TmySmtpItem.StartStart check. And I would not use a TDateTime for that check, since Now() is sensitive to local clock changes. I would use Indy's Ticks64() and ElapsedTicks() functions instead, which do not care about the clock at all. RE: TIdSmtpServer not freeing threads - bluewwol - 10-26-2018 Remy; I dropped a log intercept on the server and trapped the following: Code: 0.0.0.0:0 Stat Disconnected. I will dig into this further tomorrow. -Allen RE: TIdSmtpServer not freeing threads - bluewwol - 10-26-2018 Remy, I have opened a ticket with gmail and my network provider. Perhaps the issue lies with them. If there is anything else I might need to run down I am open to suggestion. Thanks again for your diligent assistance, -Allen RE: TIdSmtpServer not freeing threads - bluewwol - 10-26-2018 Reny; Quick note, ran this piece of software on an unrelated network for a short time this morning and all behaved as expected. So the problems are somehow related to the network and not the software. The use case we have run into is unuseual but interesting. I will circle back to your prior comments and figure out a more elegant solution to cleaning up these "abandoned" threads. Thanks for the help, and I will come back add in the resolution to this in case it can help someone else out in future. -Allen RE: TIdSmtpServer not freeing threads - rlebeau - 10-26-2018 (10-26-2018, 02:29 AM)bluewwol Wrote: I dropped a log intercept on the server and trapped the following: Setting a reasonable ReadTimeout for each client connection would handle that scenario. I seriously doubt that a high-profile email provider like GMail or Hotmail would exhibit behavior like this, though. Did you verify the hung clients are REALLY who they claim to be? You can perform a reverse DNS lookup on the client's IP address to get its actual hostname so you can check its domain name. Optionally, also use the OnSPFCheck event to check if that hostname is authentic. Just because a client claims to be Google or Hotmail does not mean it REALLY is. For example: Code: type |