TidSmtpServer Stops responding

[bluewwol]

Hi,

I have TidSmtpServer setup and running, it runs fine for a period of time, then stops responding.  When I set the server active := True it creates 2 threads for each IP binding I have made.  Each of these IP's do for a period of time handle incomming requests as expected.  Then slowly each IP stops responding to external requests and the 2 threads created when the server was set to active appear to terminate. 

The bindings all are still set, and restarting to server (Active set to False then True) does not solve the problem.

I presume that since it is the Listener Threads that are failing we should expect to see the onListenException be fired when that occurs, however I never see this get called.

Code:

procedure TdSmtpServ.WwolSmtpServListenException(AThread: TIdListenerThread; AException: Exception);
begin
 Log(now, 0, -9, AThread.Binding.IP, AThread.Binding.PeerIP, AException.ClassName, '[' + IntToStr(TmySmtpItem(AThread.Data).Id) + '] - ' +
   AException.message + ' - WwolSmtpServListenException');
   TmySmtpItem(AThread.Data).LastEvent := 'WwolSmtpServListenException ' + AException.ClassName + ' - ' + AException.message;
end;

My bindings are created as follows

Code:

procedure TdSmtpServ.SetBoundIp(ServerId: integer);
var
 Query                 : TMyQuery;
 zConn1                : TMyConnection;
 myBinding             : TIdSocketHandle;
begin
 Query := TMyQuery.Create(nil);
 zConn1 := TMyConnection.Create(nil);
 try
   try
     dmMain.SetDBReadcomp(zConn1, Query, 'SetBoundIp');
     if not zConn1.connected then
       dmMain.ConnectToDB(zConn1, 'SetBoundIp');
     with Query do
     begin
       SQL.Text := 'SELECT idIPmanager, InternalIP, PublicIP, ReverseDnsName, ServerId, Active ' + ' FROM ipmanager ' + ' WHERE Active = True and ServerId = ' + IntToStr(ServerId)
         + ';';
       Open;
       First;
       while not Eof do
       begin
         myBinding := WwolSmtpServ.Bindings.Add;
         myBinding.Port := 25;
         myBinding.IP := Query.FieldByName('InternalIP').AsString;
         myBinding.DisplayName := Query.FieldByName('ReverseDnsName').AsString;
         Next;
       end;
     end;
   except
     on E: Exception do
     begin
       Log(now, 0, -1, 'SetBoundIp', myBinding.IP, E.ClassName, E.message + ' - ' + ' SetBoundIp');
     end;
   end;
 finally
   Query.Free;
   zConn1.Disconnect;
   zConn1.Free;
 end;
end;

And myver config is set as follows

Code:

   with WwolSmtpServ do
   begin
     ReuseSocket := rsOSDependent;
     AllowPipelining := True;
     ServerName := 'WWOL SMTP server';
     DefaultPort := 25;
     ListenQueue := 50;
     MaxConnections := 0;
     MaxMsgSize := 0;
     UseNagle := True;
     UseTLS := utNoTLSSupport;
     Intercept := nil;
     IOHandler := nil;
   end;

Any pointers on where to look would be most welcome

-Allen

[bluewwol]

This seems to belong to the self inflicted injury type.

I have made 2 changes in my app that seem to have made a positive impact (although only in test for about an hour, where previously stability issues were occuring within that timeframe).

The addition of a try except block in my onConnect event

Code:

procedure TdSmtpServ.WwolSmtpServConnect(AContext: TIdContext);
var
 HostName              : string;
begin
 dmServiceSupport.SenderCS.Acquire;
 try
   inc(unique_id);
 finally
   dmServiceSupport.SenderCS.Release;
 end;
 try
   AContext.Connection.Socket.ReadTimeout := 15000;

   AContext.Data := TmySmtpItem.Create(unique_id, 'WwolSmtpServConnect', AContext.Binding.PeerIP, AContext.Binding.IP, AContext.Connection.ClassName, HostName);

   Log(now, 0, -4, AContext.Binding.IP, AContext.Binding.PeerIP, '[' + IntToStr(TmySmtpItem(AContext.Data).Id) + '] - ' + 'WwolSmtpServConnect',
     '[' + IntToStr(TmySmtpItem(AContext.Data).Id) + '] - ' +
     ' WwolSmtpServConnect connected at ' + DateTimeToStr(TmySmtpItem(AContext.Data).Start) +
     ' - ' + TmySmtpItem(AContext.Data).ThreadName +
     ' - SourceIP: ' + TmySmtpItem(AContext.Data).SourceIP +
     ' - DestIP: ' + TmySmtpItem(AContext.Data).DestIP +
     ' - ' + TmySmtpItem(AContext.Data).LastEvent);

 except
   on E: Exception do
   begin
     Log(now, 0, -4, AContext.Binding.IP, AContext.Binding.PeerIP, '[' + IntToStr(unique_id) + '] - ' + 'WwolSmtpServConnect EXCEPTION',
       '[' + IntToStr(unique_id) + '] - ' +
       ' - SourceIP: ' + AContext.Binding.PeerIP +
       ' - DestIP: ' + AContext.Binding.IP +
       ' - Connect EXCEPTION: ' + E.Message);
   end;
 end;
end;

and removed the 
TmySmtpItem(AThread.Data).LastEvent := 'WwolSmtpServListenException ' + AException.ClassName + ' - ' + AException.message;
statement from the onListenException event.

One or both of these appear to have made a positive impact.  I presume it must be the later as that impacts the Listener threads.

-Allen

[rlebeau]

The addition of a try except block in my onConnect event

If an exception occurs in the OnConnect event, you are logging it, but then you are discarding it. Don't do that. Re-raise the exception and let the server handle it so the client can be disconnected.

Alternatively, get rid of the try...except altogether, and do your error logging in the OnException event instead, which is triggered whenever an uncaught exception occurs in a client thread.

BTW, Indy has a TIdThreadSafeInteger class (and Int64 and Cardinal variants) in the IdThreadSafe unit. You could replace your unique_id variables with that class, and then get rid of SenderCS.

and removed the 
TmySmtpItem(AThread.Data).LastEvent := 'WwolSmtpServListenException ' + AException.ClassName + ' - ' + AException.message;
statement from the onListenException event.

In the OnListenException event, the AThread.Data property is nil (unless you assign your own Data object in the OnBeforeListenerRun event). That event is triggered whenever an uncaught exception occurs in a listener thread, not in a client thread. Your TmySmtpItem objects are not assigned to listener threads, they are assigned to client contexts own by client threads. So, in the OnListenerException event, AThread.Data will NEVER point to a TmySmtItem object, and as such TmySmtpItem(AThread.Data).Id and TmySmtpItem(AThread.Data).Id are invalid.

Now, why the OnListenException event would be triggered at all, is a different matter that you still need to debug. Just fix your logging to not access invalid data fields.

[bluewwol]

Now, why the OnListenException event would be triggered at all, is a different matter that you still need to debug.  Just fix your logging to not access invalid data fields.

Remy;

The OnListenException is still being triggered fairly frequently, however with the try .. excepts removed Indy handles the problem and keeps on ticking.  So my app is now functional over time now.

As to the cause of the exceptions, I will let it run over the weekend and review on Monday, however a comonality on all of these exceptions is that have occured this afternoon is that Indy shows the AThread.Binding.PeerIP to be ''.  In my judgement this appears to be a consequence of questionable probing of the IP??  right now I am logging this, no longer interfering with Indy handling the exception and the world continues function as expected.

As I learn more on Monday I will provide feedback,

Thanks again for your assistance,

-Allen

[rlebeau]

The OnListenException is still being triggered fairly frequently, however with the try .. excepts removed Indy handles the problem and keeps on ticking.  So my app is now functional over time now.

It wasn't just about removing try..except blocks, but about fixing your code to stop accessing things in the incorrect way.

As to the cause of the exceptions, I will let it run over the weekend and review on Monday, however a comonality on all of these exceptions is that have occured this afternoon is that Indy shows the AThread.Binding.PeerIP to be ''.  In my judgement this appears to be a consequence of questionable probing of the IP??

Again, you are mixing up LISTEN threads with CLIENT threads.  They are TWO DIFFERENT THINGS.  A listen thread accepts a client on a listening port and spawns a NEW THREAD to manage that client, then goes back to accept the next client on the port.  So, there is no Peer on a LISTEN thread, so of course AThread.Binding.PeerIP will always be blank, as AThread.Binding represents a listening socket, not a client socket.

right now I am logging this, no longer interfering with Indy handling the exception and the world continues function as expected.

Maybe try something more like this:

Code:

// OnException event handler
procedure TdSmtpServ.WwolSmtpServException(AContext: TIdContext; AException: Exception);
var
 Ctx: TmySmtpItem;
 CtxId: string;
begin
 Ctx := TmySmtpItem(AContext.Data);
 if Ctx <> nil then CtxId := IntToStr(Ctx.Id);

 Log(now, 0, -4, AContext.Binding.IP, AContext.Binding.PeerIP, '[' + CtxId + '] - ' + 'WwolSmtpServ Client EXCEPTION',
   '[' + CtxId + '] - ' +
   ' - Peer IP: ' + AContext.Binding.PeerIP +
   ' - Binding IP: ' + AContext.Binding.IP +
   ' - EXCEPTION: ' + AException.ClassName + ' - ' + AException.Message);

 if Ctx <> nil then
   Ctx.LastEvent := 'WwolSmtpServException ' + AException.ClassName + ' - ' + AException.Message;
end;

// OnListenException event handler
procedure TdSmtpServ.WwolSmtpServListenException(AThread: TIdListenerThread; AException: Exception);
begin
 Log(now, 0, -9, AThread.Binding.IP, '', 'WwolSmtpServ Listen EXCEPTION',
   ' - Binding IP: ' + AThread.Binding.IP +
   ' - EXCEPTION: ' + AException.ClassName + ' - ' + AException.Message);
end;