TidHTTPServer and Orphaned Connection Threads

[takyon_dg]

So I've got an HTTP Service and after about 25,000-35,000 connections in a live environment, I start to get "EOutofMemory" errors and eventually the OnListenException event starts getting triggered with "Thread creation error: Not enough storage is available to process this command". 

Eventually the service just can't accept any new connections and we get other evidence if resource limitation issues in the main service thread and anything that attempts to write to the disk. 

Now there are plenty of EIdOSSLAcceptError and EIdSocketError (usually 10054, 10053 or 10060) triggered in the generic HTTP Exception handler. I've recently updated from 1.0.1t to 1.0.2s on the openssl side to see if it mitigates some of these ssl protocol errors. Anyhow, trapped inside my OnCommandGet Handler, the most common exception (Socket Error #10054. Connection reset by peer.) is around a GET request that serves up an image via SmartServeFile. But it's not specific to any one image and it's intermittent and not something I've been able to trigger in the development environment, we figure the client just drops connection before SmartServe can complete and this is what it looks like when it happens. Most of the 'heavy lifting' for the service happens in a POST process.

Now I've tried creating some exceptions in my OnCommandGet handler (sending bad data) to see if I'm cleaning up in my try/finally and exceptions and it all looks good. Eurekalog doesn't report any leaked memory and my thread count stays stable. 

In the live environment, I see the service is steadily growing in thread count. Seems to die somewhere around 1500 threads after about 6-10hrs depending on traffic, however plenty of hardware resources left (RAM and SSD). A simple restart of the service works just fine. So this implies there's a bunch of orphaned threads. Looking at process monitor, I can see hundreds of threads what would likely have started as CommandGet connection threads, still existing from HOURS ago. CPU utilization is very low, and the application handling these thousands of connections doesn't use up much cycles... so it doesn't appear these threads are stuck in any kind of loop. The Stack for all of these 'old' threads just shows:

ntoskrnl.exe!KeSynchronizeExecution+0x5bd6
ntoskrnl.exe!KeWaitForMultipleObjects+0x109d
ntoskrnl.exe!KeWaitForMultipleObjects+0xb3f
ntoskrnl.exe!KeWaitForMutexObject+0x377
ntoskrnl.exe!KeQuerySystemTimePrecise+0x881
ntoskrnl.exe!ObDereferenceObjectDeferDelete+0x28a
ntoskrnl.exe!KeWaitForMultipleObjects+0x1284
ntoskrnl.exe!KeWaitForMultipleObjects+0xb3f
ntoskrnl.exe!KeWaitForMutexObject+0x377
ntoskrnl.exe!NtWaitForSingleObject+0xf8
ntoskrnl.exe!_setjmpex+0x68a3
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x540
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x3a5
wow64.dll!Wow64KiUserCallbackDispatcher+0x4151
wow64.dll!Wow64LdrpInitialize+0x120
ntdll.dll!LdrInitializeThunk+0x16d
ntdll.dll!LdrInitializeThunk+0xe

None of which is my windows service (obviously), but it suggests the thread(s) are still alive.

I'm wondering how I can dig deeper into these seemingly orphaned threads to figure out how they got orphaned. Even with the EIdOSSLAcceptError and EIdSocketError's the number of those that occur, do not accumulate to the number of threads. And yes, I'm re-raising EIdExceptions in my handlers. 

Any ideas would be appreciated. 

Indy version is 10.6.1.5182 on XE7. 

Basic service structure is in the OnCommandGet is...

Code:

procedure TsvcThing.srvHTTPCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
  XMLDoc: IXMLDocument;
  adoConn: TADOConnection;
  adoQuery, adoQueryOther: TADOQuery;
begin
  try
    CoInitialize(nil);
    try
      if ARequestInfo.Command = 'GET' then
        //serve up the requested file
        AResponseInfo.SmartServeFile(AContext, ARequestInfo, Path + filename);

      if ARequestInfo.Command = 'POST' then
      begin
        adoConn := TADOConnection.Create(nil);
        adoQuery := TADOQuery.Create(nil);
        adoQueryOther := TADOQuery.Create(nil);

        try
          adoConn.ConnectionString := 'valid connection string';
          adoQuery.Connection := adoConn;
          adoQueryOther.Connection := adoConn;

          if not adoConn.Connected then
            adoConn.Connected := true;

          XMLDoc := LoadXMLData(ReadStringFromStream(ARequestInfo.PostStream, -1));
          try
            //perform various queries (Open and ExecSQL) based on XML request

            //set fields in XML response

            if error_processing_xml_or_data then
              AResponseInfo.ResponseNo := 400
            else
            begin
              AResponseInfo.ContentType := 'application/xml';
              AResponseInfo.ResponseNo := 200;
              AResposneInfo.ContentText := XMLDoc.XML.Text;
            end;
          finally
            XMLDoc.Active := False;
          end;
        finally
          FreeAndNil(adoQueryOther);
          FreeAndNil(adoQuery);
          FreeAndNil(adoConn);
        end;    
      end;
    finally
      CoUnintialize;
    end;
   except
     on E: Exception do
       if Log then
          WriteLog(E.Message);
     on EIdException do raise;
   end;
end;