Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Active sessions problem
#11
(06-20-2019, 06:05 AM)ioan Wrote: The problem it seems that has nothing to do with my timeout logic. The stuck sessions appear even without it with and with keepalive=true on all forms.

When the problem appears (this is only once in a while - about once or twice a day - usually at night at low traffic times), no sessions are terminated anymore. Even if I call Terminate, nothing happens. To test, I added this code to my application:

and no sessions were terminated (about 95% of the active sessions when the problem occurs will have the levelstring='' and activeform=nil). To test some more, I'll put the same code without the "if" and try to terminate all active sessions. I'll report back when I have the results.

Hi Ioan,

Maybe I missed something but I thought that you were saying that you weren't using KeepAlive (so I would expect that KeepAlive == False in all forms) and that you had an increasing number of sessions, which is not exactly the same as "sessions not terminating"...

Looking at your code, in theory, IWApplication.ActiveForm = nil should always evaluate to false, unless you are doing something like destroying all forms yourself before the application terminates.
Reply
#12
Hi Alex,
I disabled my timeout logic in my code and I used the "keepalive=true" and I still got the stuck sessions. I do not terminate sessions by hand anywhere in my application in normal usage, the code that I posted in my previous post was just to test if I can kill the sessions that have activeform=nil, but I can't. 

So, like I said in my original post, at night, when the traffic is low, I find sometimes thousands of active sessions and some of them have the activeform=nil. How do they appear, I have no idea. I posted in my original post a pastebin with a list of the active sessions when this problem appeared, here it is again:
https://pastebin.com/raw/x0YASKBS
In that list, all the sessions that show ActiveForm=undef have the ActiveForm=nil.
Also, there are multiple duplicate sessions that have an ActiveForm but 100% they shouldn't be active.

I know how much traffic is on the server, so to me it appears that sometimes, when the server has no or low activity, all the sessions that were terminated by the server logic (user closes the browser for example) are revived. Of course this is a guess, but considering that the application doesn't have more than - let's say - 30-40 simultaneously sessions at peak times, it would take a lot of time to collect over 3000 sessions. Again, to make myself as clear as I can, most of the time the server appears to work properly and if I list the active sessions I see 30-40 at peak times, but then, at random - usually when the server should have no activity, when I check the active sessions I see hundreds or thousands and many of them have the ActiveForm=nil and they are never terminated by the server (those users aren't online at 3AM) and I can't terminate them either with the code from my previous post. The only way to fix this is to restart the application and then it will work fine again for a while until it doesn't.

As a side note, because I'm not sure if this is related: 
Playing with the code bellow, trying to kill sessions manually (again, this code is nowhere used in the application that mentioned above) I see that if I start a session to the server (opening a form in the browser), the following code will kill the session right away while the browser is still open, but it will not kill the session when the browser is already closed and the session will timeout by itself


Code:
    xList := GSessions.LockList(False);
    try
      for i := 0 to xList.Count - 1 do
      begin
        App := TIWApplication(xList[i]);
        App.Terminate;
      end;
    finally
      GSessions.UnLockList(xList);
    end;
Reply
#13
Hi Ioan,

is this app open to the internet? Some bots sometimes are able to do some weird stuff.

It would be good if you turn on ServerController.LogSessionEvents property and wait until it happens again. Also, if you can include the session ID in that log you generate yourself, we could use it along with the session logging....
Reply
#14
Hi again,

after checking your data, seems that several "ghost" sessions have the same IP address and browser as the real sessions.
So, I'm guessing that, somehow, another request from the same source is causing a new session to be created (and stay alive even when the real session ends)... If that is happening, the LogSessionEvents switch should be enough to identify which request is triggering this
Reply
#15
Hi Alex,

Yes the application is open to the internet and it was happening right now, it's almost 1am where the server runs so I'm pretty sure most of those connections aren't from real users.

Here are the files, the list of the connections and the logsessionevent file:
Reply
#16
I'll have a look and let you know ASAP.

Cheers
Reply
#17
Interesting.... which IW version are you using?

From the session data, looks like you have some content handlers... Do you terminate the session after executing these content handlers?
Reply
#18
Some interesting data I collected from your log file:
- Most ghost sessions were created by Firefox (>90%).
- They all have the same IP addresses of legitimate users (same IP address has another "normal" session, at an earlier point in time)
- almost all ghost sessions have only one request. The request created a new session and session creation events ran until completion (no exceptions)
- after this first request, session stays inactive (it is alive but doesn't receive any other request) but is not collected by timeout thread

Given this information, seems to me that (the ultra-mega-stupid) Firefox "feature" which refreshes the toolbar images in "Top sites" toolbar is responsible for that "ghost" request.

However, this "feature" is well known and we have extensively tested this behavior. I'm not sure if some initialization code you have might be interfering with this...

The normal behavior would be:
- First request creates a session. Timeout of that session is set to 1 minute (the minimum timeout).
- If no other request is received, the session will be collected by the timeout thread on its first (or second) run (it runs once each 1 minute)
- If a second request is received, the session timeout of that specific session is set to the default value specified by server controller
Reply
#19
(06-26-2019, 10:06 AM)Alexandre Machado Wrote: Interesting.... which IW version are you using?

From the session data, looks like you have some content handlers... Do you terminate the session after executing these content handlers?

I'm using IW 15.0.23 and Delphi 10.3.1.

Yes, all the content handlers end with aSession.Terminate;

(06-26-2019, 10:55 AM)Alexandre Machado Wrote: Some interesting data I collected from your log file:
- Most ghost sessions were created by Firefox  (>90%).
- They all have the same IP addresses of legitimate users (same IP address has another "normal" session, at an earlier point in time)
- almost all ghost sessions have only one request. The request created a new session and session creation events ran until completion (no exceptions)
- after this first request, session stays inactive (it is alive but doesn't receive any other request) but is not collected by timeout thread

Given this information, seems to me that (the ultra-mega-stupid) Firefox "feature" which refreshes the toolbar images in "Top sites" toolbar is responsible for that "ghost" request.

However, this "feature" is well known and we have extensively tested this behavior. I'm not sure if some initialization code you have might be interfering with this...

The normal behavior would be:
- First request creates a session. Timeout of that session is set to 1 minute (the minimum timeout).
- If no other request is received, the session will be collected by the timeout thread on its first (or second) run (it runs once each 1 minute)
- If a second request is received, the session timeout of that specific session is set to the default value specified by server controller

I'll have to collect a bigger list on stuck sessions and the corresponding logsessionevent file because usually the stuck sessions are not just from firefox (at least that what it seems in the first list I posted in my first post). What I don't understand is why those sessions appear only at random times, the server runs fine for many hours and then... bam, 3000 sessions.

Also, if it matters, the server is built using http.sys.
Reply
#20
Do you have some code in OnBrowserCheck event? It is interesting to note that user agent string of browsers are empty which suggests me that you might be creating a TFirefox instance as a fall back maybe?
Yes, not all are Firefox. On the sample I have, most are, though.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)