Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Intraweb instantly 'crashes' when above 340 users
#1
We have an Intraweb 14.2.9 web application running for our customers, that that works smooth and very fast.

However, when we reach a critical amount of 340 sessions, the intraweb application instantly 'crashes' (response becomes too slow to even show the logon page). Logged on users can work for a little while, new users are directly unable to logon. There is no build up in slowness, it just instantly happens as soon as we reach the user amount.

This takes every time about 15 minutes to automatically resolve and returns back to normal, after which users can logon again and all is as fast as prior to the ordeal.
Due to less users, during weekends this never happens...on peak days it happens multiple times per day.

We did a lot of research on the complete infrastructure and architecture, and we tried several different solutions, but non seems to resolve the underlying issue.
We are now 100% sure that the issue is within the intraweb web application itself. We have multiple applications (mobile apps, web applications, windows applications) connected to the database, including an exact copy of the project with less users, and they all remain to work fine when this crash happens.

Last night we added 8 extra cpu's to the database server, just to see if we would go over the 340 sessions. It did not, it happens every time for 15 minutes.

Before we run into technical details, is this behavior something someone recognizes? Any clues on where to dig? We are running out of options, so any input would be welcome.

Thanks a lot!
Reply
#2
(11-30-2021, 12:30 PM)AtozedJunky Wrote: We have an Intraweb 14.2.9 web application running for our customers, that that works smooth and very fast.

However, when we reach a critical amount of 340 sessions, the intraweb application instantly 'crashes' (response becomes too slow to even show the logon page). Logged on users can work for a little while, new users are directly unable to logon. There is no build up in slowness, it just instantly happens as soon as we reach the user amount.

This takes every time about 15 minutes to automatically resolve and returns back to normal, after which users can logon again and all is as fast as prior to the ordeal.
Due to less users, during weekends this never happens...on peak days it happens multiple times per day.

We did a lot of research on the complete infrastructure and architecture, and we tried several different solutions, but non seems to resolve the underlying issue.
We are now 100% sure that the issue is within the intraweb web application itself. We have multiple applications (mobile apps, web applications, windows applications) connected to the database, including an exact copy of the project with less users, and they all remain to work fine when this crash happens.

Last night we added 8 extra cpu's to the database server, just to see if we would go over the 340 sessions. It did not, it happens every time for 15 minutes.

Before we run into technical details, is this behavior something someone recognizes? Any clues on where to dig? We are running out of options, so any input would be welcome.

Thanks a lot!

Hi ,
In most cases is due to bad code. Thread safe not respecting memory leaks on code or access violation. Test with fastmm with fulldebug mode to test. 
I hope this can help
Reply
#3
(11-30-2021, 12:30 PM)AtozedJunky Wrote: We have an Intraweb 14.2.9 web application running for our customers, that that works smooth and very fast.

However, when we reach a critical amount of 340 sessions, the intraweb application instantly 'crashes' (response becomes too slow to even show the logon page). Logged on users can work for a little while, new users are directly unable to logon. There is no build up in slowness, it just instantly happens as soon as we reach the user amount.

This takes every time about 15 minutes to automatically resolve and returns back to normal, after which users can logon again and all is as fast as prior to the ordeal.
Due to less users, during weekends this never happens...on peak days it happens multiple times per day.

We did a lot of research on the complete infrastructure and architecture, and we tried several different solutions, but non seems to resolve the underlying issue.
We are now 100% sure that the issue is within the intraweb web application itself. We have multiple applications (mobile apps, web applications, windows applications) connected to the database, including an exact copy of the project with less users, and they all remain to work fine when this crash happens.

Last night we added 8 extra cpu's to the database server, just to see if we would go over the 340 sessions. It did not, it happens every time for 15 minutes.

Before we run into technical details, is this behavior something someone recognizes? Any clues on where to dig? We are running out of options, so any input would be welcome.

Thanks a lot!

Hi there,

There is no such limit in IntraWeb, so I don't see any reason other than a bug somewhere that is crashing the application.

I personally run stress test against different IW applications with 500, 1000, 5000 simultaneous users and I've never seen such issue.

We need more details about this in order to at least start looking into it. 

Are you deploying as ISAPI or SA (hopefully as a service, not as a GUI app)? Can you reproduce this reliably (i.e. can you go up to 339 users and crash it as soon as a new session is created every time)?
Reply
#4
Thank you for your replies!
 
For development I used Intraweb 14.2.9 with Delphi 10 Seattle (use also TMS and CGDevtools components).
The webapp is SA running as windows service.

Is the ISAPI a better approach than SA (in general for IW)?

Besides, I’m aware of newer versions, but is it possible for you to elaborate if there is any known problems regarding the sessions in older versions ?

In other words, will it help me to update or do I need to go deeper into my code? Currently I am already migrating to newer versions (Intraweb 15 and Delphi 11). I keep the older installations for backup and for small necessary adjustments in production.

What is strange for me, though, is that the issues occur after 340 sessions and 4 hours of working. If it’s indeed a thread safe error, wouldn't it be more logical for this issue to occur earlier?
Users do execute a lot of actions and retrieve a lot of data (which is handled instantly as long as we stay under 340 sessions).

-      I am using IWsession to make as much actions as possible, communicate with backed server, maintain cache datasets, load and get controls data.
-      I am not using global variables.
-      Access to server controller directly, is limited and accessed thru Critical section.
-      Communication with backed is done thru idHTTP, synchronously, inside the user session.

I pay attention to thread aware  and I do not using global variable, or if some parameters is accessed, they are treated in critical section.
Reply
#5
BTW, cool user name Smile
Reply
#6
(12-01-2021, 10:26 AM)AtozedJunky Wrote: Thank you for your replies!
 
For development I used Intraweb 14.2.9 with Delphi 10 Seattle (use also TMS and CGDevtools components).
The webapp is SA running as windows service.

Is the ISAPI a better approach than SA (in general for IW)?

Besides, I’m aware of newer versions, but is it possible for you to elaborate if there is any known problems regarding the sessions in older versions ?

In other words, will it help me to update or do I need to go deeper into my code? Currently I am already migrating to newer versions (Intraweb 15 and Delphi 11). I keep the older installations for backup and for small necessary adjustments in production.

What is strange for me, though, is that the issues occur after 340 sessions and 4 hours of working. If it’s indeed a thread safe error, wouldn't it be more logical for this issue to occur earlier?
Users do execute a lot of actions and retrieve a lot of data (which is handled instantly as long as we stay under 340 sessions).

-      I am using IWsession to make as much actions as possible, communicate with backed server, maintain cache datasets, load and get controls data.
-      I am not using global variables.
-      Access to server controller directly, is limited and accessed thru Critical section.
-      Communication with backed is done thru idHTTP, synchronously, inside the user session.

I pay attention to thread aware  and I do not using global variable, or if some parameters is accessed, they are treated in critical section.

No such problems are known in older versions. On the contrary, I know customers running IW 14 with up to 1000 concurrent sessions with no issues (as SA-Indy/x64).

I believe that migrating it to IW 15 would definitely help you because of a couple of factors: IW 15 performs better than 14 in several areas (render engine, lock-free session handling, even our IW 15 zLib is faster). Also you have a completely new deployment option which is Http.sys (it has the benefits of IIS - i.e. backed up by Windows http stack - and the benefits of SA Indy - because it can also be deployed as a stand alone service just like you today with very little changes).
Also, a huge list of bugs - new and old - have been continually fixed. All in all, I'd say that IW 15 scales better than 14, so even if it doesn't fix the underlying problem - whatever it is - it will definitely help in several areas.

Regarding ISAPI x SA: I personally don't like ISAPI because of the way the URL works, unless (a) you have a reverse proxy on top of it, or (b) you use URLRewrite module. Any of these can solve the url ending with "YourApp.dll", which I personally don't like.
But anyway, ISAPI is a full size truck and SA-Indy is a cargo van. They both can carry a lot of stuff, but it really depends on your cargo. We have introduced the Http.sys application in IW 15 which, in theory, is an easier to drive truck which could carry the same as the ISAPI app.

I'm curious about the issue though... you say that "issues occur after 340 sessions and 4 hours of working". Does it mean that, if you create 350 sessions in 2 hours the problem does not happen? On the other hand, if you keep it below 340 sessions what happens after 4 hours of working?
Reply
#7
Dear Alexandre,

Thank you for your response. We will try to speed up conversion to IW15 in Delphi 11. I will check also Http.sys, that I though was just in IW17, but good to know is in 15 too.
So, even we have proxy server above those IW services (we have 3 now), we will continue to use windows service, like you suggested.

About issue: So, limit of 340 active sessions, even some are dormant, is relative in time, because our server become busy at noon and evening.
As bandage, I move now, almost all access from server controller to user session (was some parameters loaded from a config file); what remains is just a write in exception log method, accessed thru critical section. I will check also other methods used, even is in user sessions, maybe is not thread safe. Also I limit the session timeout to 30 min (was 2 hours; been an application model, users want to keep it open all day) and seems to help a little bit, getting less limit, but is just a bandage, not a solution.
As detail also, maybe is relevant, I use just few forms, login recover password and main form, all detail, edit forms are in fact frames, derived from TFrame) used with TIWModalWindow.
Question: Do you recommend any stress test tool for IW?

As suggestion, maybe, you can make in Autozed examples, a model of best practice with setup of server controller. That been mentioned I will show you my setup, that maybe you can comment and tell me if is incomplete or too much.

Self.BoundIP := config read bound IP
self.Port := config read port number
self.RestartExpiredSession := True;
self.AllowMultipleSessionsPerUser := false;

TIWSecurityOptionsAccess(Self.SecurityOptions).mCheckSameIP := False;
TIWSecurityOptionsAccess(Self.SecurityOptions).mCheckSameUA := true;
TIWSecurityOptionsAccess(Self.SecurityOptions).CheckFormId := true;
SecurityOptions.ShowSecurityErrorDetails := False;

{CookieOptions }
CookieOptions.SessionCookies := false;
CookieOptions.HttpOnly := true; // we use Proxy
CookieOptions.UseCookies := True;

Self.SessionTimeout := Config get 'SessionTimeout'
Self.FilesDir := Self.ContentPath + Config get 'SharedFilesDir'
Self.InternalFilesDir := Self.ContentPath + Config get 'SharedFilesDir'

self.JavaScriptOptions.AjaxErrorMode := emNone;
self.LogCommandEnabled := true;
self.LogSessionEvents := false;///used ony in test

{exception logger}
self.ExceptionLogger.RegisterIgnoreException(EConvertError);
self.ExceptionLogger.RegisterIgnoreException(EUnknownBrowserException); {Unknown browser type}
self.ExceptionLogger.RegisterIgnoreException(EInvalidSession); {Invalid session}
self.ExceptionLogger.RegisterIgnoreException(EExpiredSession); {Expired session}
self.ExceptionLogger.RegisterIgnoreException(EIWNoSessionId); {No session ID}
self.ExceptionLogger.LogType := ltFile; // ltEventLog;
self.ExceptionLogger.FilePath := Self.ContentPath + 'Exceptions\';
self.ExceptionLogger.FileName := 'Exceptions.log';
self.ExceptionLogger.PurgeAfterDays := 1;
self.ExceptionLogger.Enabled := True;

{cache path }
CacheDir := Self.ContentPath + 'Cache\';

Like I said, I want user to use only one session, been an application mode.

Thank you again for your help and support!
Reply
#8
To stress test IW applications I basically use 2 tools: (a) Mostly Apache JMeter which is somehow difficult to configure correctly, but once it's done is simple to change/adjust, and (b) an inhouse tool created with Indy that hammers the application with some specific requests. Unfortunately it's not possible to make this tool public at this stage.

You mentioned that "what remains is just a write in exception log method, accessed thru critical section". If you have the IWExceptionLogger active, why do you need another exception log?
Have in mind that adding critical sections to a multi-threaded application is a double-edged sword: you protect it from concurrent access but you also increase the chances of an undesirable dead-lock...

Regarding your ServerController settings, everything looks good to me, except the

CookieOptions.SessionCookies := false;

unless you want the session cookies to persist when the browser is closed. In general it's a better option to set it to True, so whenever the browser is closed (not only the browser tab) the user will be forced to authenticate again.
Reply
#9
Thank you for your remarks.

1.I will try to remove my exception logger (is write in windows logs when an user try to connect, multiple attempts etc).
So, is just when try to login.

2.I will activate CookieOptions.SessionCookies

3. About the number of sessions, i change the session timeout to 30 min and now i do not encounter any problem, but maybe is cumulative, because i remove any parameter set for user sessions in server config and also because the timeout is shorter, also concurrent users goes below 300.
Only one bottleneck remain: session read directly from config file, even still in a critical section on IWserver. I will try to remove that one too.

4. I will try to make a stress test tool myself that pass over login and see if my implementation have any problem.

Thanks again!
Dan
Reply


Forum Jump:


Users browsing this thread: 2 Guest(s)