Archive
Monthly
Go
|
|
DNN Blog
Nov
29
Posted by:
Shaun Walker
11/29/2006
As many of you know, the next major release of DotNetNuke ( 4.4.0 ) is focused entirely on performance and scalability improvements. However, in order to accurately identify the areas of DotNetNuke which needed to be tuned or optimized, it was first necessary to have a solid understanding of how IIS and ASP.NET operate in Windows hosting environment. In our research we found surprisingly few resources on this subject; especially resources which presented the information in a clear and concise manner. Therefore we thought it would be useful to improve the knowledge base in this area and share our findings with the community.
First, let's cover some basic terminology:
AppDomain - an Application Domain ( ASP.NET ). A container for code and data which provides isolation for an application. This generally refers to a web site or web application. An AppDomain usually has a /bin folder where its local assemblies are located. When an AppDomain is created, all assemblies in the /bin are loaded and there is no way to granularly remove an assembly once it is loaded ( even if it is not being used by the AppDomain ).
AppPool - an Application Pool ( Windows 2003 / IIS 6.0 ). A fault tolerant mechanism for managing a group of AppDomains. Each AppPool has its own worker process ( w3wp.exe ). Once an AppDomain is loaded into an AppPool, there is no way to granularly remove it ( regardless of whether it is active or not ). An AppPool has a variety of configuration options for recycling, performance, and health which allow you to optimize your server resources.
The first thing to understand is that memory is a seriously gating resource in a Windows hosting environment. Lets say you have a server with 2GB of available memory ( this is the maximum which Windows 2003 will allocate to all user processes by default - regardless of the actual RAM capacity of your server ). And lets say you are using 5 AppPools. Guidance from Microsoft indicates you should set your memory recycling threshold at 80% of available memory ( or 800 MB – whichever is less ). So the formula in this hypothetical case would be 2GB / 5 * 80% = 320 MB. You would enter this value in the Memory recycling area of the Recycling tab for the AppPool in IIS 6.0. So now lets say that an ASP.NET application has a 10 MB working set ( I say “average” because as you load more AppDomains into an AppPool, the size of the working set decreases and begins to average out ). So this would allow you to run 32 sites in the AppPool ( 320 / 10 = 32 ) before a recycle will occur. Now these 32 sites do not necessarily represent your most active sites – they are any random site which gets a request which is being managed by the AppPool. So if we look at server density and assume that we want to host 2000 sites on a box for business/economic reasons we can calculate that we need to allocate 400 sites to each App Pool. However, based on memory constraints we already know that only 32 of these 400 sites can be active at any time ( 32 / 400 = 8% ). As soon as we exceed the memory threshold for active sites, IIS recycles the AppPool - unloading all of the App Domains. Then on the next request to the AppPool, Windows needs to first create the AppPool ( which imposes some overhead ) and then create the AppDomain ( more overhead ). So invariably there will be a significant delay on these “first requests” which can not be avoided.
So what if we modify the memory recycling threshold? Well if you allow the amount of RAM to exceed the 80% recommendation, Windows will leverage virtual memory and start to do a lot of disk paging. This is an expensive operation which will seriously affect the performance of all App Pools on the server ( it stresses the box ). So although the AppPools may not recycle as frequently, you may find that your CPU utilization or Disk I/O will spike, resulting in poor server performance.
The other scenario which triggers an AppPool recycle is when the worker process for an AppPool is idle for 20 minutes. When this occurs, similar to the memory recycling threshold, the next request needs to create the AppPool as well as the AppDomain, causing a significant impact on response time.
A common technique promoted by ASP.NET developers to keep a site "alive" is sometimes referred to as a "Keep-Alive". Essentially this is a service ( internal or external ) which constantly makes requests to an AppDomain so that it never gets recycled by an idle worker process. The effect of a "Keep-Alive" on a single AppDomain in an AppPool may not be significant. But if this technique gets overused within an AppPool, it will cause the memory recycling threshold to be regularly exceeded, causing constant recycling.
With so many factors to consider, it is imperative that an ASP.NET application strives to achieve the proper balance of response time and working set ( memory allocation ). DotNetNuke 4.4.0 will definitely deliver some significant improvements in this area.
16 comment(s) so far...
Re: Windows Hosting Performance & Scalability
Great article, and can you explain a few things a bit more? I am researching the best hardware and Windows/IIS config for a webserver hosting multiple DNN sites.
1. You said "2GB of available memory ... is the maximum which Windows 2003 will allocate to all user processes by default". My webserver will be running IIS 6.0 and standard 32 bit Windows 2003 only. Does this mean that it is pointless installing more than 3GB of memory in the server (1GB being ample for Windows and 2GB for user processes)? Or can the default be exceeded?
2. "lets say that an an ASP.NET application has a 10 MB working set" - what do you mean by "working set"? Do you mean the average amount of memory it needs? Also, when you say "So this would allow you to run 32 sites in the AppPool..." do you mean sites in the sense of an ASP.NET application (i.e. an install of DNN), or do you mean in the sense of a DNN portal?
3. I am curious how DNN 4.4.0 will deliver performance enhancements. It seems to me that everything you talked about (app pools, worker processes, paging, recycling etc) is external and inaccessible to DNN. Surely the only thing you can do is to reduce the memory footprint of DNN so there is less competition for this precious resource!
Looking forward to the release. Thanks for your extraordinary committment to the community!
By lneville on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
Thanks. Are you sure that the memory contraints are also valid for x64?
By mariette on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
In order to prepare for the next release, could you tell us if this is a 4.x only release?
Thanks
By pspeth on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Shawn,
I am only a simple user of DNN, completely ignorant of all the technical background. Thanks to this post, even I understand and finally get the issues with DNN's performance. Thanks for the work of the entire DNN team and the other members of the community.
By Carstone on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
Are the scenarios above aimed at the 2.0 or 1.1 version of the framework? Alot rings true (staying below a 1GB limit on memory allocation for instance) for load testing and optimization I have done in the past against 1.1, but I have not had the chance to look at the 2.0 framework's limitations in appDomains, etc. I was curious if 2.0 appDomains could efficiently go above the 800MB limit without incurring the IO hit.
When you look at hosting 2000 sites on a given box I assume we are targeting a shared hosting environment? I luckily have not had to face that level of coexistence on my production boxes. If you are outside of a shared environment and have full access to the box you have access to the machine.config and can for instance toggle off the setting to shutt down the worker process after an idle period which will prevent paying the cost on "1st hit" without the need of a keep alive. Also thread balancing (optimum affinity for procs, tuning thread pool sizes, etc) had been a factor in tuning in the past as well - again via the machine.config.
Has there been any thought for MARS implementation for testing? I am not sure how that would jive with some of the core code I have seen though in the latest pull which if I remember correctly was checking datareader states before using them - would that inihibit the usefullness of MARS? Also are default connection pool sizes being used?
By JohnHutchison2 on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
If the statement below is true
"Lets say you have a server with 2GB of available memory ( this is the maximum which Windows 2003 will allocate to all user processes by default - regardless of the actual RAM capacity of your server ). "
Then what benefit do you get by having more memory in your server, or am I misunderstanding the example. If you have a web server with 4GB or RAM would the recycle threshhold be 640MB (4GB / 5 * 80% = 640 MB.) or would you still be limited to the 320MB?
Just wondering because MS answer to any server slowness is add memory. :-)
By smehaffie on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Just to give you a target to work for, my hosting provider limits my AppPool to 200MB: 100MB physical + 100MB virtual. As developers, you should set that as a limit in IIS (just like you should be developing using medium trust) and see how the application works..
In my situation, I get constant restarts.
By john.henley@danalytics.com on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Here is a thought, I have several DNN portals running, some of them hosted on dedicated servers and some other on shared servers. More than half of them are really simple in terms of content, they only need html/text modules and can be cached and served really easily (if the AppPool is active of course).
Is it possible to create an AppDomain with only the needed functionallity to serve cached pages? Which considers only the mandatory core libraries and reducing the memory needed
Maybe It's crazy but it looks like a good idea. ;P
By avvweb on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Shaun,
Thank you - you are so right about this info being hard to find. As we receive solid recommendations from the core team, that will allow us as hosters and developers to have a baseline against which we can test and bring the overall profesionalism of the project to a new level.
Great Work!
By JimSteele on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
Do they AppPools actual divide memory are share as needed?
Do you know how bad of hit the virtual memory handling is on a system? That is, if it is spawning out to virtual memory, any idea of how often before it seriously degrades the system?
Good post, very informative!
By RockyMoore on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Thank you Shaun.
Although I was familiar with some of these concepts your post covers it al. I'm anxious to see what kind of techniques you will be using to increase performance.
By messmail on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
Very interesting stuff; a topic that even most "advanced asp.net" books don't cover very well. Thanks for taking the time to put it into a performance perspective.
By mattchristenson on
1/11/2007
|
Re: Windows Hosting Performance & Scalability
Any possibility to measure the memory taken by my application? I gather that I can do it somehow using profilers, but can I do it on my hosted site?
I'm having a weird problem -- my app shuts down every half a minute. From your post it looks like it might shut cause of it exceeds available memory. Any chance to check this?
Thanks
ulu
By ulu on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Hi Shaun, I'd like to know how i can implement automated "Keep-Alive" requests for my (still very low-traffic) websites based on DNN 4.3.5 - since they are hosted. Is it just a page with the meta REFRESH tag that calls a page in the Web Application that I want to not refreshed, or are there better methods?
By psiegers on
9/28/2007
|
Re: Windows Hosting Performance & Scalability
Thanks for the impressive article.
By NetRockerz on
10/15/2007
|
Re: Windows Hosting Performance & Scalability
That is configurable under the Application pool worker process
Under the Performance TAB , you will can uncheck the idle timeout ( default is 20 minutes)
Normally hosting companies , will keep that on , because it consumes a minimum memory space of at least 70 – 80 megs per process .
If you keep that option OFF , your DNN site will normally load faster .
Thanks
By George Gergues on
4/26/2009
|
|