Archive
Monthly
Go
|
|
DNN Blog
Jun
22
Posted by:
Shaun Walker
6/22/2009
There is never a dull moment when it comes to DotNetNuke, and the past week certainly had more than its fair share of excitement. The combination of a new product release, infrastructure upgrades, and broad media exposure had a noticeable impact on the ecosystem - and resulted in some unexpected challenges. The most visible issues manifested themselves in the focal point of the DotNetNuke community - the dotnetnuke.com website. So if you count yourselves amongst the thousands who visited the website last week and experienced uncharacteristically slow response times, we sincerely apologize for the inconvenience. The good news is the issues have been resolved and the website is now performing better than ever. I would like to provide some insight into what happened.
As per our standard development practice, extensive testing of our product is conducted internally before the bits are ever made available for public consumption. Once we are satisfied with internal testing, we typically make Beta builds available to select individuals in the community who are interested in assisting us with validating the software. With the DNN 5.1 release, we went one step farther and opened up the Beta process to the entire community. We released 7 public Beta packages over the past month as we progressively moved the product closer to final release. While Beta testing was ongoing, we were also testing our own upgrade experience continuously by refreshing a staging environment with production data and content and running through the upgrade process.
Last weekend we decided the software was ready for production deployment - a term we call “dogfooding” (i.e.,eating our own dog food). We successfully upgraded dotnetnuke.com and everything seemed to behave fine on Saturday and Sunday, leading us to believe we were on track for a product release on June 16th. But on Monday morning the situation changed...
Before going any further, let me explain a little about our production environment. From humble beginnings as a bootstrapped venture, DotNetNuke Corporation has always been privileged to have MaximumASP as a project sponsor. MaximumASP has provided servers and bandwidth to the project for many years and we are deeply appreciative of their ongoing support. That being said, the servers we were using in our production environment (i.e., 32 bit, dual core CPU, 2 GB RAM ) had not kept pace with the growth of the DotNetNuke project. We recognized that at some point we were going to need to update them. We had already experienced some performance issues in the past but had always managed to work through them with a combination of configuration and software optimizations. But eventually you hit a ceiling.
On Monday and Tuesday we began to notice some serious performance issues with the dotnetnuke.com website. The CPU usage on the 2 web servers spiked above 90% and did not come down. Page loads were taking upwards of 4 to 5 seconds for unauthenticated users and authenticated usage was almost impossible. Symptoms pointed to a bottleneck somewhere and further diagnosis seemed to point to a RAM deficiency resulting in excessive page file thrash. But what happened on Monday and Tuesday to trigger this behavior?
Recently we had forged a business relationship with a professional public relations firm, Eastwick Communications. The plan was to conduct a broad media outreach program to help raise awareness of DotNetNuke in a variety of traditional business channels. Over the past couple weeks we conducted interviews with a variety of notable media organizations covering our product space, including CMSWatch, CMSWire, Redmond Developer News, Webhost Industry Review, Windows IT Pro, etc... We also met with analysts from both Gartner and Forrester to ensure growth of the DotNetNuke ecosystem was not going unnoticed. All of the extra media attention resulted in a spike in visitors to our website and our website unfortunately could not keep up with the demand.
To deal with the performance problem, we took a two pronged approach. While the Engineering team analyzed the software for performance bottlenecks in the application logic and database layer, we also reached out to MaximumASP to get new modern web servers. Scott Willhite was instrumental in getting the new servers (64 bit, quad core, 8 GB RAM) configured in record time (with assistance from Microsoft Professional Support and MaximumASP). After deploying a DotNetNuke 5.1 package with some performance optimizations Thursday night, the health of the site was back under control and the system was serving pages faster than ever before.
So what about the 5.1 release? With the performance issues resolved, we are now gearing up for the final release of both the 5.1 Professional and Community Editions. With a successful dogfooding exercise under our belt, I am confident the 5.1 platform is ready for primetime deployments. The launch will likely occur early this week.
Going forward, there are some lessons learned which we will incorporate into our process. For the early detection and resolution of performance issues, we are going to invest in a serious load testing environment where we can push the software to its limits in a controlled, methodical manner. This should mitigate some performance problems during the development process; however, it will not completely emulate the demands of a real production environment. So we will continue to eat our own dogfood in the future, as we feel it is a great benefit to our users and customers to know that we utilize the latest version of our own platform. Our ongoing commitment to the community is to ensure we provide great value in terms of a world class, enterprise-class software platform and project infrastructure.
We would like to thank everyone in the DotNetNuke community for their patience last week, as we worked through issues with the website and validating the software for production use. I must also thank the members of the DotNetNuke Corporation Engineering team (i.e., Charles, Joe, Scott) for pulling a couple of all-nighters as they worked hard to diagnose and correct the problems. And I also want to thank MaximumASP for all their support and for helping us get the new web servers added to our environment so quickly and professionally.
I’m looking forward to a highly successful launch of DotNetNuke 5.1!
11 comment(s) so far...
Re: A Perfect Storm...
thumbs up for the engineering team and maximumasp!
By tin lam on
6/23/2009
|
Re: A Perfect Storm...
Thanks for the update. However, I'm a little concerned that the new 5.1 build caused you guys to actually throw more hardware at the problem.... Does this mean that 5.1 will be more resource-intensive than 5.0.1 or 4.9.4?
By southwo8 southwo8 on
6/23/2009
|
Re: A Perfect Storm...
Actually our production environment was severely underpowered by today's standards, so we needed to perform a hardware upgrade at some point anyways. My blog post was not meant to imply that you will need higher powered servers to run 5.1. Ther servers we procured will allow us to better meet the needs of the community going forward.
By Shaun Walker on
6/23/2009
|
Re: A Perfect Storm...
My sympathies to you for pumping the media wagon only to have the flood of new (VIP) visitors be the straw that broke the camel's back. Congrats on being able to recover so quickly!
By Juan de Vashon Isle on
7/5/2010
|
Re: A Perfect Storm...
Thanks for the update, Shaun!
Is there any chance to see some numbers about the load DotNetNuke.com was carrying when the problems started occurring, compared to the load before the problems? It might be of interest to others in similar load conditions.
Also, it would be great to hear about the load testing software that will be used in the future releases.
By Daniel Gilleland on
7/5/2010
|
Re: A Perfect Storm...
Actually, it is very impressive that DotNetNuke.com ran on a 32-bit/2GB server as well as it did (although it did hiccup sometimes). I'd bet most people would be surprised that DotNetNuke,com was running on so limited of servers with the amount of traffic it gets. Congrats on getting the servers upgraded, and hats off to all those involved in getting DotNetNuke.com back up and running again. Pulling all-nights to get things like this fixed is no fun (we all can probably relate to what those involved were going through).
By Shawn Mehaffie on
7/5/2010
|
Re: A Perfect Storm...
Is there any plan to publish the volume numbers vs. farm configurations that brought the servers down? Since we haven't been able to find any published performance and scalabiltiy metrics for DNN, it would be great to be able to include these numbers in plannig the needs for our server farm.
By Mike Mooney on
7/5/2010
|
Re: A Perfect Storm...
I hope I am not too late for you to respond on this one.
I personally think that now that DNN is getting matured and starting to get adopted not only by webmasters with a niche target audience but also by webmasters targeting a larger audiences, resources need to be invested to beef up performance. Time and effort has largely been invested in making DNN cool (skins, modules, upgrades, etc) sometimes overlooking performance.
I generally don’t have any issues with page load throughput (rendering of pages) on my portal as I do not experience overwhelming site visits since the portal is generally new. However, the target audience is international which implies that with time the portal will experience the same issue. For now, I am happy to declare that page load throughput is generally optimised in DNN. System engineers just need to be wary about the size of servers as the number of site hits increase.
Apart from page load throughput, DNN is also a portal generator, which implies, in lack of a better expression, there should be what we call ‘portal generator throughput’ in DNN. DNN is not optimised for ‘portal generator throughput’ based on these facts:
My portal is designed to provide a first time visitor with a free DNN portal in administration mode (www.iflaker.com). The visitor can tweak the portal the way an administrator of any DNN portal would, without the need to sign up first. The portal uses a cookie to save changes made by a user so that when they visit the site again in the future all their tweaks are intact and still without the need to sign up. When the user signs up, the portal provides them with a friendly URL (iflaker.com/chosenname) that they can share with anyone and tweak from any PC using credentials supplied on sign up. The initiative was inspired by initiatives such as iGoogle, Pageflakes, etc. The difference is iFlaker is a web2.0 portal generator while iGoogle is a web2.0 startup page. Now, since a new portal is generated on every unique visit, iFlaker currently has 2,300 portals. The portal is hosted on a shared environment in Dallas USA. I am based in South Africa and it takes me just over a minute to generate a portal for a unique visitor on a 4MB line. Distance might play a crucial role but over a minute is still unacceptable. The workflow for generating a portal is more involved than the workflow for rendering a page and requires hitting the database more often. This plays a critical role is making a new portal rendering to be too slow. Methods that can improve portal rendering include writing a windows service that generates portals so that when a new user visits, a portal is not generated from scratch but fetched from the database or a cache. The same applies when a user signs up on iFlaker since the workflow for achieving a user to signup and retain their portal tweaks is quite involved. In fact, now that the portals are over 1,000, iFlaker just times out when a user signs up.
I also have to always clear table eventlog on a regular basis otherwise DNN throws ‘Server cannot modify cookies after HTTP headers have been sent.’ Now it happens even when the records are just a couple of hundreds in the eventlog table. I am afraid it might get to a point where clearing the table does not help anymore.
I think a forum dedicated to performance with dedicated and well versed members need to be established where all aspects of DNN performance will be tackled.
Gabriel Nkuna
FOUNDER iFlaker
By Gabriel on
7/5/2010
|
Re: A Perfect Storm...
Hmm Hats of engeeneering Team :-)
Regards
Admin
www.r4card.com.au/
By Nintendods on
7/5/2010
|
Re: A Perfect Storm...
Thank you guys, keep up the great work!!!
By James Campbell on
7/5/2010
|
Re: A Perfect Storm...
Shaun,
I have been using the site intensively for the past two days and have to say its performance is very uneven. Page loads in excess of 15 secs on a significant percentage my requests. This appears to be across all modules.
Are the perfomance issues you speak about in this blog resolved? Do I need to look at my own system? I use firefox and have not noticed issues with other sites.
Thanks,
Matthew
By Matthew on
9/5/2010
|
|