Archive
Monthly
Go
|
|
DNN Blog
Jun
21
Posted by:
Joe Brinkman
6/21/2010 7:13 PM
 This is one of those posts that is always difficult for me to write. Nobody likes to admit failure, but the only way to fix a problem is to admit when it exists. Once again we have had to remove a release due to a severe bug making it through the release process. We probably could have just left the 5.4.3 release as it was and worked on rushing out a new 5.4.4 release to address the bug, however, we felt it would be prudent not to put our users in the position of installing and trying to use a version that we know is going to break a lot of modules from 3rd party developers.
The bug in question is a breaking change that will break any module that uses the core ExecuteSQL data access layer function. It is hard to gauge just how many modules use this function, but we know that this function impacted modules by at least 4 core team developers, including many administrative modules that we use to run DotNetNuke.com. The fix for this bug is relatively simple and we have already checked in a fix and are working to test it. We will have an updated package ready for testing tomorrow by a select group of commercial module developers with a final release of 5.4.4 coming later this week. Given the security issues which were already made public we are limiting the scope of this release to ensure we don’t leave the community overexposed.
This issue highlights one of the challenges of rapid growth in any software development organization. How do you bring new developers up to speed and get them productive as quickly as possible? How do you continue to ensure that your software continues to meet the quality standards that you and your customers expect, even as your engineering team expands and your processes are undergoing change?
So how do we plan to address this challenge? The solution is really one that must be addressed at every level of engineering process – from the developer who first touches the code, to the build manager who is responsible for building the packages, to the QA team who tests the code, to the community beta testers who are performing ad-hoc testing, to the release manager who signs off on the final release. At each phase there are people involved who have some responsibility to ensure we are putting out a product that we can all be proud of. We are currently working with people from each of these levels to improve our processes and our tooling to do every thing we can to eliminate these problems.
Our engineering team is working hard to add more and more unit testing to help catch these types of issues earlier. Part of our challenge here is to make needed architectural changes that will allow us to unit test more and more of the framework. We also recognize that there is quite a bit of cultural and historical knowledge about the platform that we need to pass-on to new members of the engineering team. We had already planned an engineering retreat this summer for the entire Engineering and Product teams and a focus on engineering quality will certainly be a big topic covered that week.
For the last 9 months we have been building out and enhancing our automated build processes. With each release we have improved the process to automate more and more tasks. We know that there are many processes that still remain to be automated. Based on the results of the 5.4.3 release we will be stepping up our efforts to incorporate automated API testing to ensure we are not breaking binary compatibility between releases. This had been on our list of things to do but will now be moved to the top of the pile so we can get this in place in time for the upcoming 5.5 release.
Our internal QA team has been working hard to keep up with the pace of our releases. Over the last several months they have produced and executed well over two thousand feature and regression test cases. Given the current size of our team, we must prioritize the number of tests that are run with each release. We continue to expand this team and will be re-evaluating key testing steps to ensure that we are covering a wide enough set of baseline tests with each release. Based on some of the release issues this spring we will be adding in test cases that exercise installation and upgrade of key 3rd party modules.
One area that we still struggle with is how to best engage the community to make them an integral part of our release process. We have tried closed betas, open betas, open repository and even special advisory committees all in an effort to get code and packaged releases out to the community in an effort to get feedback prior to a release. Nothing we have tried has resulted in adequate feedback. Last November we started posting source code up on CodePlex. By mid-February we had automated this process so that code was made available almost every night as it was checked into our own repository. While we continue to get a substantial number of downloads we do not seem to be getting much response on the code quality.
Likewise, following the release of 5.3 and 5.4 we formed a Module Developer Advisory committee made up of a group of commercial module developers. This group receives access to build packages around the same time that we delivered packages to our internal QA team. We will continue working with this team to find out how we can better identify these types of issues before a release.
Finally, the last person who touches each package before it is released is me. For the last 18 months or so we have been using a release checklist to ensure that we follow the same steps with each release. A large part of this checklist is focused on making sure that the engineering and marketing efforts are complete, that we have performed the necessary QA steps and that we have posted files in the proper locations and updated the web pages which need updating. Up to this point we have not had a product “owner” who would perform any sort of acceptance testing. With the recent formation of a formal product team we will begin incorporating this step into our release processes. Given the size of this team this will not be an all encompassing set of testing like that performed by QA but will include some basic sanity checks to ensure that every part of quality chain is doing their part to help ensure we put out a solid product.
We think that by taking a step back and re-evaluating how we perform every step of the engineering and release process that we can make significant improvements in the quality of our releases. You deserve better from us and we demand better from ourselves.
06/24/2010 UPDATE: During testing of 5.4.4 we found an additional issue that required an additional round of testing which pushed us outside of our release window for the week. We will be releasing 5.4.4 on Monday June 28.
13 comment(s) so far...
Re: DotNetNuke 5.4.3 Has Been Pulled
I can't say I'm happy... since I've updated to sites... because of some issues with 5.4.2 that 5.4.3 is supposed to fix!
But at least you have the courage to admit the failure! That's always good, you should indeed change the way you check this!
I hope 5.4.4 doesn't take much long!
By Miguel Clara on
6/21/2010 9:18 PM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
All I know is I'm glad I'm not Joe Brinkman.
How did you get stuck writing all these "We F'd up again" posts? This is like the fifth or sixth one in 10 months or something. It's every other release - what a job! I did notice that they are getting longer.
I vote that Shaun or Nik has to write the one for 5.4.5 - and Joe get's to write the announcement for the 5.4.6 fix release that comes out three days later. That way he get's to look like the hero once in awhile. Then you could have a lottery for who has to write it for the 5.4.7 release. Hey! I know - a custom DNN "dunk tank" type module where DNN community members get to pick who has to write the "We F'd up again" blog post for every other release!!
By Greg Brown on
6/22/2010 6:20 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
Look here's the deal. If you want to drop releases that break the major core functionality in a community release that is one thing - we expect the community release of any product to have some bugs for the community to find - that's the nature of the beast and - unless there's a security issue - it's why I generally have my clients hold off on updating to the latest release right away.
But once you start charging for your product the rules of the game change. Your for-pay releases need to be as stable and bulletproof as you can possibly make them - even if that means you release new functionality FIRST to the community and SECOND to the full paying customers. It seems backwards - but in fact it's a valuable modus operandi - it allows your community to find the serious bugs for you BEFORE you release to the folks who expect a more stable experience.
No one will test your product on multiple operating systems, with different combos of 3rd party products as well as your community customer base - no matter how much time, effort and resources you throw at the QA team.
Now comes the thorny subject of how to user test the release features that are NOT in the community edition - might I suggest that is what your 3rd party reseller community can do for you.
Just some thoughts...
By oscubed on
6/22/2010 6:22 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
I've been using DNN for many years and have never been in a situation like I am now. I upgraded our production server to 5.4.3 after several reviews indicated its stability and fixed some problems we were having.
We've been running on 5.4.3 for a couple of days with our clients making changes to their sites including some major work preparing to launch 4 related sites next week (i.e. teams of people working around the clock). I can't roll back our database and files and risk losing some of our largest clients (i.e. going out of business). At the same time, we are seeing some major issues with security in some of our third-party modules (i.e. sitemaps that are exposing admin pages to guests). How can I resolve this situation - and quickly?
Thanks, John
By John Tisdale on
6/22/2010 6:21 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
Hello Joe,
At 41 years of age, I have worked late once or twice a week since I was 18. I assume you do also. So first of all, for your late evenings and nights, I want to say thank you.
Secondly, in the last year, I have plenty of money from the CE version of DNN. You and your colleagues have contributed significantly to that income I have received, I also want to thank you for that.
I enjoy working on DNN, and it is currently giving me real joy in daily work, so I thank you for that also.
Finally, you had a bug in 5.4.3, I can live with that, my software also have plenty of bugs.
Thanks for all your late nights.
Give your self a break and be aware of all the good stuff you have achieved. 5.4.4 will come and so eventually will 6.0 and 7.0.
By Mark Breen on
6/22/2010 3:02 PM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
if I leave 5.4.3 in place will can I just apply the 5.4.4 when it becomes available..of do i need to remove 5.4.3 ?
By Brian Mandel on
6/22/2010 3:03 PM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
@Brian - You will be able to upgrade from 5.4.3 to 5.4.4 just as you can any other release.
By Joe Brinkman on
6/22/2010 3:04 PM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
Hi Guys, Myself and many others have been having difficulty since version 5.3.0 due to problems caused by the user profile. These are some threads on the forum addressing this issue and will this now be fixed in the new releases. www.dotnetnuke.com/Community/Forums/tabid/795/forumid/200/threadid/372408/scope/posts/Default.aspx
www.dotnetnuke.com/Community/Forums/tabid/795/forumid/200/threadid/365858/scope/posts/Default.aspx
The problem is this. after install the register links do not work they return to the home page. Going to admin/site settings and changing the register page to not specified resolves this in some versions but not all. Also when you get to the register form and fill it out on clicking submit the site errors. To resolve this the smtp email address has to be set to the same as the Host users email address and once again this only works on some versions. This all started after 5.3.0 was released and I now know of over 30 people with the same issue. I have 40 plus sites all with different versions of DNN and all experiencing different issues set around the user profile that was introduced in 5.3.0 All having different release used for there update. Any input into a fix or when a fix will be released would be much appreciated.
By Paul Manctech on
6/23/2010 5:31 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
@john, what security issues are you coming across? I don't believe any known security issues were added in 5.4.3, I personally am running DNN sites that have 5.4.3 without any issue at all (at least that I know of), I have no desire or need to rollback.
By Chris Hammond on
6/22/2010 9:15 PM
|
Community review / contributions
Hello Joe - thank you for the update. A comment on one point you raise:
"Nothing we have tried has resulted in adequate feedback. Last November we started posting source code up on CodePlex. By mid-February we had automated this process so that code was made available almost every night as it was checked into our own repository. While we continue to get a substantial number of downloads we do not seem to be getting much response on the code quality."
Firstly, the SVN code repo is NOT updated anything like nightly: e.g., the last update on there at the moment is 14th Jun (note no v5.43 tagged commit), so there's no way for the community to be reviewing your internal builds:
dotnetnuke.codeplex.com/sourcecontrol/list/changesets?ProjectName=dotnetnuke
Secondly, I'm surprised that you say there is little feedback! Gemini is chock full of bugs and issues that need resolving, and this is my first port of call if I come across a problem. Secondly, the "Reviews" section on each download page is normally full of comments for every release, normally RE missing files, upgrade problems, etc; e.g.:
dotnetnuke.codeplex.com/releases/view/45478
Perhaps the problem is that the DNN QA team aren't reading these comments?
Finally, you (and Philip Beadle) have mentioned encouraging patches from the community - partic those accessing the SVN repo. The public SVN is out of date, so we can't help with breaking issues such as v5.43. And I have personally submitted 3 different patches via CodePlex since May, but none have got past "Being evaluated" yet (Philip cited needing a detailed Gemini report with each patch, which just isn't practical).
This leaves me with the impression that DNN remains a classic "closed development" system (despite having open source code), rather than a modern Open Source project that harnesses the power of community patches. Of course, widening the patch acceptance criteria places even more importance on QA processes - community patchers tend to tackle single problems rather than being aware of wider issues (partly due to poor / non-existent documentation for most of the DNN code/API).
Cheers, Ben
By B Johnson on
6/23/2010 11:28 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
@Ben - 1. No tagged 5.4.3 commit because we pulled 5.4.3. We did tag the commit on the 14th as being 5.4.3. Because you can no longer see the 5.4.3 release, you also no longer see the 14th as being associated with the release.
2. CodePlex is kept fairly up to date. The only time it gets out of sync is when a process on the build server stops and we have to go restart it. Nightly updates don't occur because there are not code updates every day. There is certainly room to improve the sync process but that alone should not prevent community feedback on the code-checkins that are present. In fact, the error which ultimately resulted in pulling down 5.4.3 was in the checkin that was made on the 7th.
3. Saying that Gemini is full of bugs does not address the problem of getting specific feedback on the code that is about to be released. The vast majority of bugs in Gemini are ones that are UI bugs or that have workarounds. Those are not showstopping bugs.
4. Finally, on the topic of SVN patches, we try to review all the submitted bug fixes prior to each release. We are still improving in this area, but if you look at the last 2 releases you will see that we have had more than 30 bugs fixed by community contributions. And yes, if you submit code without a Gemini issue detailing the problem that the code fixes, it will not be accepted. We don't just willy nilly add code to the core because a community member submitted it. I will review your particular submissions with Phil to see what additional information is required in order to get them incorporated in the 5.5 release.
By Joe Brinkman on
6/23/2010 11:45 AM
|
Re: DotNetNuke 5.4.3 Has Been Pulled
As a DNN Pro customer, I like the idea of letting the community vet releases for a short time before they are made available to me.
I have waited since 5.2.3 for a new release to use. I knew a .3 stability release was coming so I assumed it would be safe to use.
For me, I think it is safe. So far none of my modules have any errors.
That being said, it is disconcerting to read the DNN Support team suggesting that I rollback. If there was an error in our system, rolling back would cost us serious money. (10 people working on our sites for 3 days, and upset customers.)
Thanks for your hard work. I know what it's like to be the guy who has to eat humble pie from time-to-time. This time, I am lucky that I avoided any crisis. But I don't want to depend on luck in the future.
By Lucas Jans on
6/24/2010 3:43 PM
|
Re: DotNetNuke 5.4.3 Has Been Pulled *UPDATED 6/24*
05.04.04 has indeed put an end to the issue. Thanx for the early release.
By Marc (ម៉ាក) Vanhemelryck (វ៉ាន់ហេមិលរីក) on
6/29/2010 8:53 AM
|
|