Peter Donker

Analyzing an exploding EventLog

peter@bring2mind.net — Sat, 26 Nov 2011 09:56:11 GMT

Murphy’s law has it that real issues in DotNetNuke (core. modules, skins) don’t manifest themselves until you’ve rolled out in production. We can test all we want, but the fact of the matter is that this has become a highly complex eco-system with many interacting bits and it is next to impossible to test for all scenarios. And I’m pretty sure the kind folks at the Corp’s helpdesk will agree with me on this as well. So let’s not kid ourselves: sh*t happens. It’s our ability to deal with it that separates the good from the bad (or experienced from the inexperienced), IMHO. It is also the reason we have something called dogfooding. This is the last phase of testing where you use your own product (dogfood) in your own production environment. This is after you’ve tested it on a staging server of course. It’s rare, but sometimes it does give rise to issues. After I rolled out DNN 6.1.1 on my own site and had began dogfooding my own DMX 6 my server ground to a halt. Lucky for me I did this over thanksgiving so it was fairly quiet on the interwebs. The first major stumbling block in analyzing what was happening was the fact that the eventlog had exploded. I mean, my 250 Mb odd SQL database had become 4 Gb and that was just one table: the EventLog table.

First aid

As the site ground to a halt I needed to do 2 things: get the site running again and analyze what was happening. But at this point the system is more or less frozen with SQL clipping at 99% CPU. OK, first I brought down the site to get the server back down to earth. Then I proceeded to remove all but the last 100000 records from the EventLog. I could possibly have deleted more, but this brought the DB to a reasonable size (something like 1 Gb). Backup the DB, zip this and bring down to my dev environment for analysis. Final change to the DB: truncate the EventLog table and shrink the DB again. Now to get the site working I got it back online and disabled the GENERAL_EXCEPTION in the EventLog (go to the settings for the eventlog and edit this row). Now the SQL is no longer working overtime. At this point we have to realize the error is still happening but the fact that it is no longer logged prevents SQL server going wild.

Analysis

For those of you who’ve not been here before, the Eventlog table has one major drawback: the actual details of the errors are in an NTEXT field as an XML blob. This prevents us from doing any SQL analysis on the data. For one I’d like to get an idea of which error is prevalent. This requires comparing the details between errors. We need to get to the contents of the XML. Given that it’s in NTEXT, SQL server will refuse to operate on it as it is stored as blobs and can’t be compared. Converting it to NVARCHAR(MAX) is impossible on a serious volume of data (I tried and the process just blocked). I also tried exporting to XML to open up in XML Spy for instance, but this lacks the analysis capabilities that I need. It was useful to run through the first few records to get a first idea, though. You can export to XML using a utility called “bcp”, or bulk copy. This is a tool that comes with SQL server, so you run it from a cmd window:

"C:\Program Files\Microsoft SQL Server\100\Tools\Binn\bcp" "select top 100 LogProperties from new_bring2mind_net.dbo.PortalCore_EventLog" queryout D:\Websites\local.bring2mind.net\eventlog.xml -U "test" -P "test" -c

Here “new_bring2mind_net” is the name of the database, “test” and “test” are username and password. Now all you need to do is to open the resulting file and stick a and around the contents to make it a valid XML file. Open in XML Spy (or similar tool) and you can walk through the log. But you’ll quickly notice that the actual details of the event (in the “message” property) are so large, they make it difficult to “see” what is happening in grid view.

So how to actually use the eventlog for analysis? Well, we need to get the XML data to SQL so we can run queries. And this SQL 2005+ can do. The approach I have now pursued is to (1) create a table for the log properties, (2) run a cursor over the EventLog and (3) read the LogProperties as XML and push that into the new table. So let’s look at some code. First we create the new table:

CREATE TABLE PortalCore_EventLogProperties (
 LogEventID INT NOT NULL,
 PropertyName VARCHAR(50) NOT NULL,
 PropertyValue NVARCHAR(MAX)
)
GO

This sets up the table that will receive our data. Now for our bit of magic:

DECLARE @LogProperties XML
DECLARE @LogEventID INT

DECLARE c CURSOR FAST_FORWARD FOR
SELECT LogProperties, LogEventID FROM PortalCore_EventLog

OPEN c
FETCH NEXT FROM c
INTO @LogProperties, @LogEventID

WHILE @@FETCH_STATUS = 0
BEGIN
 PRINT @LogEventID

 INSERT INTO PortalCore_EventLogProperties(LogEventID, PropertyName, PropertyValue)
 SELECT
  @LogEventID,
  Tbl.Col.value('(PropertyName)[1]', 'VARCHAR(50)'),
  Tbl.Col.value('(PropertyValue)[1]', 'NVARCHAR(MAX)')
 FROM
  @LogProperties.nodes('//LogProperty') Tbl(Col)
 WHERE NOT EXISTS (
  SELECT * FROM PortalCore_EventLogProperties 
  WHERE LogEventID=@LogEventID
   AND PropertyName=Tbl.Col.value('(PropertyName)[1]', 'VARCHAR(50)')
 )

 FETCH NEXT FROM c
 INTO @LogProperties, @LogEventID
END

CLOSE c
DEALLOCATE c
GO

This migrates the logproperties to its own table. Note that this can take a while depending on the number of records (I ran 30K records in about 45 minutes while typing this). Optionally you can limit the amount by adding “TOP 1000” or another limiting instruction to the SELECT statement. But now we have the data where we want it. Ready for analysis. Let’s run a few queries.

SELECT * FROM
(SELECT DISTINCT(elp.PropertyValue),
 (SELECT COUNT(*) FROM PortalCore_EventLogProperties elp1
  WHERE elp1.PropertyName='RawURL' AND elp1.PropertyValue=elp.PropertyValue) AS cnt
 FROM PortalCore_EventLogProperties elp
 WHERE elp.PropertyName='RawURL') tmp
ORDER BY cnt DESC

This will show you a list of requested URLs with the number of occurrences in descending order. It may give you a hint where (i.e. which page) is giving you a headache. Now replace the value “RawURL” with “Message” and you’ll get the same but for the actual errors. The message is in effect the stacktrace. So this is the best way forward in determining where an error is occurring. You can of course also export this to a text file so you can work through it.

"C:\Program Files\Microsoft SQL Server\100\Tools\Binn\bcp" "SELECT cnt, PropertyValue FROM (SELECT DISTINCT(elp.PropertyValue), (SELECT COUNT(*) FROM new_bring2mind_net.dbo.PortalCore_EventLogProperties elp1 WHERE elp1.PropertyName='Message' AND elp1.PropertyValue=elp.PropertyValue) AS cnt FROM new_bring2mind_net.dbo.PortalCore_EventLogProperties elp WHERE elp.PropertyName='Message') tmp ORDER BY cnt DESC" queryout D:\Websites\local.bring2mind.net\StackTraces.txt -U "test" -P "test" –c

Good luck debugging!

More ...

Category: Development

“DotNetNuke 6 Launch Party” 26 Oct in Zurich

peter@bring2mind.net — Fri, 28 Oct 2011 21:55:57 GMT

Last Wednesday we had a Swiss DotNetNuke Usergroup get-together in Zurich around DNN 6. It’s not that we are somewhat slow in Switzerland (although evil tongues would have it that we are) that the DNN 6 Launch Party only took place now. It is more due to the fact that this was the earliest possible opportunity and we really wanted to get DNN 6 across the footlight to folks here. “Hey, remember that DotNetNuke thing we had a UG meeting about a couple of years ago? Well it’s new and improved!”

The format was an “after-work” event. Starting at 4:15 PM and ending at around 8:30 it allowed people to just about finish their business day and head on down to the Microsoft offices in Wallisellen on the outskirts of town. The Swiss DNN User Group is led by Daniel Mettler from 2Sic (http://www.2sic.com/). Together with the dev contacts at Microsoft, he put this meeting together. Erik van Ballegoij was kind enough to attend and represented “The Corp” whereas I represented “the core team”.

Content

With so little time and given the approach of a fairly concise presentation, the sessions were kept to 20 minutes each. This means: get your message across and don’t get lost in details or overly long demos. No time for in-depth technical sessions. In my opinion this format worked quite well. The audience got a heads-up on a number of aspects and the socializing time we had afterwards provided ample time to follow-up in case there were questions.

Daniel hosted the day and did most of the (German) talking. He was kind enough to do this in “proper” German (in contrast to Swiss German) so others like myself could at least get a grasp of what he was talking about. He had done some exciting things with DNN 6 and shared it with us. This included connecting to SharePoint and Mobile support which of course also are part of the professional offering of DotNetNuke. Erik presented sessions on DNN 6 PE and DNN 6.1. I, myself, spoke on localization and the efforts of the DNN Internationalization team. Other speakers included Benjamin Gemperle and Raphael Mueller (both from 2Sic like Daniel).

Venue and Attendance

We had about 30 people show up for this event. Although these are not shocking numbers, we have to keep in mind that we have never attracted huge audiences here. Zurich is not a cheap place and for in-depth technical content the Swiss tend to look to the neighboring User Groups: the Swiss German to the German User Group and the Swiss French to the French User Groups. Both are substantially larger and offer more in terms of activity and forums.

Microsoft was kind enough to use as guinea pigs for their all new presentation/recording room in their office. I’ve been they only finished installing this the day before and we were the first ones to use it. When video does come out of this event, I’ll add a link to this post.

Thanks

I just want to round off with a word of thanks to 2Sic and Daniel Mettler in particular for having put this together. Thanks also to Microsoft and Ken Cassada in particular, for their support for the event. The munchies were very nice Finally thanks to DNN Corp for putting Erik on a plane to Switzerland.

More ...

Category: User Groups

(Http) Modules

peter@bring2mind.net — Fri, 12 Aug 2011 13:15:00 GMT

To our second installment on this two part series on handlers and modules in ASP.NET web applications, their significance and how to begin debugging situations where things have gone haywire. Like I mentioned in the previous post on handlers: modules and handlers go together like peas and carrots. The main difference is the following: whereas a handler runs only when ASP.NET has decided it should handle the incoming request and ASP.NET stops after it has found the handler, the module always runs on every request and ASP.NET continues. So the way that looks is that upon a web request ASP.NET first passes the request to each and every module in the list of modules and then tries to find a handler for it and passes the request on. Most crucially a module can alter a request as it passes through it or set the stage for other components before they run. For the remainder of this post I assume you will have seen/read through the previous post and you are familiar with Fiddler, web requests and responses and IIS 6 and 7 differences in the web.config.

DotNetNuke modules

Below is the modules section (IIS 7) of a default DNN site.

If you’re interested look up one of these in the source code and see what it does. Did you know that:

DNN’s Url rewrite module also initializes PortalSettings
DNN’s membership module takes care of user login, roles, authentication cookies, and user language

Basics of an http module

A module can flag to ASP.NET to be kicked into action at the beginning and/or the end of a request. ASP.NET hands control over to the module and it can look at the request, and change it (at begin request) or the response (typically at end request). A common example of the former is url rewriting. A Url rewiter changes the url at the beginning of the request. Url rewriters are used to make web urls more human friendly and get better results in search engines. Lets take the example of a request that asks for resource “/Blog/ThisWonderfullBlogpost.aspx”. By analyzing the resource string and looking things up in a database the url rewriter is now able to deduce that this is a request for page id 3 and blog post id 18. It then substitutes the found values and the request continues its journey in ASP.NET as “/Default.aspx?tabid=3&itemid=18” for instance. Another common example of logic in a module that fires at the beginning of a request is to add variables and their values to the application property bag. These can later be used by the application. An example of this is the variable PortalSettings which holds all the current DotNetNuke’s portal settings in an object. This way subsequent processing doesn’t need to recreate those all the time. Instead they are present throughout the request to both the DNN main application as well as the extensions.

An example of a module that latches on to the end of processing is a compressor. A compressor changes the response (output) of the server by running it through for instance a GZip compression component.

If you want to find out more about http modules check out the various web resources about these and/or examine the DotNetNuke source code for some great examples.

Debugging modules

Errors in modules lead to different symptoms than errors in handlers. A routing issue with handlers usually breaks stuff immediately. A module failure can hide itself better. But the approach in debugging is still similar to what was mentioned in the handlers post. We start by examining the web traffic for hidden error messages and routing issues. We also should not forget to look at the DNN event log to see if we spot exceptions that have come out of a module. It helps at this point to have a rough idea of what each module does.

Upload components

A special mention here about upload components. There are a number of upload components (Telerik, ComponentArt, NeatUpload, etc) that use Ajax callbacks to stream an upload to the server. What the component does is chunk the file that is being uploaded into small bits and upload one by one using Ajax and using the server’s responses to drive a progress bar/indicator. This has several advantages, notably user feedback and better memory management on the server. Typically these upload components use a module to get the stream of uploaded bytes. So when debugging upload issues, check out the modules in the web.config and make sure the necessary one is there and running (failed request tracing as described in the previous post can help you see running modules as well). Try removing non-essential ones to see if this helps. I have witnessed one upload component “eat” the bytes destined for the right upload component before it got to it which led to failed uploads as the upload component that had consumed the upload didn’t know what to do with it and the expecting upload component was left empty handed.

Finally

As mentioned before you need to use your knowledge of IIS (in these two posts), the application and Google to work your way through any issue with handlers and modules. It is a skill which you’ll develop and you’ll come to appreciate. Due to the nature of DotNetNuke extension programming, a developer cannot guarantee that all goes well. The configuration is not under our control and it is impossible to foresee what happens in the future. These two posts hope to help you cover some ground in this respect.

More ...

Category: Development

Handlers

peter@bring2mind.net — Fri, 12 Aug 2011 11:24:27 GMT

This post is the first of a two part series about handlers and modules and is an attempt to offload what I know about handlers to you. Knowing what handlers are and how they work is what separates the savvy from the not-so-savvy when it comes to IIS administration. Although very few modules in the DotNetNuke ecosystem use/rely on handlers, those that do experience a steady stream of support calls along the lines of “hey, XYZ doesn’t work” and it is down to a handler issue. Examples of modules that use handlers are Ultra Video Gallery and Document Exchange. DotNetNuke uses handlers itself as well. So what is a handler? In a nutshell a handler is piece of code that handles a web request. To understand why it’s good to know more about this you need to look closer at http traffic (i.e. the bits and bytes that go back and forth between your browser and the server).

Fiddler

Before we go any further I advise you to get Fiddler (http://www.fiddler2.com). Fiddler is an amazingly simple yet powerful tool for examining http traffic. What the program does is to temporarily create a proxy on your local PC. I.e. is nestles itself between your Internet Explorer (it works like that with IE and Chrome, for Firefox you need a plugin) and your network card. This way everything all traffic (well, only http traffic actually) between your browser and the internet goes through the program. It can then show you this traffic. This is a screenshot of its main UI:

I use Fiddler on an almost daily basis. It is the essential tool to have to debug handler issues. I also wrote a blog post on how to debug DMX WebDAV traffic specifically using Fiddler. The UI is split vertically in half. On the left you find the so-called sessions. A session is a pair of request and responses. On the right we can drill down into any one such session. My preferred way to examine traffic is using the “raw” view (shown above). So on the right I select the “Inspectors” tab which shows me details of the request and response and I select “raw” on both the top and bottom panel to show me exactly what went over the wire. I see both the headers and the content of the traffic.

Requests and responses

Whenever you type a url in your browser, it emits a request over the internet. You need to specify a server (or host as it’s called) and a resource. So the request

http://www.bring2mind.net/default.aspx

requests resource “/default.aspx” from host “www.bring2mind.net”. The host is looked up through the DNS and resolved to an IP address so the browser knows who to call. Then the server is contacted and the resource is requested. At this point I’m going to assume the server is running IIS. I’m sure Apache will do things similarly but here we’re just concerned with IIS. So the browser has contacted the server and requested the resource. It does so through a specific port (this is defaulted in http to port number 80) and with a particular verb. The verb specifies what the browser wants with the resource. In regular http traffic you’ll only see two verbs: GET and POST. GET is the most common and tells the server “gimme this resource”. POST is used when you click a button to submit a form and we won’t go any further into details about that. The most important thing to note at this point is that a full web request consists of a host, a port, a resource and a verb. In Fiddler you can see these details as you request a web page. At the top of the request panel you’ll see the “GET http … etc” ending with HTTP/1.1 which is telling the server the browser understands http protocol 1.1.

Any request can be accompanied by a number of headers. Headers are pairs of “Attribute: Value” pairs. So a common header like “User-Agent: Mozilla/5.0 (Windows NT 6.1 … etc” tells the server we are arriving with a Mozilla browser and running Windows 7. This header informs the server what browser and O/S we are using. It can also be used to determine if this is a search engine (e.g. Google) that is crawling our site. A very important header to note at this point is the Cookie header. The cookie header sends, as you’ve probably guessed, the cookies to the server.

The response from the server begins with a line stating the status of the response. Normally this will be “200 OK”. This means the request can be answered and the browser will get what it asked for. There are a whole bunch of status codes in http and if you’re interested you can read more about those here. Fiddler will color the sessions in the list on the left in part based on these response codes: the red ones are 4xx responses meaning there was an error. Like the request the response can also have any number of headers. These headers will be an important indication of what is running on the server and as we’ll see later, who is “picking up the call”.

Who is answering the phone?

Whenever the browser makes a request to a server, the server has to determine what needs to happen. In the “classic” web world, the web server just served out files that were on its hard disk. These files typically had extensions like .html, .js etc. A web server was no more than a file server. But this all changed when “web applications” came about. Here the response from the server was not some fixed file on the server but the result of some program running on the server. ASP.NET is but one of these mechanisms and IIS supports many of these. So when you request “/Default.aspx” the server must be told this is not just a file on the hard disk but that ASP.NET knows how to handle the request. This is where the word handler comes from.

So how does IIS determine what should happen? Well, it first assumes that the resource is actually there as a file and it should just serve it. It then looks at its own configuration if there is no exception to this. In IIS 6 this is slightly different than in IIS 7. In IIS 6 it is a two stage process where IIS first looks at the “Application Configuration”:

Here you can see aspx is mentioned. If you open this up you’ll find it points to c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll. This is the base ASP.NET component responsible for handling ASP.NET requests. If you open up the setting you’ll see this:

Here you see something else that is interesting: the handlers can be limited by verb. The above is telling the server that only for GET, HEAD, POST, and DEBUG verbs should IIS redirect the request to ASP.NET. You’ll find that using another verb the server will respond with an error as it tries to serve up the raw aspx file which is prevented internally from happening (security). Which brings me to something very important to note here:

If there is no handler for the requested resource it is assumed that it is a file on disk and IIS will look for it and attempt to serve it.

For instance, if you request a jpg file (which by default is not mentioned in the handlers) it will just find it on disk and serve it.

IIS 6 vs IIS 7

After IIS 6 has gone through its application configuration and determined the request is for asp.net the so-called ISAPI handler for ASP.NET takes over. This handler then goes to the root of the web application’s directory and looks for the web.config file. The main difference with IIS 7 is that in the latter there is no “Application Configuration” any more. Instead ALL handlers are in config files (which incidentally also makes IIS 7 settings more portable than IIS 6 settings). For both IIS versions there is something called a machine.config which determines the configuration of the “default” website and then a web.config which outline the specific configuration of a particular web site. So the above is an intermediate stage in determining handlers only used in IIS 6.

The significance of machine.config and web.config

So ASP.NET is going to handle the request. Now you may say this is the end of the line. ASP.NET will just “run the script” and we’re done. But no. ASP.NET can be tuned to send requests to different bits of code. How is this useful? Well, lets take the example of a thumbnail generator. You’ve created some code which takes an image (e.g. a jpg stored on your server’s hard disk) and resize this to a thumbnail and serve this to a browser. This is perfectly possible in ASP.NET. You could do this as an .aspx page but this is not really the “correct” way to do this. An .aspx page is expected to emit an HTML page after all. So what you do is to create a handler. And you tell the ASP.NET handler that if the request is of the kind “/thumbnail.axd?file=mypic.jpg”, then you would like handle the request because you know what to do. This is done in the web.config file which is an XML file. The web.config has section where it lists all the applications handlers which tell ASP.NET how to route the call.

More IIS 6 vs 7

To keep us on our toes, Microsoft has changed the way this is done from IIS 6 to 7. The IIS 6 handler section is under configuration/system.web/httpHandlers and the IIS 7 handler section is under configuration/system.webServer/handlers. This has not been too great for those of us working to support customers as we first need to determine which IIS version the customer is using before we can continue. Note that IIS 6 will happily ignore the IIS 7 handlers section and visa versa. For the remainder of this post we will assume we are working on IIS 7.

This is how the IIS 7 handler section looks for a default DNN 6 installation:

If you examine the first one you’ll see that the request for “Logoff.aspx” is redirected to type DotNetNuke.Services.Authentication.LogOffHandler, DotNetNuke. I.e. to class DotNetNuke.Services.Authentication.LogOffHandler in DotNetNuke.dll. If you open up the source version of DNN you can find this class and you’ll find it implements an HttpHandler! And also you’ll find there is no actual file Logoff.aspx in the distribution. Instead if anywhere you use Logoff.aspx you will be logged off.

The machine.config

The web.config is specific to a web site and is always found in the root of the application. But this is only part of the story. There is also a machine.config (which looks the same but is a whole lot longer) which is hidden deep in your system files. This is the “base” configuration. Most importantly it tells IIS the default behavior for a site and what a web.config is allowed to change in this. Normally we don’t need to touch this file. But you should be aware of its existence and significance. The web.config is meant to be editable by the site administrator. The machine.config is locked down and only accessible by the server’s administrator.

IIS 7 handler administration

If you open up IIS Manager in IIS 7 you’ll see that an application has the following management screen:

Under IIS you’ll see “Handler Mappings”. Click to open and you’ll see something like this:

What you’re looking at is a merged set of handlers from both the machine.config (i.e. the default) and the web.config! So even though the .aspx handler is not mentioned in the web.config you’ll see it here on the list. All changes made to this list go into the web.config. So you can remove a handler which shows up in the web.config as a “remove” node. This is a major advantage of IIS 7 over IIS 6 where this did not exist.

The can of worms

So let me give an example of how being unaware of handlers can get you into trouble. In one of my modules I naively thought I’d add a custom rss feed aspx. So I went ahead and added Rss.aspx to my module and began programming my logic. But whatever I did, whenever I requested /DesktopModules/Bring2mind/MyModule/Rss.aspx, I’d get the core rss feed. Now look at the handlers section of DNN in the picture shown earlier again. Aha. RSS.aspx is wired to DotNetNuke.Services.Syndication.RssHandler, DotNetNuke. That means that my code would never run as ASP.NET had redirect the request (and subsequently processing) to this handler. All I needed to do to resolve this issue is to rename the aspx and I was fine again. I just needed to avoid getting my call trapped.

But this illustrates a complexity particular to DotNetNuke extension programming. If you were to make your own web application, you’d be well aware and in full control of the handlers. But in DNN module development we don’t control the web.config. Sure, there are now mechanisms to make changes to the web.config when installing a module, but this still leaves us open to mishaps when DotNetNuke or some other module makes changes to the web.config. Not too long ago, when DotNetNuke added the Telerik components to the framework, a handler was added for the upload control for Telerik. Unfortunately for me, DMX used a similar handler for its own (also based on Telerik) upload component. The result was that the DMX upload broke on the DNN upgrade. There are no guilty parties in this. It is part of the game when you’re in this field.

Debugging handlers – some pointers

So something that uses a handler broke. Now what? My entry point is Fiddler and the web.config. I fire Fiddler up and do whatever it takes to reproduce the error in the browser and begin to examine the http traffic (tip: compare the traffic between two sites if you have one where things are working correctly and one that has an error). It’s often not hard to find the calls you’re looking for. In the case of an upload component complaining it can’t find the handler then check those calls and see what the server response is. Often we don’t see the error returned by IIS on screen as the calls are Ajax calls which remain hidden from view. But with Fiddler we can examine the response before it got to the browser and find more detail. Do you see an ASP.NET error? If so you’ll probably see a name of a component that produced the error. Is that component the component you expected to handle the request, or another. If it is an unexpected component then there is an issue in the routing most likely. Get the web.config and read through the (correct) handler section and see where it could have veered off track. Note the handlers have an order to them! I.e. the first handler that is a hit gets the call and subsequent ones are ignored. If the request was not handled by our designated component but we are stuck why this is so we can go to the next stage.

The second stage is more intrusive and is for IIS 7. In the advanced settings of your web application you can set the “Failed Request Tracing”:

Then you need to go to the “Failed Request Tracing Rules”

There you can add a rule that specifies which failures interest you. The analysis files are stored in the location specified in the Advanced Settings panel shown above. It will allow you to see how ASP.NET has walked through the various bits and determined which handler to use.

Wrapping up

At this point there are no more general pointers to give. Instead I suggest that you Google your way through what you find. The specifics of the application come into play here and things can get very complex. In a next segment I’ll look at (ASP.NET) modules. They go together with handlers like peas and carrots.

NB: Cross-posted from Blog2mind

More ...

An hour with Shaun Walker

peter@bring2mind.net — Wed, 22 Jun 2011 14:00:10 GMT

Is what I spent with him in a room for an interview. And you get to see over half of that. What am I talking about? Well, at the WebConnections conference in Hamburg (a 2 day DotNetNuke event) we interviewed a number of participants. Vincent Bourgonje from we-do-it in Holland and myself were approached beforehand if we’d be interested to do this. We both immediately said yes. Vincent is a “pro amateur” video maker who owns and enjoys operating some pretty serious equipment. As for me I’ve always wanted to interview these guys that I’ve come to see as my friends and colleagues as I felt there was very little public about the human beings behind DotNetNuke. This provided an opportunity for me to show you that there are real people with real lives working hard behind the scenes to get you this great platform. The open source world can be a tough and unforgiving place. With happy but sadly also sometimes angry voices hiding behind anonymous usernames. And sometimes we tend to forget that all these voices are not directed at an avatar or some abstract entity, but at the individuals behind this. The people I interviewed had one thing in common. We all make our living with this platform. So I hope that what you’ll take away from these sessions is that the people behind the avatars/usernames/profiles are likeable, decent human beings who are enthusiastic about the DotNetNuke project. And hopefully this will inspire you.

This is the first interview we finished editing. The interviews each lasted for about an hour as did this one. Although the original plan was to have “mini interviews” that could be cut down to lets say 10 minutes, it quickly became obvious there is a lot more to tell about the project. So sit back and grab a coffee. I hope you enjoy it.

More ...

Category: Events

Localization Editor v 03.04.00

peter@bring2mind.net — Sun, 29 May 2011 23:00:00 GMT

Hi all. Time for an update on the Localization Editor. In case you’re not familiar with this project, it is a tool to help you create language packs of your favorite DNN bit. Whether the core or a module. This new version of the editor aims to support something called CORE and FULL packs for the DNN core. Allow me to explain what’s going on.

A brief history of the full pack

Back when static localization was introduced (DNN 3) the distribution zip of DotNetNuke included a whole bunch of modules that were automatically installed. So when the opportunity was created to make a translation of DNN and upload it (as a so-called language pack), the translator was faced with the daunting task of not only translating all the core tidbits of DNN, but also each and every module that came with it. Pretty soon you saw the emergence of two types of packs: the CORE and the FULL pack. You probably guessed it. The FULL pack included all modules and the CORE didn’t. So far so good. People were comfortable with this division and who could blame a translator for not wanting to translate the forum module which, on it’s own, was about the same amount of work as the complete core at the time.

Later on, we discovered that the amount of stuff you have in the bin folder (i.e. all the module dlls and the core) has an impact on performance. So in an effort to improve performance and to “clean up” DNN introduced the “optionally installed” packaging in DNN 4. This meant that modules that were packaged as zip would get installed and those that were renamed to .resources were kept “at hand” and you could install them by checking a checkbox. This had one quirky impact on the language pack domain: when you installed a FULL pack and you did not have some modules installed, the installer would happily create the necessary folders for the localization files and dump them there. As there was no impact on anything (performance or otherwise) this was never seen as an issue and we all happily continued.

Then DNN 5 came along. And with DNN 5 we got a new installer. And as is often the case: those that designed this bit were not used to making or installing language packs on a daily basis. The new installer made everything a “package”. A package contains 1 or more items to install. A feature of this new approach is that you are assumed to “package” all your bits into a single file and properly announce them in one manifest. This means that if you upload a language pack consisting of more than one pack (e.g. core plus several modules) you list them all in the manifest. The manifest also required a much more rigorous approach to anything being uploaded. This included having language packs bound to a “parent” package. It’s easy to see the advantage of this approach. But when you now make a FULL pack and follow the rules and make that a collection of packs in a single package, the installer will refuse to install it unless all items are installed. And it will fail the complete package! By its very design the installer assumes you install packs one by one for each bit. Now we add a second complexity at this point.

DNN 5 also broke off quite a few bits that were previously integrated. If you check out the Install folder of a distribution zip you’ll see what I mean. This now includes a whole bunch of providers. All of these are separate packages. It’s a really great feature as you can roll out a lean DNN with providers of your own choosing. But unfortunately it meant that if you were to follow the rules and upload language packs for each bit of the core, you’d end up uploading something like 20 packs for each language. So it’s hardly surprising that since the early days of DNN 5 we have heard some resentment from the DNN translators.

Moving forward

With DNN 6 we hope the issue of the installer failing on a compound package will be resolved. I.e. if one of the packs has a dependency on an object that is not there, the installation of the complete pack should not fail. This is an important step to support FULL packs in the future again. For DNN 5 it is too late. The only way to install full packs in DNN 5 is to package it all together in one pack which you label as being a core pack. It’s not pretty but there is no other way for now.

Implications for Localization Editor

The localization editor was not previously well equipped to deal with compound packs. This came down to the fact that each version of DNN can have a different set of items (modules, providers, skins, etc) that are installed by default. So the first thing to do was to implement some kind of bookkeeping. This is done in the new table LocalizationEditor_ObjectCoreVersions. There we keep track of every object that is distributed as zip (i.e. installed by default) with the core version. If you already have the Localization Editor running with stuff in it, you’ll need to somehow make the records yourself if you want to benefit from this aspect. If you start from scratch I’ve made a complete SQL script to add ALL core distributed texts for all DNN 5 versions! In the first lines you’ll see ModuleId being added so you need to change the value of that for the module ID of the localization Editor you’re targeting (find out the module id if any module by clicking “Module Settings” and examining the querystring where you’ll see a value for ModuleId). If you run this under Host > SQL … you’ll probably see an error if you haven’t allowed for uploads bigger than 8 Mb as this file is 9 Mb long! So set the maxRequestLength in your web.config to something a little bigger. Anyway, if you use this you can hit the road running and have a Localization Editor ready to start receiving your translations. Note you must add at least 1 permission (i.e. binding a user as translator to a specific locale code) before you see the list of objects in the editor:

Pack Upload

In this version I’ve also spent some time on language pack uploads. It used to be hidden on the object details screen but now it is on the main screen:

Follow the instructions on screen to upload an existing pack. This will allow you to migrate your old packs into this new system. Lots of time has been spent looking at various packs to see how they were made to come up with a solution that is able to parse all these packs.

Pack Generation

In the new version downloading a DNN core pack brings up the following on your screen:

As you can see you can now download the core or the full pack. The CORE pack will include all objects that were installed by default and the FULL pack will include every object that came with the release. This also takes into account the version numbers of those specific objects.

Final words

The downloads can be found on CodePlex:

http://localizationeditor.codeplex.com/releases/view/67313

The development is done by myself. The DotNetNuke Internationalization team is involved as peer group. If you have any comments talk to one of us. We realize this is not a “perfect” translation tool, but we all have limited time to devote to this project and our aim is to provide a mechanism with which you can manage language packs.

More ...

Category: Internationalization

Newsfeeds 04.02.00 Released

peter@bring2mind.net — Thu, 12 May 2011 18:27:00 GMT

It’s been a while since my last post here. Recently, DotNetNuke has begun releasing release candidates of DNN 6 and all of us extension makers need to keep a close eye on this development. If you aren’t already you definitely need to take a look at this prereleases and test any stuff you have to make sure you’ll be ready when it is released. I know they do all they can to avoid breaking changes, but (as they probably say in the Langley office) it is hard to make an omelet without breaking a few modules.

DNN Newsfeeds is no different to other modules in this respect and all module leads have been encouraged to implement a number of changes as we gear up for DNN 6. Some of these are basic modernization changes to limit the risk of dependencies on really old functionality. That is why the new release has a dependency on DNN 05.04.04, has a brand new DNN 5 manifest, and has its solution file upgraded to VS 2010.

The other change that is noteworthy is the ability to syndicate the module’s aggregated feed. This means you can now use the module to “wire” RSS throughout your site. I.e. you put it on one page and have it aggregate a number of feeds and use that feed again in another Newsfeeds module somewhere else to aggregate with other parts of the site. It brings one goal I had in mind closer: to be able to build a site like for instance the BBC news site where at every level you can grab a feed and where news bubbles up through the site right to the front page if so desired.

Find the module here on Codeplex:

http://dnnnewsfeeds.codeplex.com/releases/view/63201

The full list of changes for version 04.02.00:

Enhancement: Dependency to DotNetNuke 05.04.04 (20320)
Enhancement: Solution file in VS 2010 (20320)
Enhancement: .Net dependency to 3.5 (20320)
Enhancement: DNN 5 module manifest with license and release notes (20320)
Fix: Fixed saving of referenced XSL files for preprocessing (19826)
Enhancement: Selectable link target (18288)
Fix: Fixed bug where feeds were not initially displayed (17998)
Enhancement: Syndicate module's aggregated feed (20321)
Fix: Fixed the aggregator changing pubDate on retrieved feeds (20322)
Enhancement: Only users with edit permission can now clear the feed cache for a module (20323)
Enhancement: Prettyfied the default transform
Enhancement: Hid datagrid headers in edit screen if no feeds are defined yet

More ...

Category: Development

News Feeds (RSS) 4.1.0 Released

peter@bring2mind.net — Thu, 27 Jan 2011 12:06:00 GMT

Hello everybody. It has been over a year since the last release of the newsfeeds module, so about time for an update. This new version includes a couple of major enhancements over 4.0.

Caching

The single most important challenge since the old days of Newsfeeds 3 is to reduce the number of calls the module will make to the source of the feed it displays. I.e. caching mechanisms. Newsfeeds 3 let DNN handle it as any regular module can. This means DNN decides that a module doesn’t need to “run” and just outputs the HTML blurb that has been cached. There were a couple of issues with this. Notably that (1) this bypassed the “intelligence” of the news source regarding caching and (2) a spike in loading if the site had been asleep for a while. Regarding 1: any news source can communicate in its feed how long it should be cached. A feed from CNN may well have a shorter refresh time than one from let’s say the DotNetNuke site. And regarding the second point: if the site has gone asleep and DNN decides all content should be refreshed then the module will download its feed when a user comes to visit. The biggest issue there was that the page would not load until the feed was in. This caused large delays (and unwanted speed dependencies) for DNN sites using this module.

Newsfeeds 4.0 approach

The introduction of feed merging in Newsfeeds 4 has compounded this issue as now a number of feeds may need to be retrieved upon load. The first intervention was to break page loading from the feed fetching. This can be done using Ajax. The page renders the Newsfeeds module as an empty container and after the page has loaded this empty container goes to the server to get its contents. It was subsequently tweaked to decide whether to do this or not based on the state of the cache. So if the module needed a refresh Ajax was used and if the cache was valid the module would render that cache immediately. This has worked well but I felt this was only a half-way solution.

Newsfeeds 4.1 enhancement

In the latest version we introduce a background task. A DNN scheduled task now looks through the cache and refreshes feeds if necessary. This background loading means it is definitively decoupled from page loading. If successful, then a refreshed feed will tell the module it needs to run the aggregation logic again (this is done within the page calling logic) but this now no longer depends on the loading of feeds and is virtually instant. The actual contents of the feed (plus pre-processing as described below) are cached in the database in the feed record:

The aggregated feed is cached in the module cache on disk (note we intend to expose this as an aggregated feed in a future version) as Feed_[ModuleId].resources:

As a result we have now set the Ajax feature by default to OFF. You can switch this yourself in the module settings (as well as indicating whether the background task should refresh the feeds of this module):

Pre-processing

In Newsfeeds 4.0 we introduced aggregation. This has been the largest technical challenge to date. We copied in logic from an old project called RssToolkit but this has also led to a number of issues. Most notably the sensitivity to feed validity. In part this is inevitable. Allow me to explain. For feeds to merge they must be similar in structure. Over the years many standards have evolved regarding syndication, the most popular of which is RSS. But even within RSS there are different flavors. If you’re interested check out Wikipedia on syndication, rss, rdf, atom, etc. To make matters worse, various news outlets make their own version of one of these standards. Over the past year I’ve been shocked to learn that large, respectable news outlets feed “illegal” feeds. I.e. their feed does not conform to the set standard. RssToolkit is quite strict about what it accepts and will reject feeds that do not conform to one of the set standards. In part this is necessary to avoid a mess when trying to merge different feeds. Internally all feeds are transformed to Rss 2.0 before being merged.

A new approach

For this version I created the possibility for the admin to define a “pre-processor” for a feed. This means you can intervene before the feed gets to the RssToolkit and make it compliant even if it isn’t. This is done through an XSL transformation or a so-called pre-processor.

XSL pre-processing

You can define a feed to go through XSL before it gets to the aggregator. My hope is that for some of the more popular news outlets people will share their XSL if the feed does not conform to RSS 2.0.

Note that since release we discovered a bug regarding the ability to link to file or using the dropdown with your own transformation. If you see an error about failure to compile the XSL then use the URL option to point to the XSL sheet. We’ll solve this for 4.1.1.

A custom pre-processor

Sometimes XSL is too cumbersome and you really want to process based on the string representation of the feed. This is now also possible. For this we created a new interface:

Namespace PreProcessing
 Public Interface IPreProcessor

  Function Process(ByVal xml As String) As String

 End Interface
End Namespace

We included a Twitter pre-processor in the module which will transform a tweet to properly format links to tags and people.

The Twitter pre-processor is listed in the dropdown for pre-processing:

You can now create your own pre-processor and specify its full type specification in the module’s UI when editing a feed.

Where to get

The Newsfeeds project and its downloads can be found on CodePlex:

http://dnnnewsfeeds.codeplex.com/

More ...

Impressions from DotNetNuke OpenForce EU 2010 (video)

peter@bring2mind.net — Mon, 08 Nov 2010 23:00:00 GMT

At the last OpenForce event I took my little point and shoot Panasonic along with the intent of using it as a video camera. Nothing works better to capture the atmosphere than video, IMO.

Category: Events

The never-ending language wars

peter@bring2mind.net — Fri, 05 Nov 2010 20:57:17 GMT

Those that attended one of the OpenForce events (or were paying attention in the forums or on Twitter) may have heard that DNN Corp is contemplating switching the programming language of the “Open Core” from VB to C#. There are a number of reasons for this. Notably that it is easier for them to find software engineers for the project and that it would attract more (C#) developers. There is now a Twitpoll on the issue as this is not a trivial matter and DNN Corp would like your input on this. In this post I’d like to weigh in on the discussion.

Category: Development