Thanks John

Things are a little bit nuts in the evenings for me this week but I'll try to email you over the weekend.

I've seen dome discussion about sharePoint not being able to crawl a site that has the AD provider inabled.
I added this to codeplex but thought that I would add this here also. I've found a fix to allow SharePoint to index DNN.

I've not included the MOSS servers in the IP inclusion for auto login, but SharePoint still returns an error when trying to crawl the site.
It seems to be an issue with the AD provider - AuthenticationModule.vb
After the code goes through the IP strings to determine auto login it then runs the
Response.Redirect(url) *** Strange.... but this is where SharePoint returns the error.

A few lines before this, there is code that looks for the USER_AGENT for Google Spider. I added a line of code to look for MOSS (SharePoint 2007) robot ("MS Search 5.0 Robot"), and now the sub will exit out and SharePoint indexes the site as an anonymous user.

If Request.ServerVariables("HTTP_USER_AGENT"). Contains("gsa-crawler") Then
 Exit Sub
End If

If Request.ServerVariables("HTTP_USER_AGENT").Contains("MS Search 5.0 Robot")Then
Exit Sub
End If

It would be greate if a text box (similar to the IP addresses for auto login) could be added to the settings page and allow the user to add a string of HTTP_USER_AGENTs that they want to exclude.