I am hoping someone has some sort of idea on how to troubleshoot this one before I go insane :)

With increasing frequency DNN will just have a meltdown. Stops serving pages, CPU on server @ 100% for about 10-15 minutes. When I run SQL profiler I just see an endless stream of

exec dbo.GetTabPermissionsByPortal @PortalID=<portalID>

and

exec dbo.GetHostSettings

Over and over again. The Portal IDs change in GetTabPermissionsByPortal in no apparent order. The thing that baffles me is that I can understand if there's some kind of permissions indexing process that needs to happen that would explain the multiple hits to GetTabPermissionsByPortal with different portal IDs. But doesn't explain why there would be a thousand calls to GetHostSettings with no parameters.

I've seen other posts on this subject that point to a circular tab reference but we've gone through all tabs (and run the script somebody posted on another forum to verify no circular references). Cleared out IsDeleted tabs & modules too.

What gives? Hopefully someone will have some suggestions before I go throw hardware at the problem.

Thanks!

Good news / bad news...

The bad news is that I still don't know what originally causes this issue, or even why it's happening.

The good news is that we reconfigured the install to run on multiple servers with a load balancer and everything runs a LOT more solid.

For those interested that may be attempting this I'll provide the steps we took here to hopefully eliminate someone's headache down the road. We did not use a file replication method and reconfigure the DNN install at all, as the DNN Web Farm PDF suggests. We have just enough files riding around on a daily basis to make me uncomfortable with a multimaster file replication method. Also we did not do any SQL replication.

1. Created an active directory account for DNN to run under (this is within a domain).
2. Configured IIS default site on distributed application servers to host off a UNC share from the central server.
3. Adjusted .NET security policy to allow FullTrust on the share (via Caspol.exe)
4. Configured IIS to use the DNN account for access and configured appropriate permissions on the install.
5. When ASP.NET runs, it establishes a file monitoring object on each folder in the application. This became a problem very quickly ("the network BIOS command limit has been reached" errors). Contacted Microsoft in regard to KB 911272 and obtained the ASP.NET hotfix that allows you to turn File Change Notification on and off. Downside here is that we will manually need to restart the 'cluster' to recompile the application. Note that KB 911272 hotfix does NOT work with .NET Framework 2.0 SP1; only the pre-SP1 Framework 2.0. Also adjusted the lanmanserver and lanmanworkstation parameters in the registry to allow for more requests across a SMB share just in case.
6. Configured MIME types in IIS (by going to properties of the Server object) and adding .flv as flv-application/octet-stream to the MIME types list. (Wondered why flash videos weren't displaying at random).
7. Implemented a load balancing device in front of the distributed servers. Initially tried with plain Windows Network Load Balancing services but even if it hadn't totally freaked out our Cisco infrastructure by spoofing ARP addresses to make it happen, I wasn't feeling comfortable enough with it to sleep well.

The .NET framework permissions and the load balancing were probably the most difficult and frustrating pieces to get in place but now we probably have the best DNN performance under load we've ever had.

Oh, and no GetTabPermissionsByPortal endless queries as of yet, either! :)

Brandon