I may be mistaken, but I believe I recall that the calendar popup works okay in transitional. Regardless, SOMETHING happens in that mode, and it sounds like the original poster isn't seeing any activity at all.
In my experience, this is usually caused by a misconfiguration of the very exacting markup required by the extender. If faced with this situation, I'd use the exact markup with the toolkit samples and see if I couldn't get that to work. Once that was operational, I'd slowly change the toolkit example to meet my needs.
You sohuldn't need a scriptmanager on your page. DNN will include it automatically with the proper configuration.
I've used the calendar popup extender inside a DNN module before, so it can be done. DNN also plays nicely with .NET 3.5.
Brandon