Hi JH,
I probably have as much integration experience with DNN as anyone here. I've successfully paired DNN and a non-.NET legacy system (via a custom provider), DNN with MS SQL Reporting Services (with forms Authentication and extensions to DNN's role provider), DNN and MS CRM 4.0 (forms auth), DNN and MSCS 2007 (via UpmMembershipProvider), and DNN-SSRS-MSCS (with Upm). It seems like there might have been one or two other combinations I tried out in a non-production environment.
DNN really does deliver what it promises with it's provider-based authentication system. You should be able to accomplish what you desire here, especially if your custom application relies on forms authentication.
Hope this helps!
Brandon