Just an update:
The site went down again today, and even though the Physical Memory Usage was still at around 90%, there should have been more than enough RAM left for the site to operate.
I changed the app pool to a web garden, with 3 maximum worker processes and the site popped back up IMMEDIATELY without the need for recycling the pool. I also noticed that as I surf now, the site resources are now split between the processes. So instead of having a single worker thread consuming 100+ Megs of RAM, each smaller process only uses about 30-49 megs, and shuts down once idol. Unlike the BIG process, which got larger and larger as I worked in it.
If this hold up then it's solution I was looking for, but I'll keep an eye on it.
Robert