Its amazing how one single line of code and a bad assumption (or misunderstanding) can cause all sorts of strange production issues.

The day started out with static content throwing exceptions that contained our UnitOfWork in the call stack.  My first question was, why on earth is a static resource request hitting our begin unit of work.  Well it turns out that we map all requests through the ASP.NET pipeline because of some nameless .NET document management module.  Unfortunately we generally don’t have wild card mapping enabled for ASP.NET in our dev environment because:

1. We weren’t aware of the requirement.
2. There’s no documentation surrounding proper dev environment setup (e.g. script).
3. Only one dev in our entire organization has ever actively touched this module.

Why is the wildcard mapping a big deal?  Because a single page load will create 10, 20, or more HTTP get requests for static content.  If you have a threading bug in your begin request, they’re much, much more likely to show up with numerous requests. 

The sad truth of the matter is that because our environment we have a lot of NHibernate session factories, which is not normal for most applications. All of these session factories need to be configured and were using Fluent NHibernate to do so.  FNH is a great tool and makes configuration so much easier for us devs, but the fact remains FNH is not thread safe.

Seems easy enough, put a lock around the FNH stuff.  Well, we did, but the locker class was not static, we were relying on AutoFac to ensure that our NHibernate session factory class (the class with the lock) was a singleton.  Not too hard since AutoFac defaults to singleton.

Here’s the mistake.  ASP.NET creates not one, but many HttpApplication instances for each app domain.  I was falsely under the assumption is only ever created one, maybe two, but only ever kept one around.  Our AutoFac registration was setup in the Global instance constructor, so each app instance was getting its own container instance.  Can you guess why this is bad?

Each request had its own NHibernate configuration class with its own instance lock, thus multiple threads were entering into the critical section at the same time since they couldn’t see each other.  Lots of weirdness ensued.  We saw all sorts of the strange Nhibernate and FNH errors, things like property “XYZ” doesn’t exist on class “ABC".  Odd stuff that definitely smelt of threading.

I was starting to think SELECT was broken (e.g. AutoFac), but soon realized the errors of my ways with a simple Debug.Assert. Yep, multiple container, produce multiple “singletons”.