IIS queue fills up and does not recover when Azure is slow
===

We've been experiencing a very serious issue with IIS and Azure.  Not sure if this is something on IIS side or our custom code side of things.

I'm involved with two websites that are running in Azure (Site A and Site B).  (Standard WebRoles, ASP.NET MVC3).  Both of theses sites are architected completely differently and have nothing to do with one another, but both have shown similar behavior under similar circumstances.

Site A is running with Session state enabled.  Session is stored in SQL Azure database.  Most of the calls to Site A are secured via ASP.NET SQL Membership provider that points to SQL Azure database

Site B is running with Session state enabled as well.  Session is stored in Azure AppFabric cache.  Site B also has an http handler that talks to AppFabric Cache and Azure Table Storage.

Issues begin but do NOT recover when key Azure resources (like SQL Azure or Cache) become very slow.  When these resources become very slow and processing time for each request exceeds a minute, Azure's load balancer terminates those connections but IIS on the web roles is not clearing/removing these requests from its active queue.

So, the issue is not that the sites are non-responsive when SQL Azure or AppFabric Cache is very slow.  The huge issue is that the sites **do not recover** when SQL Azure or AppFabric Cache come back and start working normally.  Requests are sitting in the Active Requests list and do not go away for long time (hours?).  Frankly, I don't know how long they sit there for, because we reboot those servers ASAP.  Since Azure resources can have occasional intermittent issues, and since traffic to both sites is very high, both sites quickly boggle down under the weight of non-clearing requests.  IIS queue fills up and until someone goes in and restarts the App Pools, the sites are unavailable.