Worker process for websites contantly crashing - ASP.NET 4.0

[brokenhalo]

I am having an issue where the worker process for all of our websites crashes as the result of many unhandled exceptions. We thought this issue might have been caused by storing the session state in SQL instead of using a state server so we moved the session state back to a state server and the issue seems to have gotten worse. This issue also existed in ASP.NET 3.5, but seemed like it was not as frequent - we are downgrading back to 3.5 now to alleviate the problem a bit.

I have opened a ticket with Microsoft which they have escalated and there is still no resolution. They offered a workaround which involved switching the worker processes to ignore the unhandled exceptions and not recycle, but that was causing the worker processes to eventually hang and bring the associated sites down. Apparently Microsoft had a fix for this issue for ASP.NET 2.0 KB 957995 but this fix apparently did not carry over to later versions.

Below is a dump file that might be of some help:

[blue]
Exception Details
System.NullReferenceException
Object reference not set to an instance of an object.

at System.Web.Configuration.HandlerFactoryWrapper.ReleaseHandler(IHttpHandler handler) 
at System.Web.HttpApplication.RecycleHandlers() 
at System.Web.HttpApplication.ReleaseAppInstance() 
at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error) 
at System.Web.HttpApplication.ResumeStepsFromThreadPoolThread(Exception error) 
at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar) 
at System.Web.HttpAsyncResult.Complete(Boolean synchronous, Object result, Exception error, RequestNotificationStatus status) 
at System.Web.SessionState.SessionStateModule.PollLockedSessionCallback(Object state) 
at System.Threading._TimerCallback.TimerCallback_Context(Object state) 
at System.Threading.ExecutionContext.runTryCode(Object userData) 
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) 
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx) 
at System.Threading._TimerCallback.PerformTimerCallback(Object state)
[/blue]

I really hope someone here could help with a resolution to this. Thanks in advance!



Brad L.
Systems Engineer
Prestige Technologies
bradlaszlo[at]prestigetech.com

http://www.prestigetech.com

"Some things Man was never meant to know. For everything else, there's Google.

 

[jmeckley]

assuming you have logging in your application I would log informational and even debug statements for a period of time to see what's happening in the background. if you don't have logging, then I would add it and start capturing more information in the system. log4net is a solid logging framework.

if the current configuration recycles the process and turning of recycling brings the system to a crawl, then there may be a memory leak somewhere.

are you using events? if so you need unregister the event when your finished

foo.TheEvent += HandleTheEvent;
foo.TriggerEvent();
foo.TheEvent -= HandleTheEvent;

are you properly disposing of objects? if you are not calling Dispose in a finally block, then objects are never released, thus a memory leak.

using(var disposable = new ObjectImplementingIDisposable())
{
   disposable.DoSomething();
}

ObjectImplementingIDisposable disposable = null;
try 
{
   disposable = new ObjectImplementingIDisposable();
   disposable.DoSomething();
}
finally
{
   if(disposable != null)
   {
      disposable.Dispose();
   }
}

This is especially common with ADO.Net objects.
Beyond that there may be objects you are attempting to access in the background that just aren't available. for example accessing asp.net objects running on a background thread. or maybe there are cross threading bugs. these are worst kind, because they are so difficult to pin down.

have you profiled the application at all? this will also provide insight into how your application is behaving.

I would recommend the book Release It! There is a lot of practical advice on how to structure a system to avoid total system crashes for large scale systems. Not to many code samples, but concepts about how to architecture the application to minimize production bugs.

Jason Meckley
Programmer

faq855-7190
faq732-7259