iisreset's needed a few times each day

[andegre]

Hello all, I realize my question that I'm going to ask will not be able to be "solved" per se, but I would just like some suggestions on how to find the problem. This explanation may be quite long, so I apologize. Here it goes...

Background Info:
Servers: I have a large production environment that is load balanced between 8 physical Windows Server 2003 servers. We are using Microsoft's NLB load balancing software. Each server has the EXACT same code loaded (obviously) and is running my website.
Application: This is a VERY large application (a couple million lines of code in total). It consumes a fairly decent amount of memory, each server is usually taking up about 600 to 800mb in the w3wp worker process thread and usually consumes 20 to 50% of the cpu utilization.
Database: all 8 app servers connect to a single (different) SQL Server with runs SQL Server 2005.

Problem: At least 3 to 4 times throughout the day, IIS on one of the servers starts to "hang" or not respond. When I get a call that "the system is running slow" or something like that, I run a little bat file that will load all 8 "localhosts" onto my build server. Then, whichever login page does not load at all, I know is the server that is causing the problem. I then go to that server, do an "nlb stop", then "iisreset", open up "

http://localhost"

to make sure it loads (always does after the reset), then I do an "nlb start" and the problem is fixed. This process repeats itself 3 to 4 times throughout the day. And sometimes, after getting one of the servers reset, I will run my script again, and a different server will be hanging, so I will have to repeat for that one. About half the time for each call, a second server starts hanging after fixing the first one.

Question: Does anyone know of any tools out there that could be used to monitor either the w3wp worker process thread, or IIS in general to help me figure out why IIS keeps getting hosed up (yes, technical term there)? This is getting very frustrating as I'm not really able to leave for time off because I'm afraid the servers will act up again.

Thanks for any replies!

 

[jbenson001]

The first thing that comes to mind is that this possibly code that is not disposing of objects that need to be or connections to the DB that are being left open.

 

[andegre]

FYI-I also have this same question in the IIS forum...and a person over their suggested a memory leak.

Would not be disposing objects correct be caught using a ANTS Memory Profiler also...aka is not disposing of objects correctly the same as a memory leak?

 

[jbenson001]

Would not be disposing objects correct be caught using a ANTS Memory Profiler

I have no idea, I don't use that software.

also...aka is not disposing of objects correctly the same as a memory leak?

Yes

 

[ca8msm]

I'd hope that ANTS would pick something up, but it really depends on the load that you throw at it. You may want to run Wcat alongside it to see if that can pick anything up.

What is the memory usage like at the time when it hangs? Does it appear to be higher than the other servers in the farm?

Mark,

Darlington Web Design[tab]|[tab]Experts, Information, Ideas & Knowledge[tab]|[tab]ASP.NET Tips & Tricks

 

[jmeckley]

something to mention about profiling for a memory leak. they are difficult to pinpoint because the cause of the leak is not listed by the profiler. in other words you will be reviewing the memory profile for what's not there.

Here is one example of pinpointing a memory leak. The profiler will show you there is a memory leak, but not where the actual leak is coming from. For that you will need to crack open the code. If it's not your source code Reflector can assist you.

I would also recommend the book Release It!. It's a very simple and enjoyable read about various enterprise scale systems. In it the author discusses common pitfalls he has encountered and how to remedy them.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259

 

[andegre]

What is the memory usage like at the time when it hangs? Does it appear to be higher than the other servers in the farm?

The memory usage (according to Task Manager) is about 100 to 150mb higher than what the other servers are showing at the time that that specific server is hanging. Which is between 850 and 900mb. Servers average anywhere from 600 to 750mb when they are "working".

As for the profiling. I tried running ANTS Memory Profiler and it crashed everytime I tried to get the second memory snapshot. On to Plan B

Next, I tried using Microsoft's DebugDiag tool. We brought MS in to help us figure out why our performance was bad when we did our .NET 3.5 upgrade (from .NET 1.1) and that is the tool that they used and I remember looking at the reports and it was pretty easy to see what the problem methods were.

Anyway, does anyone have any experience with using DebugDiag? After generating the dumps, I'm not able to read the reports (or they aren't showing me anything useful) so I'm not sure if I'm doing it right. I followed the procedure in this link that tells how to use it for memory leaks ([link]http://support.microsoft.com/kb/919790/[/url])

Also, I remember when MS helped us the first time, they said they needed 2 dumps to be able to analyze (makes sense) so they had something to compare to. Well, that link doesn't say anything about that so I'm kind of stumped on what to do from here, or how to do it "the right way". If anyone can point me in the right direction on this, that'd be great.

 

[JohnYingling]

I have II6. I discovered back in 2006 through PERFMON that sometimes an onslaught of connections requests would come in (INTRANET) and cause memory to go to max.

Fixed with follwing:
Open Internet Information Services Management from Administrative Tools,
expand Application Pools
righ-click on desired APP POOL
select Properties.
select Recycling Tab
Set Memory Recycling Checkbox
Set memory value to ridiculously high but still within virtual size value

You lose whatever was happening in that WP but it will restart automatically. NO IISRESET NEEDED.

Hope this is it.

http://www.moldyoldy.com/VBComparedemo

- free online Compare/Diff of snippets

 

[andegre]

To update everyone on this subject...

Boss had us get Microsoft involved with this issue and after MS requesting performance logging their suggestion was to implement the Web Garden approach.

So, today is the first day for the web garden, I've set the number of threads to 8 and also with a maximum memory used (not the virtual setting) to 350mb. (By the way, there are 8 load balanced servers also so essentially we have 64 threads running our application).

Now after checking the event logs of each server, we averaging a thread recycle approximately every 3 minutes. With everyone's experience with Web Garden's, is this recycle rate too high, or just fine? My initial thought is that it's too high so I want to raise the Max Mem Used setting to 375 or 400, but I don't want the server to start throwing "System.OutOfMemory" exceptions.

If anyone has any suggestions, or a "don't worry about it, that's normal", that would be great.

Thanks

 

[jmeckley]

you might want to ask in forum41. at this point you are moving away from software/application and into infrastructure.

Jason Meckley
Programmer
Specialty Bakers, Inc.

faq855-7190
faq732-7259