Exchange Front CPU high

[Solo4357]

I have an exchange front-end server that's been working well but the last few days has had 100% CPU usage on the w3wp.exe. I tried a few hotfixes but they were either superceded by SP2 or just had no effect. If I reboot the firewall (thus removing all the connections) w3wp.exe goes to zero, but otherwise just sits at 100 or as close to it as possible. It's making my mobile sync very slow.

My guess is I have an OWA user that's messing it up or I'm simply overloaded. But I wouldn't think so given the number of users. Is there a way to monitor my front end to see if there's a single user causing this or to see how many are coming through?

Email seems to flow through it ok, it's just anything http/https. Maybe the foms compression should be high/none? Which is better?

Help!

 

[Zelandakh]

Try using perfmon to look at some of the IIS counters. Check the IIS logs. Check AV is up to date.

Look at memory - pages/sec.

Reapply SP2 and restart.

 

[Solo4357]

Re-applied ap2. No dice. Not sure what to look for in the logs. AV is up to date.

 

[Solo4357]

my total allowed async I/O requests is the only one really high. 86000.. not sure what that means...Ton of connection resets... not a lot of network traffic though. Tried to reset the firewall again and it still stayed high. I'm installing a new fresh one but I'm stumped.

 

[Solo4357]

Figured it out. I used a program called IIS guard on the server to see the ip addresses coming in. I would go into IIS and block the entire class A of a particular address.. restart IIS to clear the w3wp and wait. Usually it happened within 15 minutes. This lead me to believe it was a phone because we by default set phones to active sync ever 15 minutes. I kept whittling down the ip's until I figured out which one it was and then blocked the class b of that set of addresses. I unblocked all the others I did to make sure and viola, no more pegged cpu.

Next day I waited until I got a call from a user about his cell not working. Unblocked the ip's synced him up and boom, there went the cpu. Verified I wasn't seeing things and then we went through his inbox and got rid of anything he didn't need. Re-synced and everything is good.

I haven't been able to verify this for sure but I have a theory. I couldn't find any errors in my exchange database nor did any of his email look nasty, just some newsletters in html. My guess is that since activesync is essentially a web page fed to a phone in a special way, (it does use the w3wp service after all), and since we only send partial email to said phone (.15k, they have to request the rest, typical phone behavior).... is it possible that the html code coming down, cut off in a random spot (.15k to be exact) was in a spot that caused the web server to keep thinking it needed to resend that page (email) to the "browser" (activesync) requesting it, thereby causing the web server to flake out? It's my only theory and may be way off.

Anyway if you have this problem, these steps worked for me. I do know it was a possibly corrupt email and it was a phone causing it. Clean out bad emails in a box and see if that fixes it.

 

[xmsre]

In Exchange 2003, EAS uses the IIS virtual directory Microsoft-Server-ActiveSync. Last year, there was a problem with large messages and the way Exchange allocates memory - kb 941439. Your situation would be similar. A lot of small messages and the same sort of allocation problem - even though each is only 1500 bytes going to the phone, Exchange has to allocate 4K of RAM each... Was 941439 one of the hotfixes you tried?

 

[Solo4357]

Yeah it was. It had no effect.