Exchange CPU at 100% and unresponsive 1

[JonReid]

Recently, our exchange server has been running at 100% CPU. We have tried just about all of the recommendations, with no luck. Microsoft seems to think it is corruption in the database. We are working on that now.

This is what has typically happened over the last several days. CPU is at 100%, with Store.exe and Services.exe splitting about 90% of that. Early in the AM, Exchange seems to function fine, in spite of this. Sometime between 8:30 and 9:30 (when I expect just about everyone has logged in), Exchange and Outlook become unresponsive. A restart of the store seems to fix this problem. While the CPU quickly spikes to 100% again, Exchange and Outlook run fine the rest of the day, in spite of being at 100%.

Today something different happened. Services.exe CPU was not high and Store.exe CPU was around 75-80%, with at least 20% idle time. But same problem occurred at roughly the same time. So far, the same solution!!

We also have isolated incidents of messages getting stuck in people’s outboxes for days.

 

[bigdog92]

just a thought, have you ran a virus scan on that machine? if you know there is no virus then i would just defrag that database (eseutil /d priv.edb) which is easy but time consuming depending on the size.

 

[JonReid]

I have tried both without much difference.

Thanks for the tip.

 

[N0ktar]

Have you enabled logging on Exchange? Have you checked the Event Viewer recently and when it first started happening?
Have you installed any new software of made any significant changes to the server?
You must first locate a moment in the past when you first started having problems and look for the triggers there; have you rebooted exchange yet? applied patches? reinstalled SP4?

 

[pdubs410]

I had a similar problem with a customer awhile back, where Store.exe just started hogging 75-90% of my CPU. Evidently, there is some sort of "memory leak" in Exchange where Store.exe does this, and SP3 was supposed to fix it-SP3 didn't fix it for me. I just added another GB of RAM, adjusted my page file, and now the server runs great. CPU runs a little high, and Store.exe still takes up a boatload of memory (that is a given)but it seems to function okay.

 

[JonReid]

Well, I have been in conversation with Microsoft Support on this one also and finally was able to resolve this problem.

It turns out that there is a knowledge base article that describes our problem exactly. The article is KB822044 (

http://support.microsoft.com/default.aspx?scid=kb;en-us;822044)

Also, we sent them a performance monitor dump of when the symptoms presented themselves along with debugger information and they pointed me to the same KB article. They also sent me the post-SP4 hotfixes which are not available on their public web site.

The Q822044 hotfix resolved the problem and we are now back to normal, running at approximately 5-20% CPU.

Thanks all for your suggestions.
Jon Reid

 

[p4ul]

Jon
We have the same problem.
How / where did you get the hot fix
Paul

 

[JonReid]

We have Microsoft Prime Support and they let us download it off of their ftp site. I suggest calling Microsoft Support. They may not charge you for this patch.

 

[CCoulter0001]

My question is directed to the excellent post Mr. Reid provided concerning store.exe causing server CPU pegging. Can you pass the hot fix along to me or is there a web place that it can be accessed? I don't feel that Microsoft deserves $245 for me to ask MS Tech Support to provide me with this patch to address a problem with Exchange 5.5.

Thank you kindly...

Chuck Coulter

 

[JonReid]

Chuck, I just got off the phone with Microsoft Support. They say it is ok if I give the files to you. They recommend, however, that you call Microsoft Support to determine that you are having the same problem that we were. (I would review technet article q822044 to make sure you are having the same symptoms.)

The technician stated that this fix contains the latest store.exe (ver. 5.5.2657.51). He says that applying this fix may not resolve your problem but it won't break your system. (Be aware that we are running Exchange 5.5 Enterprise Edition. I don't know if there is a difference in the store.exe between Standard and Enterprise...)

You can contact me at jon.reid@po.state.ct.us and we'll work out some method of transfering the files.

Jon Reid

 

[exchangerecovery]

If you are having this CPU problem you either need to install the ROLLOVER PATCH SP3: or SP4 will fix the issue

 

[JonReid]

Unfortunately, with this specific problem, that is not true. We were running SP4 before the problem occured and we also reapplied SP4 during the problem and it did not resolve the problem. Microsoft sent us two specific hotfixes based on performance monitor logs and a debugger dump data that resolved the problem.

Jon Reid

 

[beannie]

Jon, Could you share what hotfixes they had you apply.
Beannie

 

[jomorgan]

Can I ask, has any experienced these cpu issues after a bulk move mailbox? It looks like that's the common factor in our case.... MS also gave us the latest store.exe as a fix - but we still experienced 100% cpu spike after a bulk move mailbox?

 

[cmeagan656]

Depending on what Antivirus software you're using on your exchange server a bulk move mailbox could cause a 100% CPU spike.

Depending on how you have your Exchange antivirus configured it could be the cause of the CPU to pegging out.

 

[jmm7772]

it's happened again - convinced it's our av software. though we do stop av before move of mailboxes as soon as we restart it spikes. we have background scanning on - as the though is it would be even slower with bs off as we get an av update at least once a day. any help much appreciated. it's a vicious circle - we rebuild server move mailbox - cpu spikes. again and again.

 

[zoeythecat]

What virus program is protecting your email server and file servers?

 

[Skadaddle]

Jon, I saw your post on the Tek-Tip website pertaining to the Store.exe memory leak. I took your advise and contacted MS and got the "HotFix" but I came in this morning and found the same problem was happening. You mentioned in the post that MS gave you two "HotFixes" can you tell me what the other one was? I got the one called Exchange5.5-KB822044-x86-enu. Can you send me the other one.
Thank you sir.

-Michael

 

[jmm7772]

We use eTrust v6 and this was the root of all our problems. With all the exchange options off and server rebooted everything is back to normal. Think it's a combination of large attachment database (due to lots of moves & loss of single instance storage), company procedures, and still using E5.5
E2K3 has a new version of MAPI API and this apparently works much better with av software.

 

[Skadaddle]

Thank you Jon for replying to my request. However, I applied the patch you sent me and all seemed fine. Until This morning. I came in and the exchange server was pegged at 99% CPU. If anyone can help I would be so greatful. I will call MS and see if they have any further fixes.

System info.

Exchange 5.5
Windows 2000 server
running scanmail 3.81
Norton Anti-virus

Problems started happening last thursday March 4th