MSExchangeIS Error Virtual Memory 1

[spike03]

I am getting an error in the event log that says:

Virtual memory is fragmented in a way that normal operation may being to fail. You need to restart exchange services to correct this issue.
FIX: on the Microsoft website it says that service pack 1 fixes the problem. We currently have service pack 3 installed on the server.

I am running W2K server serivce pack 3 with exchange serive pack 3.

I have to keep rebooting the server because it is slow.

has anyone experianced this problem before?
thanks in advance

S.

 

[bigpapapump]

Thanks Matt,

Would you mind posting those batch files?

 

[mdunph]

Mia03,

I've been battling this for about 10 weeks now. When the problem first surfaced I was seeing 9582 Errors, and that's when I applied the changes recommended by Nino Bilic in his presentation entitled "Troubleshooting Virtual Memory Fragmentation on Exchange 2000 Server." After making these changes the errors went away, but then just last week 9582 Warnings surfaced. I applied the hotfix, but two days lalter the error returned.

Last night I reapplied Exchange SP3 and a new post SP3 rollup package which I think was released on April 20th. Since doing that no warnings, but the server has only been up for 15 hours or so since doing that. I fully expect to see the warnings come back, and when they do I think I'm going to ignore them and see if in fact the warnings will bring my server down. I'm not sure whether this is a good idea or not.

However, I think the only way to elminate this issue once and for all is to install W2K Advanced Server, or W2003 Enterprise Edition with Exchange 2003 Ent Edt. There is no upgrade path so it will be a long and potentially painful process including losing the better part of a weekend and having to rely upon a restore of the Information Store, mailboxes and public folders from tape or disk. It seems the only thing which really resolves this issue is the use of the /3GB switch in the boot.ini file, and Standard Server does not support it so you have to install Advanced Server.

I'm not sure what the benefit of opening a pay for support call to Microsoft on this issue would accomplish. It seems these tweaks only work for a few weeks and are not a solution.

 

[Mia03]

Hi everyone,

I thought of going the 2003 route but it's a bit drastic. I have an entirely Windows 2000 AD network so would have to upgrade the Active directory DC, the version of windows on the server holding exchange and Exchange itself to fix the problem. That's just a little too scary for me!

Did you experience the 9582 warnings appearing repeatedly without developing into the error? I noticed that I can let the warnings appear for a few days before they eventually turn to errors. I'm reluctant to leave it and see what happens after this so I have always restarted the services at this point.

I'm going to pursue this with MS for a while because I'm not paying for the support call thankfully!! I have a feeling mdunph is right, W2K advanced server seems to be the way to go, even though it's still going to be a terrible hassle. I'll hold out for a while and see what MS says.....

Anyone else feeling like a career change?

 

[matt4x4]

Hi all,
Emiller wanted these batch files
to stop the services:

net stop MSExchangeMGMT
net stop MSExchangeES
net stop RESvc
net stop POP3Svc
net stop MSExchangeMTA
net stop MSExchangeIS
net stop MSExchangeSA

to start the services:

net start MSExchangeSA
net start MSExchangeIS
net start MSExchangeMTA
net start POP3Svc
net start RESvc
net start MSExchangeES
net start MSExchangeMGMT


As for your plan (mdunph) to upgrade to advanced server, it is a long and painfull path since there's no upgrade path, it may be more beneficial in the long run to go 2k3 for both server and exchange because knowing microsoft, as soon as you get your advanced server running, they'll stop supporting 2k.

matt

 

[matt4x4]

career change? - DAILY, If people really knew what we have to put up with, they'd likely pay us millions.
I'm going to be migrating my network to 2k3 this fall and am hoping that I won't see any errors for at least the first week...
This is an issue that seems to be so frequent, Microsoft really should have addressed and FIXED by now.
For now, the batches work great, feel free to use them and schedule them to run during your off peak hours, I have even stopped and started the store with users logged in and it has yet to affect anyone adversely. Luckily, no one has posted the same time the store was down.

 

[dmandell]

responding to emiller & Mia,
Installed the patch last night.

Looked at the perf logs this morning.
VM free block size started dropping within 2 hours.

Took 8 hours to get to under 16mb to start the 9528 warnings kicking off.

I can appreciate that many of you want to upgrade to solve this issue, but this has been going on for almost a year. I am sure we are not the only ones with this issue.

MS should have a workable fix for this. Spending thousands of dollars by upgrading is not an option for all companies. I'd like to see a fix solution from MS.

Would it be too much to ask those of you that openned tickets with MS to post the ticket numbers here? I'd like to reference them in my call to MS.

Let me know,

thanks!
Dana

 

[mdunph]

LOL I hear ya on the career change stuff. Well, seems like we are all on the same page there. i think we all would like to enjoy our weeknights and weekends a little more.

Scheduling a job which runs net start / net stop in the middle of the night is not a bad idea, and I may do it just so I can go on vacation someday without having to worry. However, I've been coming to grips with the pain associated with rebuilding this environment for some time. Matt, I agree with you in that I'll probably just go for broke and go the W2003 Advanced and Exchange 2003 server instead of W2K Advanced Server. I don't think I'll need to upgrade my W2K domain controllers and AD in order to achieve this though. I already have a Windows 2003 Server running in my W2K domain and have had no problems.

Two more interesting points worth mentioning since I've been posting to the MS Exchange 2000 Admin newsgroup as well. A tech there has replied with the following:

1) Don't worry about the 9582 Warnings ... if they are only happening during
parts of the day stay at the warning level then they are not something that
I think you need to worry too much about. You should not see any issues
from the warnings.

2) Contact Microsoft support there are other options that can be tuned that
have to be tuned by us.

Now, I'm all for calling MS if in fact there is a resolution and it sounds like there might be, but I also don't want to get transferred around the world twice without a clear solution.

 

[Mia03]

Hi mdunph,

I would love to be able to ignore the errors but if the virtual memory is visibly degrading, is that safe to do? Can't help feeling that course of action would eventually come back and bite me in the ass....



Mia

 

[matt4x4]

Hi mdunph and mia03,
Upgrading to a 2k3 server with exchange 2k3 is the least painfull solution. As for calling microsoft support and pay in order to tune more settings to fix a problem obvoiusly created by MS to begin with - I don't think so - this should be available in their knowledgebase (i have found some variable tuning articles, but they do not clear the warning on 2k std). Microsoft should not be allowed to charge for supporting you with their glitches, that is just ludicrous!
As for the VM degrading (mia), I originally had the warnings happen for over a month before I rebooted and started researching the issue (never got an erro though)- so if it is just warnings you're getting you should also be ok to continue running, if however the same event logs as an error you will have to reboot or restrat your services within 72 hours of the error unless you want to chance exchange corruption.
I however am leaving it with my batch file for the time being since I have more to do than run around attempting to fix MS bugs. Since it's worked fine for 6 months, i think I can chance a holiday now that the weather's turning nice!

 

[xmsre]

With 2K3, you get the /userva swith, and that could help. If you go this route, upgrade Exchange first, then the OS. Exchange 2000 will not run on Windows 2003.

 

[dmandell]

Mia,
Any more words from Microsoft on your open case for this issue?

Dana

 

[bigpapapump]

Well, the Hot Fix isn't so hot. It didn't fix a thing. I have backed my server down to 2 GB's of RAM and the error is gone.
Has anyone noticed if this error effects the nightly maintenance\online defrag of Exchange?

Matt, thanks for the batch file it works great!

 

[dmandell]

I have created a ticket myself with MS to see if they can solve it.

It does not seem to affect my nightly maintenance on our server.

I am currently monitoring the VM largest block size with perfmon. The size of the largest free virual memory block drops consistantly in exactly one hour increments. (I am monitoring in 1 minute increments)

There must be some exchange or windows process that kicks off every hour that is causing it to drop.

It's not public folder replication, I already checked that.

MS is still scratching their heads. I'll see what they come up with and let you know.

Thanks for letting us know that dropping the memory to 2GB solved the problem in your case. How long have you been running successfully with 2GB?

Dana

 

[Mia03]

Hi all,

No word back from MS yet but I have been out of the office for most of the past 2 days.

I'm now getting about 20 hours after a restart before errors start appearing. Note - I'm now getting errors straight away without warnings first.

Thanks for those batch files Matt. I think I'll have to use them over the weekend.....

As a matter of interest does anyone use Mailsweeper? I'm starting to have problems with this on the mailserver too and don't know if they are related.

Mia

 

[dmandell]

Mia,
are you monitoring VH Largest Block size as suggested in KB 325044?

I am curious if you are seeing the same type of behavior as I am before the errors pop up.

MS has recommended some more tuning parameters, and I changed the msExchESEParamCacheSizeMax attribute and it definitely changed the fragmentation behavior. After 20 hours (same as you) instead of a warning, the fragmentation is now sitting at just above the threashhold of the alarm (16mb) for win2k server.

I'll let you know if it stays there or continues to drop.

Dana

 

[bigpapapump]

I spoke to soon, the message is back. Maybe because the system reports 2,096,624 KB of RAM.
Either way I guess I'll add the third GB of RAM and wait for MS to stop scratching their head.

I am also going to do an offline defrag this weekend.

 

[matt4x4]

Well, I guess it's safe to say that we're all just chasing our tails on this one, maybe they'll get it figured out at MS eventually (likely right after we all upgrade).
For anyone who has changed the registry and is now getting the errors without any prior warnings, I would recommend you change the settings back to default since a warning will allow for you to take some proactive action. I also read that once you get the error, there is a good chance that your store can corrupt quickly to the point of no repair. My suggestion - deal with the warning in due time rahter than scrambling after the error takes you down.

 

[dmandell]

Well, the good news, is that you will get log errors below 32MB VM largest block size, but your IS will not dismount until 10MB VM largest block size.

So it's not as detrimental as you might think.
(that's why setting up perf monitoring can save you)

Although I am getting errors in the log, from my perf log I can see that I am not in the danger of corruption, because my VM largest block size has never dropped below 16MB.

The guy I working with from Microsoft seems good, we are doing more tests and hopefully I will have something to report after the weekend.

Good luck, y'all,
Dana

 

[xmsre]

I was sitting there thinking about how ESE uses and allocates virtual memory, and what tuning steps can be don on 2000 Standard. The "quick fix" of the /3gb switch is not possible. HeapDecommitFreeBlockThreshold has already been set to try and aggregate larger blocks before decommiting. You already played with the short list of options in the tuning guide. What can you do?

In Exchange 2000, each storage group is an instance of ESE running as a child process of store.exe. As each instance needs memory, it dynamically allocates space from the ESE multiheaps that are managed by exchange 2000. The default number of multiheaps allocated by exchange 2000 is four times the number of processors. The more multiheaps you have, the more virtual memory space is committed to store.exe.

How many processors do you have? I can certainly see where a 2 or 4 way box with only a gig of ram could have virtual memory issues very quickly. Have you tried limiting the number of ESE multiheaps? The reg key is:
HKLM\Software\Microsoft\ESE98\Global\OS\Memory
The key is "MP Heap Parallelism" of the type REG_SZ. If the key is not present or the value is not set, the default is four times the number of processors. You may want to try setting this for 0. With the value set to 0, the formula used is #of processors + 3. On a dual processor box, that would be 5 vs 8. On a quad, 7 vs. 16. This might just provide the relief you're looking for.

 

[dmandell]

Actually, I'm starting to have some success with the msExchESEParamCacheSizeMax setting.

Still too early to tell if it will work well in the long term.

Interesting idea.
I will keep in mind.

Dana