IIS, SMTP & Exchange Services Stop Responding - HELP

[WinNT511]

Over the past serveral weeks we have noticed that Event ID 7031 is repeatly logged to the system event viewerm with the following details:

Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The IIS Admin Service service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 1 milliseconds: Run the configured recovery program.


Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Microsoft Exchange IMAP4 service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 0 milliseconds: No action.
Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Microsoft Exchange POP3 service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 0 milliseconds: No action.


Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Microsoft Exchange Routing Engine service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 60000 milliseconds: Restart the service.


Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Simple Mail Transport Protocol (SMTP) service terminated unexpectedly. It has done this X time(s). The following corrective action will be taken in 0 milliseconds: No action.

Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The World Wide Web Publishing Service service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 0 milliseconds: No action.

Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Network News Transport Protocol (NNTP) service terminated unexpectedly. It has done this X time(s). The following corrective action will be taken in 0 milliseconds: No action.

THis happens serveral times a day with no particular event triggering these error messages. These services will stop and then restart or at times we have to re-start the services manually. The exchange server is Exchange 2000 with SP3 and all hotfxes applied. The exchange server is installed on Windows 2000 Server SP4 with all patches up to date. The exchange server hosts about 55 e-mail accounts.

I have performed the following to the system:
**Have Ran 2 AV scans (Yielding 0 Infections)
**Have logging enabled on the on both IIS and exchange but the logs have yielded nothing suspioious.
**Installed Service Pack 4 (6-1-05) with no improvment.

The Server also Hosts (DNS, DHCP, and is a Domain Controller). I know that the (MS KB Q316612) Code Red Worm can have a effect on these services, but we have run multiple AV scans with no trace of virus infection.

We need to fix this problem as we plan on adding additional activity to our e-mail system in the next few months.

Please Provide HELP on how to proceed.

Winnt511

 

[Zelandakh]

Search this forum for 7031. I've covered this numerous times.

 

[WinNT511]

I searched the forum for 7031 and got some additional Ideas.

We cleaned out the "Bad Mail" folder thinking there might be coruupted mail folder.

We will also be Checking our Av product "CA Associates to see for any compatiability issues. We have also diabled real time scanning on the exch directories.

Any further Ideas Plese advsie.

Winnt511

 

[Zelandakh]

http://support.microsoft.com/?id=827214

 

[Zelandakh]

Sorry, wrong ref.

try:

Go to the IIS Admin service. Change the properties:

After first fail: Restart service
After second fail: Restart service
After third fail: Run file

Filename c:\iisadmin.bat

Contents of IISadmin.bat:

rd c:\exchsrvr\mailroot\vsi1old /s /q
ren "c:\exchsrvr\mailroot\vsi 1" vsi1old
c:
cd \exchsrvr\mailroot
md "vsi 1"

net start "IIS Admin Service"
net start "Microsoft Exchange IMAP4"
net start "Simple Mail Transport Service (SMTP)"
net start "World Wide Web Publishing Service"

net send emailSVR "IIS Admin service failed and has been restarted automatically, please check the logs

 

[WinNT511]

Well at 2:31 today all the exchange services stopped and it took a 1 hour to get the services re-started.

I disabled realtime montoring on the exchange directory and IIS folders. Mail was stuck in the end users outbox on the client machines and a re-boot finally re-started the services.

I have the SMTP patch 885881 in the above link. I will re-apply the patch (But I think the same result will continue)

There is a AV upgrade to the exchange server which I will apply.

I will also try the batch file.

Any additional suggestions.

 

[zk9575]

Were you able to find a solution to this problem, because i am having the exact same issue.

thanks

 

[WinNT511]

I have re-applied patch MS 05-025 (SMTP_DNS Patch)about 3 weeks ago. No problems reported for about 2 weeks until Thursday 6-16. When a user reported that we could not send a receive e-mails to a particular doamin. We investigated and found out that e-mails for that domain was stuck in the Queue (35 Messages) We forced the connection with no luck.

Rebooted the server and a IIS error message popped up. All exchange services and IIS services did not start. Finally they sstarted after about 30 minutes. Once they started 20 of the 35 messages were pushed out of the Queue. I ahd to do another report to get remaining messages deilvered.

On Friday 6-18 same occurance as outlines above. This time a noticed a 100% CPU usage was constant. I looked in Task Mansger and there was nothing in the services tab that was indicating a 100% CPU Usage. Re-booted the server and messages in the queue were delivered after we forced the connection. This problem affected us from send and recieving e-mail internal and external.

I did upgrade the AV software on the exchange server with no luck. Services had stopped on Monday 6-20 also. The only thing I notice is a CPU spike 95% when mail is in the queue in a 5 minute duration? The only spikes for a minute or 2.

Also E-mail to a particular domain might trigger this behavior?? When we send e-mail to Verizon.net it seems to trigger these issues. ( I am only guessing at this point).

ONly other issues We can do is Re-Build Exchange or un-install and re-install IIS and metabase. Which is 2 tasks I do not want to do if I can help it. WE are going to open a ticket with Microsoft if this happen again. It has been going on for 4-6 weeks.

Any Ideas as I am running out of options ti what might be causing this behavior.

 

[Zelandakh]

Reapply Windows SP4. Restart.
Reapply Exchange SP4. Restart.
Reapply post SP4 hotfixes for Exchange.

Clear out badmail and empty the bin.

Take a good backup of your server.

This worked for me and I've been clean now for about 6 months.

 

[WinNT511]

Called MS Support and opened a case number with them. Apprently there is a issue with Hotfix 893066 which causes the IIS ADdmin services to stop and the cascade effect begins. This is true when this hotfix is installed on a exchange server.

I applied the HOtfix supplied my MS to correct this issue(hopefully it works). Also installed ADPLUS Dump Reports and MPS Reporting tool along with a Dr. Watson Dump.

If this happens again MS will have the information they need to correct is issue.

Also Re-installing SP 3 (Exchanges HOtfixes), SP 4 for windows might help...but I will try MS first.

I will let everyone know if the hotfix corrects the issue.

Stay Tuned.

 

[WinNT511]

Well hotfix did not correct the issue. At 10:15 AM Services crashed and Took 20 Minuites to restart. Sent Dump information to MS for review.

MS Support concluded that "Our Mail Disclaimer program" is causing a .dll conflict with the exchange services crashing.

Here are the release notes for the correction in the current upgrade.

Version 1.6 Release Notes (20/05/2005)

New condition: Message class matches/does not match wildcard string.

Resolved potential problem where attachments were not visible when mail was viewed in Outlook Web Access.

MailDisclaimer now correctly handles HTML messages sent to Exchange by MAPI.

Active Directory lookups now work correctly when MailDisclaimer is running on a domain controller.

A memory leak that could cause Exchange performance problems has now been resolved. (THis is the big issue)

Well I upgraded to Version 1.61 and will review for performance issues. Let hope this solves the problem. If not I will remove the software.

Stay Tuned.

 

[jhoweth]

Hello WinNT511 - I was wondering if you have any update on your status or situation ? We started having the same issue about 6/29/05. Our only fix for now is having the services set to auto restart on failure, which is working well as a band-aid solution, but we'd really like to patch this correctly. We too, are on Win2K SP3 and Exch2K SP3 w/TrendMicro Officescan. We're thinking it might just be time to apply Windows SP4, as I'm sure MS would ask us to if we contacted them. Looking forward to any updated info you may have.

Thanks,
jhoweth

 

[overcast32]

You have Veritas Backup Exec 10?

Think I know what your problem is - did you up date the tape drivers?

If so - It might boil down to this issue

http://seer.support.veritas.com/docs/275924.htm

 

[jhoweth]

overcast32: I don't see how the Veritas article you mentioned is tied to this issue ? The event ID in the log is 7031, which is not mentioned in the Veritas article, as well as it's not the Routing and Remote Access services with any issues. Please correct me if I am wrong. Hope to see WinNT511 turn up here soon with some info.

 

[overcast32]

Well - you're right, it could be a far different problem, but that's how mine started.

One of your errors...

Event Type: Error
Event Source: Service Control Manager
Event ID: 7031
Description: The Microsoft Exchange Routing Engine service terminated unexpectedly. It has done this X time(s).
The following corrective action will be taken in 60000 milliseconds: Restart the service.


The routing engine is shutting itself down, causing SMTP, NNTP and other services to fail. It's caused by some problem with the default tape driver - mine specifically was the Backup Exec 10 trial that came with our tape drive.

Now - what I did was re-apply the Exchange Service packs in an attempt to fix this - since my core issue was exchange shutting down. Which was linked to the SMTP service stopping.

Routing Would not start.
In turn SMTP, NNTP, and a couple others would fail as they are dependant on the routing
Then Exchange services would stop because SMTP wasn't starting.

Yeah, it was a mess.

I suspect - if you don't have veritas backup exec 10 on the same machine, this issue won't apply to you, at all.

When I re-installed the Service pack - the internet information services started failing, and then in the end, I couldn't even log into the console until I disabled the "remote storage" service. Then I could log in. The wonderful fix was to re-install exchange. Needless to say, that sucked - but I managed to get it done before going home that night.

See if thie MS KB sounds like the problem...

http://support.microsoft.com/default.aspx?scid=kb;en-us;842696

It was the event ID number that caught my attention - I may be way wrong on your particular issue, but I felt compelled to mention this - as it's not really evident at first.

 

[WinNT511]

Update:

Applied the Mail Disclaimer update on 6-23 (see 6-23 post) and as of 7-6 services have not crashed or stopped and exchnage seems to be running with no issues.

Except after I applied the update I noticed that the inetinfo.exe service is consuming 500+ MB of RAM which is abnormal behavior. Store.exe normally consumes the majority of RAM on a our exchange server.

I restarted the server last week and Inetinfo.exe consumes 500+ RAM over a 3-5 day span (is this normally) as we have not noticed this behavior before the update was applied. The server still seems to be operating normally with no performance issues.

I have diabled the mail disclaimer program to see if this in deed causing the memory spike. Is this a memory leak or just normal behavior for Inetinfo.exe??

I did call tech support for Mail disclaimer and there own tech support documented that in there beta testing of the current version 1.61 it was documented that the inetinfo.exe would consume about 350 MB RAM and level off which I find unacceptable (Ours is currently at 585 MB).

Any Ideas??

Winnt 511

 

[WinNT511]

No I do not have Veritas 10 on the exchange server.

8.6 agents

 

[WinNT511]

Well I disabled the Maildisclaimer program and the Inetinfo.exe is normal (35 - 50 MB) and stays at this level if I were to enable MailDisclaimer it would climb to 550 MB.

I will be writing a e-mail to softek about this issue as they should correct is issue. The past 2 releases address exchange issues (see Post dated 6-23) and the update causes a memory leak in the inetinfo.exe service and there tech support says this is normal. It was not normal on Version 1.52.

Well I will keep everyone updated to the progress in this issue.

Thanks for all your advise.

WINNT 511

 

[MudMan2000]

This problem started for me about 3 weeks ago, but I THINK it's been fixed (it's been 72 hours now without a shutdown). I re-applied W2K SP4, then I re-applied Exchange SP3. Windows update didn't figure out that I needed to re-apply all the patches I've installed since then, so I used the demo version of that Shavlik.com patch tool to determine which ones I was missing and needed to reinstall. I also cleaned out my badmail folder from exchange (c:\Program Files\Exchsrvr\Mailroot\vsi 1\BadMail) - not sure if that was necessary but there was over 3000 in there so why not). So far so good since then. I'll post again if the problem recurs.

 

[wademan]

I am seeing the same thing; IIS; SMTP; Exchange; POP3 and a few other services are shutting down. Plenty of hard drive space so not sure what is going on. Smells like a virus..

Let me know if you find anything out.