Event ID 200, 116's

[houghl]

We have started to get these in event log as the IS does its online defrag, and other similar errors when the nightly online backup tries to run.

Evenmt ID 200
Source ESE97
Category Database Page Cache

MSExchangeIS (395) A read of the database file e:\exchsrvr\mdbdata\PRIV.EDB between offsets 0xFFFFFFFFECD75000 and 0xFFFFFFFFECD75FFF failed after 16 failed read attempts with error -1018. There is a software or hardware problem affecting the database drive that must be corrected to preserve database integrity.

Followed by around 12

Evenmt ID 116
Source ESE97
Category Database Page Cache

MSExchangeIS (395) Synchronous read page checksum error -1018 ((1:13553012 1:317647) (0-324106444) (0-831617155)) occurred. Please restore the databases from a previous backup.

After looking around found a document which tells you how to pin point the error but as you can see the offsets are strange ie 8 preceeding F's instead of the normal 10 0's, and so when the calculations are performed it a nonsenceical number.

After consulatation with MicroSoft they said we should run ISINTEG to fix the errors, we are running this on a box to which we have restored an offline backup from lastnight. (we get the same error as the defrag occurs, so we think the hardware is ok on the original box). e ran it the first time and it said we had around 5000 fixes, 120 warnings, and 0 errors. We spoke back with microsoft and they said that we may need ISINTEG 3 times for it to fix our problem any experiences with having to ruun it 3 times or are they just kidding us along? It took 18 hours to run last time this was on the slow recovery box but still quite along time the is is 59Gb.

Any other ideas or suggestions would be great thanks.

 

[zbnet]

1018 errprs are almost always indicative of underlying hardware errors - is the firmware on your array controllers up to date? Search the MS KB for 1018 errors, there is a specific article on diagnosing the cause of these faults.

It is true that multiple passes of ISINTEG may be needed to fully resolve all errors - received wisdom is always to repeat until a pass completes with no errors.

If you still have log files, a restore of a last known good backup of your database is often a prefered way (and a faster way) of correcting database corruption than ISINTEG is. DOn't just 'fix' the database of the 1018s, you need to find and solve the cause of these or else they will just come back again in the future.

 

[houghl]

Thanks for the reply, again (you replied to another one of my posts)

Yes we thought the same but we have restored the offline IS to a different server all together different RAM, RAID, everything.

The artical you talk about gives a way to pinpoint the error but the calculations thta you perform to pinpotint the problem Mailbox, Attachment or Deleted item does not functions because of the rogue offset calue being wrong as in 0xFFFFFFFFF which pushes the calculations into the terabytes area

I shall check on the RAID controller to make sure there is nothing weird to do with files of around 60Gb's incase as you say the firmware doesnt support this sizing.

Thanks for your reply again

 

[houghl]

Found this article listing the Jet errors

http://support.microsoft.com/default.aspx?scid=kb;en-us;177761

-1018 0xFFFFFC06
JET_errReadVerifyFailure ReadVerification error 4294966278

I have now ran ISINTEG 3 times it found various things and fixed them im now goin to start the IS up again if i still get the errors when defraging im moving on to look at eseutil commands.

I have taken a copy of the exchange folders so i can roll back if needed.

 

[houghl]

Hi zbnet is this the article oyu talk about if so look at the calculation and try using the event error i got and as you see the number it talks of is like 47terabytes.......

Maybe i should tell my 8,000 users to get pens and paper out for monday and chuck the exchnage server in the bin. Havent quite got to that stage yet........

 

[houghl]

ooooppps forgot the article link dooooh

http://support.microsoft.com/default.aspx?scid=kb;en-us;262196

 

[zbnet]

No, it was this one:

http://support.microsoft.com/default.aspx?scid=kb;en-us;314917&Product=ech

You should take the warning seriously, 1018s are like the first drops of rain before a massive thunderstorm.

 

[houghl]

Just an update as to where im at

Ran isinteg 3 times found and made lots of changes but found zero errors, tried a offline defrag which failed with read errors, ran eseutil /p which failed with read errors.

My next plan is to put another exchange server into the org and move mailboxes across i am going to do this in the test enviroment first off then if its successful do this in the live.

I may aswell move to exchange 2000 at the same time thus "killing two birds with one stone" so to speak.

 

[cygnetrower1]

Did introducing a new Exchange server and moving the mailboxes resolve the problem ?

I am in the same situation
I ran ESEutil which worked but removed loads of data from my own mailbox so god knows what it's done to ezeryone elses. I'm now planning to move back to the original info store that I have saved on a different server.

Won't moving the mail boxes accross just move the same problem accross ?

Any help will be really appreciated

 

[houghl]

Here is where im at now......

I installed a new server into the exchange test site and moved all the mailboxes across to it (including hidden ones). All of the mailboxes moved accross without error even though it did take 6 days (8,000 mailboxes and 59Gb of data) plus its on our crappy test servers. Once all this had moved i ran the eseutil tests on the new box and it all passed, offline and online defrags now also work error free. I am going down to emc2 tomorrow to look at a SAN setup for our exchange system and so may wait until we get this installed before doin gthe moves in the real enviroment, as like i have said we will need to do this when i move to 2003 anyhow. Within the IS i need to check that there is absolutley nothing else remaining within in it, i have done a mailbox resources check and it is empty, so i could stop the IS delete it along with its log files and then restart the IS service to generate a new virgin IS then move all the mailboxes back pheeeew.

With all the mailboxes moved i stilll get the original errors reported with the live IS, i am going to look at some 3rd party IS maintenance products in the meantime to see if i can pinpoint the error.

It may still be pointing to a deleted mailbox or message or as i am suspecting to a purely fictiotious address which simply does not exist and occured due to some disk corruption at some stage.

PS cygnettrower1 if your IS is ok now but it found loads of errors then i would think you blow away the existing IS and replace it with your new clean one. But you say it has removed lots of mails which points i would think to a true corruption within the IS therefore i think you need to restore from the previous nights backup to your corruption and run eseutil on that to see when the corruption occured. Then you could replay log files to regenerate missing messages. Because i dont think that you can run the risk of having various unknown people having missing messages around your workplace

OI fim not making sense then post again and ill try to type more clearly

Cheers

 

[Alshrim]

did you get a resolution to this yet?? I'd like to see how things went..

Cheers

Alshrim
System Administrator
MCSE, MCP+Internet

 

[houghl]

Havent resolved the issue well not in the real world

I looked at data recovery products but they are basically to try to repair damaged messages within a live IS such as the tools from Ontrack.

We are now in the process of purchasing a new Email server setup, and so as i said in a previous post we will be moving to Exchange 2003 and so it would be pointless moving all the mailboxes in before the new system is in as this needs to be done anyhow.

The new setup is as follows 3 mail servers (active-active-passive) Windows 2003, EMC2 CX500 with 20*73Gb disks, 28 port FS's, 6 HBA's (2 in each server pointing to one FS), 2*LTO 2 drives in robotic library, 1* NAS box (We are looking to move user home folders onto this box)

Software on this setup is Exchnge 2003, Navasphere SAN software, SNAPview, MIRRORview, We will have 3 IS's on the 2 servers splitting the load of users.

The setup is going to be around £150K all in wowsas

But this will give us 2 things a resiliant fault tolerant Email server, with quicker recovery times + a SAN to begin migration from Fileshare servers.