Exchange 2003 and Backup Exec Disk Failure - RAID 0 4

[KLewisBPM]

I am very twitchy at the moment, Backup exec has been backing up the exchange environment and on 3 occassions the system has crashed with one of the HDD's blinking orange! as soon as you pull the power and plug back in it boots up fine.

I have a DELL PE2650 with PERC 3/Di inbuilt. There are 5 x 33.9Gb drives in a RAID 0 configuration with total capacity 169.5Gb as drive C:

I cannot believe it is set up like this! (We inherited it)

Only 26gb is being used, it is running Exchange Server and is a Domain Member, DNS Server, Backup Server.

Is there anyway that this RAID 0 conversion can be converted into RAID 5 and retain the data?

I also have a Dell Powervault attached via a PCI PERC 3/d it has 4 73gb HDD's in a RAID 5 configuration. there are still 6 slots left and i have a couple of spare 73gb HDDs.

Perhaps i could make a new Container within the Powervault with these new HDDs and use that as a bootable partition?

Any ideas are Welcome!

Kind regards

KLewisBPM

 

[58sniper]

Well, if you're going to make a change, making it all RAID isn't the best answer either. OS should be on it's own volume. Exchange databases should be on their own RAID 10 volume. Exchange transaction logs should be on their own RAID 1 or RAID 10 volume.

You'd be better off moving those to other volumes. You could then, theoretically, do a disaster recovery rebuild of the server after changing the RAID level for the initial volume (which would now be just the OS volume).

Make sense?

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[mofusjtf]

Pat is right, if you're going to make a change. Make the right changes. Don't forget to keep you Page File on a separate drive too. Especially separte from the Exchange db and logs.

It sounds like your hard drives need a firmware update. I have had experience with some Dell servers and certain hard drives (Fujitsu I think) that will be mismarked failed due to faulty firmware on the hard drive. Just a thought. You certainly don't want to make all those changes and have your drives do the same thing.

 

[pasupport]

Yeah, the Dell firmware can be a vary big probleem, I got the notice about the firnmware problem, about 2 weeks after having to do a full recovery on my email server. Funny little game, if one of your RAID drives goes offline it can take another one with it, read that as a total failure in a RAID 5. The others are correct in their outline of the process, but without the fireware update you will continue to have issues.

 

[xmsre]

Well, first, you shouldn't use RAID 5 for exchange; the write penalty's too high.

Second, as drives get larger there's more and more stress on the remaining drives during a rebuild after a single drive fails. With 500GB drives it's not that uncommon to see a second drive fail during rebuild. So what kind of protection is that?

You could move to more advanced parity schemes like RAID6, but I don't think you'll find that on a Perc controller in a server. For these types of deployment's I'll stick by RAID 10.

 

[KLewisBPM]

Hi all,

Thanks for the responses, the problem is with only one of 5 hardrives they are all indentical drives, so i feel the firmware probably is ok, the issue only occurs when Backup exec is running an exchange backup.

I was thinking of setting up a second server with exchange on it to replicate with the main server so I can keep email running while I reconfigure the main Server.

The main server is a dual 2.2ghz zeon the spare server is a dual 1.4ghz zeon - we only have 40 users

Do you think this would be of benefit?

Many Thanks



KLewisBPM

 

[58sniper]

so i feel the firmware probably is ok

Why? I've gotten brand new servers that had outdated firmware.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[KLewisBPM]

5 Identical Hard drives! No problems for the past 3 years until on day when 1 drive keeps failing at a certain point in the backup of Exchange. In my opinion that would not relate to outdated firmware but a physical problem with the disk, I could be wrong. If there was an issue with firmware I would expect issues with all 5 hard drives.

Thanks for you input!

KLewisBPM

 

[KLewisBPM]

Update - This happens just as it starts to backup the First Storage group.......

here is the log file that is certainly different to a completed backup!!


======================================================================
Job server: EXC-BP-MAR
Job name: Exchange
Job started: 23 September 2008 at 02:02:39
Job type: Backup
Job Log: BEX_EXC-BP-MAR_02173.xml
======================================================================

Drive and media mount requested: 23/09/2008 02:02:40

Drive and media information from media mount: 23/09/2008 02:02:40
Drive Name: CERTANCE 1
Media Label: LTO00006
Media GUID: {1a81639c-237b-4727-8847-8c935d434841}
Overwrite Protected Until: 02/10/2008 23:00:13
Appendable Until: 02/10/2008 23:00:13
Targeted Media Set Name: Daily
======================================================================
Media operation - append.
Hardware compression enabled.
======================================================================
EXC-BP-MAR\\EXC-BP-MAR\Microsoft Exchange Public Folders
Family Name: "Media created 22/09/2008 23:00:02"
Backup of "\\EXC-BP-MAR\Microsoft Exchange Public Folders "
Backup set #4 on storage media #1
Backup set description: "Exchange"
Backup Type: Full - Back up messages - Reset archive bit
Microsoft Exchange Server Agent: Started
Backup started on 23/09/2008 at 02:04:16.
Backup completed on 23/09/2008 at 02:04:49.
Backed up 987 mail messages in 13 folders in 0 mailbox(es)
Processed 39,990,470 bytes in 33 seconds.
Throughput rate: 69.3 MB/min
----------------------------------------------------------------------
\\EXC-BP-MAR\Microsoft Information Store\First Storage Group
Family Name: "Media created 22/09/2008 23:00:02"
Backup of "\\EXC-BP-MAR\Microsoft Information Store\First Storage Group "
Backup set #5 on storage media #1
Backup set description: "Exchange"
Backup Type: Full - Database & Logs (flush committed logs)
Microsoft Exchange Server Agent: Started
Backup started on 23/09/2008 at 02:04:51.

and thats where it crashes!

How important is the First Storage group in the grand scheme of things?

I'm currently running a backup of all my mailboxes as a precaution.

I ordered 6 new 33.9gb hard drives and have created a RAID 5 configuration on a powervault 220s attached to to this server, I hope to image the current live server array to this new partition and make it bootable, if all goes well i can then run the backup and see if it crashes again. If it doesn't then i'll replace the drive that comes up with the blinking orange light during the previous backup crashes. Change the RAID config and then look at the suggestions above to spread the OS and exchange data over the 2 redundant arrays.

I'm gonna look at this tomorrow night whilst the office staff aren't in... If anyone has any further suggestions in the mean time that would be great!!

Cheers











KLewisBPM

 

[58sniper]

First Storage Group is very important.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[KLewisBPM]

Update!

Bad News for me!

I ran the Acronis True Image and within 2 minutes the software reported a read error and then multiple delayed write failures.

I have got the system back up and running after a couple of reboots, so at least we can keep running for now.

But it appears this drive error is gonna keep haunting me until I do a complete rebuild.

I have a spare poweredge 2550 dual 1400 zeon so I can prepare this offline, my only concern is the exchange first storage group that causes the drive to fall over.

I'm gonna do a full system backup now excluding the first storage group... Can anyone see any problems here?

Any alternative suggestions would be greatly received!!

cheers

KLewisBPM

 

[KLewisBPM]

The story so far...

I am still backing up the mailboxes. I am yet to solve the issue with first storage group.

I have built another w2k server and am now looking to replicate our existing exchange server to this new server.

Should I install this exchange install as an additional server? and then move mailboxes to it... with the idea of switching the old one off?

I would like to hear any suggestions on the best way to do this... I understand that in exchange 2007 you can create a backup server!

KLewisBPM

 

[58sniper]

Read MS KB 822931. This will walk you through moving everything from one server to another.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[KLewisBPM]

Hi Guys,

I am still here and the problem is still here.

Since my last post I have done the following! On a Dell 2550 2003 Server I have installed VMWARE, I have created a Virtual Windows 2000 environment and run the exchange 2003 setup ready to try and restore into it as a test run. However I can't seem to get it working properly as a secondary server, i wanted to try and move some old mailboxes on to the virtual server directly. The system manager works but is connected to the original server it doesn't show the new server.

I Dismounted the Mailbox Folders DB and Public Folders DB and copied the entire contents of the Exchsvr\MDBDATA folder including all the 5mb log files that have been accumulating since the backups were stopped!

So the Server did go down again whilst copying! with the same drive failing. I rebooted the server and checked what had been copied to the new ARRAY i previously created.

I have attached a picture that shows the original and new directory's.

The priv1.edb copied fine! Now there a few files missing.. I think the next thing for me to do is to try an copy the missing files one at a time to see if I can actually isolate a single file!

Sorry for the delay in getting back to you guys and thanks for your input!!

KLewisBPM

 

[Zelandakh]

Have you tried backing up the server from another machine? Sounds like you have a hard corruption on disk that is inside the first storage group.

My first thought would be to move the store inside that storage group to another volume to get you out of that hole. My second thought would be to mirror the OS or Acronis or however you want to do it and copy the partition to a fault tolerant drive rather than building another server.

 

[KLewisBPM]

hi zelandakh, please read the whole thread... Acronis, new RAID Array etc its all there fella!!

KLewisBPM

 

[PaulGillespie]

How about exmerging out the mailboxes so that you have a backup in PST format that you can then reimport if it goes poorly?

 

[Zelandakh]

I'd already read the thread. Moving Exchange to the PV will get the databases off the drive with the problem.

Then you can either blow the OS and install Exchange in disasterrecovery mode or mirror the OS to a fault tolerant drive, boot to that and blow your current problem.

 

[KLewisBPM]

Hi Zelandakh

Apologies i misread.. as soon as i try to image the partition it will fall over again.

My ideal solution now would be to Virtualise the Exchange Server, I have the spare hardware to it. However I have installed exchange and it isn't showing a second exchange server so i cant move mailboxes to it?



KLewisBPM

 

[Zelandakh]

OK, ignoring the cloning (which I'd do from DVD not parallel i.e. the server isn't running)...

Did you install the server into the same domain? Did you then install Exchange into the same site?