Exchange 2003 and Backup Exec Disk Failure - RAID 0 4

[KLewisBPM]

I am very twitchy at the moment, Backup exec has been backing up the exchange environment and on 3 occassions the system has crashed with one of the HDD's blinking orange! as soon as you pull the power and plug back in it boots up fine.

I have a DELL PE2650 with PERC 3/Di inbuilt. There are 5 x 33.9Gb drives in a RAID 0 configuration with total capacity 169.5Gb as drive C:

I cannot believe it is set up like this! (We inherited it)

Only 26gb is being used, it is running Exchange Server and is a Domain Member, DNS Server, Backup Server.

Is there anyway that this RAID 0 conversion can be converted into RAID 5 and retain the data?

I also have a Dell Powervault attached via a PCI PERC 3/d it has 4 73gb HDD's in a RAID 5 configuration. there are still 6 slots left and i have a couple of spare 73gb HDDs.

Perhaps i could make a new Container within the Powervault with these new HDDs and use that as a bootable partition?

Any ideas are Welcome!

Kind regards

KLewisBPM

 

[KLewisBPM]

xmsre/Paul

Thanks for your responses. This has been a real mind bender!

I totally agree... My main priority is to make sure that the Exchange data is safe..

If I can run a disk check and block out the bad sectors after moving the edb/stm etc I could choose to image the RAID 0 and move the whole image to a proper Redundant array..

Or even a vmware image one day!





KLewisBPM

 

[Zelandakh]

Great idea, wish I'd suggested that.

 

[KLewisBPM]

I don't believe it, I was all ready for moving some mailboxes to the new storage group I created yesterday!

I went to create a mailbox store and i got the error message "This Storage group already contains the maximum number of stores allowed"

Why didn't it throw that error up when I created the Second Storage Group! lol typical... I thought this was going too well.


KLewisBPM

 

[KLewisBPM]

I'm just waiting for my suppliers licensing department to get back to me on upgrading from standard to enterprise.

I understand its a straight forward operation to perform the upgrade. does anyone know if the upgrade looks at changing the edb or stm files? Just a bit twitchy if it does anything to stm for obvious reasons.

cheers

KLewisBPM

 

[PaulGillespie]

I would have thought so. I wouldn't be upgrading without a backup first personally.

 

[PaulGillespie]

Do your users use outlook in cached mode? If they do, you could go round each one, take them offline and export their mailbox to a pst file.

I wouldn't normally recommend this but it seems like you're running out of options.

 

[KLewisBPM]

Hi Paul... Regarding the backup before upgrade, I would love to be able to backup the server lol... its like a vicious circle.

Yeah that was another option. all users are in cached mode which has been in the back of my mind for a while. if it really goes Pete Tong then maybe we start a fresh server with brand new mailboxes. There is some software out there i believe to be called C2C that will hoover up pst files from your network and archive/compress them down and make them accessible through your exchange mailbox.

I'm gonna bang my head on a brick wall and see and re-read this thread all the way through!

cheers

KLewisBPM

 

[KLewisBPM]

To top it all off the drive has just completely failed! Server down....

KLewisBPM

 

[PaulGillespie]

Oh well, at least your options are now very limited and you can have a clear objective. Good news is that you have the bulk of the emails in OST files.

Good luck with it!

 

[KLewisBPM]

Thanks paul... I have just got our Common drive and user files up and running again on a different server so now to the server rebuild... At least the users can keep working!! only email is affected now!

i'll keep ya posted!

KLewisBPM

 

[Zelandakh]

Hope things are getting there and that once you are sorted you can let us know the outcome.

 

[danielwestendorf]

If you end up dragging any contents of OST files into your new Exchange mailboxes via Outlook, make sure you give them a few minutes, even if it says the action is 100% done.

I had a new client who the previous admin never made a backup, and the server blew up. I ended up building a new one and copying all the users emails in via Outlook and OST. It wasn't too bad as it was only about 50 users. I did run into a problem with larger mailboxes only importing ~70% of the mailbox contents even though Outlook said it was complete. I found that waiting an additional 5-10 minutes before closing Outlook resulted in all the contents being copied.


Good luck!

 

[KLewisBPM]

Hi all.

Wow what a long haul..

The server went down at 11:45 GMT on Tuesday and although the user files and common drive were up and running again quite quickly I ended up with a double whammy, when reconfiguring the Power Vault so I could remove the spare drives on their own array it took the configuration of the data Array down and so I ended up with no Exchange and no Data files!!

It took me 12 hours to get to the stage where I could actually put the data tape in to start the recovery. One being I couldnt boot from the cdrom, eventually i replaced it with one from another server! Rebuild server, install windows 2000, put the service packs on and install symantec backup exec and check the data tapes. With no warning i discovered that the most recent backup was from the 31st October, the backup tapes had been ejecting each day with no warning of failure last week. Windows and Data was the first stage of the reinstall which was done under normal operating conditions. The second was to install the system state this had to be done by loading up in directory services restore safe mode. once the system state was done another reboot and ntdsutil authoritative database restore, reboot again and I could now see Active Directory! This process was completed by about 2:00am GMT.

Before performing the Exchange Disaster recovery I did some server tests to ensure the other 2 DCs were happy to replicate with the recovered server and to my horror they were not! I spent the next 6 hours trying to get the replication working and had no joy! I spoke to the MD at 8:00am GMT and we decided to get a MS guru in to take a look. They arrived at 10:00am GMT and we had a long discussion about all the steps I had performed and we eventually decided to run the System state and NTDSUTIL Authoritative database restore once again. as other actions I had tried had started causing DNS issues etc. System state and NTDSUTIL were run and ready again by 12:00 GMT. The next 4 hours was spent getting the server accepted by the other DC's and the main problem from before was not giving the recovered DC time to replicate fully!! The MS guru said it would be better to not have the exchange server as a DC so we demoted it (dcpromo). 16:00 GMT time to install Exchange.. All the users now had access to data files so we were half way there! The MS Guru said that an Exchange Disaster recovery was not an option because we didnt have the first storage group! so a standard install was performed and it actually integrated quite well it also kept some of the settings which could have only come from the server recovery!! 17:00 GMT we did a restore of one mailbox! Mine! I went home at 17:45 GMT I had been at work for well over 33 hours!!! the mailbox restored ok. We left running a full restore of all mailboxes and public folders

I went home got into bed and slept all the way till 8:00am GMT Thursday!

Knowing that the mailbox was from October 31st we decided to archive the existing email stored on the laptop to a PST file. We then created a new profile and new connection to the server uncached and that worked! The MS Guru popped in for an hour to make sure it was all ok and then left. I moved my emails from 01st November to 11th November out of the pst back up to the server. Then tried to reconnect the mailbox in cached mode but it didn't work. I did a bit of digging and found a cool trick, start outlook with the switch /resetfolders and voila Outlook started caching and all is now well! I then spent the rest of today going round doing the same for all the users. I am abut half way through and will continue tomorrow!!

The Verdict! You can make a successful recovery from a backup of just the mailboxes, you don't need the The First Storage Group or original edb or STM files.

This was a bit a of special case, this post was all about the issues with not being able to back up the first storage group or even doing a file level backup on the STM file.

Thanks for all of your inputs! If you want to know anything more technical of the steps taken give me a shout! i'll try my best to remember!!

Remember to always check your backups!! We were lucky to only lose a weeks worth of data!

I have been looking at VMWARE recently and i think this going to be something I am going to get serious about!

KLewisBPM

 

[Zelandakh]

Excellent closure - thanks for that.

Suggest you start checking the backups are working - try to get an email alert on completion, successful or not. After a couple of weeks of successes, turn it to only email you on failures to keep your inbox clean.

You've learned a lot of lessons in this as you have detailed but there are a few for management to learn...computers go wrong, IT need overtime and just expecting it to all work seamlessly just won't work for ever.

 

[PaulGillespie]

Nice one. Glad it's all back up, hopefully you'll come out the hero and not the villain!

Only had one all nighter this year myself....the year's not finished yet though!

 

[KLewisBPM]

I took a well deserved day off yesterday! so feel quite refreshed!

IT is always under valued and under budgeted. It takes something like this to wake people up! When a company is not making money then you really have to try and work with what you have and be the firefighter when things go wrong.

That said I'm still going to look at getting extra budget for the coming year!

KLewisBPM