RAID 5, drive failed, replaced drive is Ready, how to get it Online ?

[FredWagner]

Production server, on of the volumes with 10 drives in RAID 5, one drive went defunct. Used RaidServe, clicked on Remove defunct drive, then pulled it. Put a new drive in the slot, let the sysetm scan for it. Now the new drive shows Ready, but I'm at a loss on having the array rebuild and change it to Online.
I can make it a Hot spare, but that doesn't put it into the online status. I deleted it as Hot Spare, now it shows Ready again. Suggestions, please!

Fred Wagner

 

[FredWagner]

I talked it over with the server team (I'm Sysadmin on the FileNet CS system that this server runs), and we think there were two Hot Spare drives on this server before the failure, and that one of them automatically took over for the failed drive. We've made the drive I ust installed a Hot Spare, so it will also get used next time we get a failure. The server was installed in February 2001, with a total of 30 drives (3 arrays), so the MTBF's are coming due more often now.

Fred Wagner

 

[goombawaho]

You are 100% correct - you're do for more drive failutres. I'd keep 2 hot spares in that machine to keep you covered.

It's really time to replace that system if you were conservative (and had the money). That's a long life for a business system to remain.

 

[FredWagner]

We know we're due to replace the server - it's the oldest one in our data center - we do have it under service contract, and a good supply of drives. There are two hot spares in the array, we'll keep it that way, and it's being monitored several times a day.
I have a newer test server, with SAN drives, and I've warned the folks that I'll need those increased to be able to handle a restore of the backups from the old server - which I'm planning to do when we do get the replacement server hardware and start a migration to a new platform. The test server has a gigabit NIC, the old one has a single 100BaseT NIC - the full backup takes 22 hours! There are actually two other NICs installed, but back in 2001 they couldn't figure out how to make them loadshare for additional bandwidth. other than the long backup time, it works remarkably well. We've split the backup into two jobs, one 8 hours, the other 14, so the libraries aren't offline as long.
The migration to the new platform will really be fun!

Fred Wagner

 

[goombawaho]

Sounds like you have it under control. Lots of people just run their stuff forever with the management attitude of "we'll replace it when it breaks".

That's a rough few days when that event finally occurs.

 

[FredWagner]

We had a dry run for 'that day' back in April - we'd gotten complacent - over a year since the last drive failure - I was off site at a conference for the system that runs on the server - got a call from a user who loads data onto to - couldn't connect. Turned out that a volume that runs the SQL had lost TWO drives in its RAID 5 volume, and couldn't recover. I didn't get to see it before the Server group replaced drives, so I don't know if there had been hot spares that had been used, how many were defunct, etc. Anyway, we had to restore that volume from the most recent full backup - only a day's worth of data capture were lost.
What was a good drill though - our Backup Exec Admin, when asked to restore that drive, did so. But the sytem didn't work. We got with him, and discovered that the SQL on that drive is backed up separately - we hadn't asked him to restore that, so he hadn't. Once he restored the SQL backup also, we had a system as good as the last full backup, with the lost day's data files lost in the encryption on the data drive. We keep a good supply of hot spares in the rack now, and monitor it pretty closely. I'll feel a lot better once we're on the new server, but since that will include a migration to a new platform, that will be fun in itself!

Fred Wagner