Event ID: 24603, Event Source:CPQCISSE

[sluggoqmagee]

Event Type: Warning
Event Source: CPQCISSE
Event Category: None
Event ID: 24603
Description:
Recovery of logical drive 1 configured on Embedded Array Controller, was aborted while rebuilding physical drive with SCSI ID 2 on SCSI Port 2 due to an unrecoverable read error. The physical drive reporting the error is SCSI ID 0 on SCSI Port 2.

We escalated this to HP/Compaq Support one. This is what they told me. If the server doesn’t have enough quiet time to hot fix a bad sector and a failed drive occurs on another drive “or” during an array rebuild a sector goes bad on another drive, the array will not rebuild and it will abort. The only way to fix this problem is a BMR after replacing the one bad drive and the one with the bad sector.
I would like to get verification that this is indeed the case.

 

[newBPhoneGuy]

sounds like you might have some corrupted data have you tried pulling the new drive and reseating it to allow the rebuild to try again

 

[sluggoqmagee]

Corrupt data would be symptom rather then root cause. The real cause here is bad SCSI Hard Drive hardware. This problem isn’t just an HP/Compaq problem rather a know issue with RAID5 arrays. Although they provide fault tolerance, it’s not without its own set of problems. I manage a fleet of 2000+ servers and this has happen to less then 1% of them but last month we had three in two days.

Here’s a scenero:
The server has four hard drives configured in a distributed data guard. Drive1 failed and drive3 has several bad sectors. The drive1 gets hot swapped and the array begins rebuilding. Sometime either during the first failure or during the rebuild of the array, drive3 logged more bad sectors. Due to the array already being degraded with a failed drive, its ability to correct these errors in the distributed data guard was unavailable and recovery read and writes errors began to increment. The rebuild aborts and send an event to the event viewer.

HP/Compaq's array controllers have the ability to abort the rebuild if this happens. It appears that some Hardware vendors don't. I've seen servers by Dell crash that had this problem. Anyway the only way to recover from this problem is to reload the OS onto a new array with a couple of new drives and restore all the data from tape.