Help FC4500 problem

[HoundofCulaine]

We have a few FC4500's on site, inherited from a previous hardware buyer. And in the most we leave them alone and they don't bother us.

Unfortunately last week one of the FC4500 had a problem and basically the two servers who use it for data storage can no longer see some of the RAID arrays. One typically would be an Exchange server used by about 300 staff.

The FC4500 itself has 2 enclosures each with 10 36GB disks.

The enclosures are setup with two RAID 5 arrays of 5 disks each in the bottom enclosure, with a RAID 5 set of 5 disks and 4 disks and a hot spare in the top enclosure.

To start the first thing that happened was one of the servers was found with the blue screen of death. Don't ask me what it said, the guy who found it didn't write it down and simply rebooted the server. DOH!

When I came in I could see that some of the drives for each server on the FC4500 were no longer visable.

When attaching the dumb terminal to the serial port all drives appear to be setup correctly, although I do note that each of the RAID arrays that are still visable by the server show as ENA where as the Arrays we can't see show as RDY.

One of the confusing parts is that orignally we thought the two 5 disk RAID 5 sets we could see from the servers where the 10 disks in the bottom enclosure, but it turns out that the two RAID sets are actually the first five disks of each enclosure. therefore meaning the second five disks of each enclosure are the ones we can't see??????

We've had a couple of hardware engineers in and they've done just about everything we could think of to sort the problem (well everything somebody who isn't a guru on this hardware could think of). We've swapped every cable, checked all components, swapped Sp's, even got some guidance from somebody who claimed a procudure using the F5 on bootup of the servers would solve our issue.

 

[HoundofCulaine]

Sorry completely forgot to say.

In Navisphere Supervisor the drive we can't see come up as unowned LUNs under the Storage tab.

And when you look at them the SP is set to default to one or the other sp's but in current owner it shows N\A.

also of note is that if you look at the details for SPA you can see what appears to be the correct data (eg model number, firmware, prom and enclosure cabling order) but on SPB these details are blank. And even after swapping SP's physically and just by changing the dial at the rear, SPB always shows as blank.

Also we'd prefer not to have to unbind and rebind and arrays as I can't vouch for the integrity of the backups, as the department who now covers this isn't relable (never had a problem when our team did it). We once paid for a EMC engineer to fix this before, and he did some sort of non destructive fix that seemed to take just a few minutes. Typically nobody at the time asked him to write down what he did, or even seem to pay much attention to what he did. If we have to get him in again I'll make sure he writes something down and watch him like a hawk.

Any (non destructive) help would be much appreciated.

 

[HoundofCulaine]

Also just noted its a 4400 not a 4500.

 

[maultier]

It sounds as if you have a 'dirty cache' situation. When disks go 'unowned' that is usually the cause. The dirty cache flag can be cleared with most of the data being recoverable but data integrity cannot be guaranteed. The recommended course of action is to unbind/rebind and reload.
You can get EMC to come in again and clear the flag. It is done from fcli and requires a special password to clear the flag. They may be able to walk you thru the procedure over the phone to save the cost of a site visit. As for the sp, if you are connected to the serial port, the only sp details you see are for the sp you are connected to. You must move the connector to the other sp to see the details for that sp.

 

[HoundofCulaine]

Ah we've been connecting to the system using Navisphere using one of the servers (the Exchange machine has been switched off), so could it be as this server will be connected to the SP by fibre that is why we get these blank reading on SPB?

 

[maultier]

If you don't have a direct, available path to the sp you want to query, you will not see details.

 

[maultier]

Just a side note: The dirty cache flag is usually set when the Clariion is either powered off incorrectly or power is lost to the clariion and the Standby Power Supplies are both faulty. The proper way of shutting down the Clariion is to bring down the processors accessing the clariion and then set the SPS power switches to the off position and wait until the DPE turns off ( about two minutes to allow cache to dump to the vault disks). There is no need to power down the DAE's.