SunFire v440 LSI3100 raidctl

[csgonan]

I have a Sunfire v440 and need to build mirrored drives. I created one volume from the root disk

(raidctl -c c1t0d0 c1t2d0)

but when I tried to create the other it failed. I learned it was because it has the LSI3100 SCSI HBA. I execute

/usr/platform/sun4u/sbin/prtdiag -v |more

which shows 2 entries for the LSI3100, but do they reference the same device? Is there a way that I can create another volume with this configuration?

THANK YOU

Type MHz Status Path Model
------ ---- ---------- ---------------------------- --------------------
pci 66 MB scsi-pci1000,30 (scsi-2) LSI,1030
okay /pci@1f,700000/scsi@2

pci 66 MB scsi-pci1000,30 (scsi-2) LSI,1030
okay /pci@1f,700000/scsi

 

[csgonan]

I followed a suggestion and did this

ok> setenv auto-boot? false
ok> reset-all
ok> probe-scsi-all

and the errors went away.

Afterwards when I switched it to

ok> setenv auto-boot? true
ok> reset-all

The system rebooted to disk0.

When I try to boot off the mirrored drive (disk2) it says "cannot open boot device". Is there more that has to be done?

Thanks

 

[csgonan]

I just tried to boot the other drives and I got the message "the file just loaded does not appear to be executable". When I try and boot disk2 (the mirror) I get "cannot open boot device". I am wondering since the live boot drive has not failed, whether I have to break the mirror and mount disk 2 to be able to mount it.

Any thoughts?

 

[csgonan]

I deleted the mirror and then I was able to boot off the second disk. There were some boot errors but nothing that stopped it.

Thanks for the help.

 

[maximusGeek]

I use Sun Volume Manager to set up mirrored disks. I can post instructions if your interested.

 

[csgonan]

Thank you very much, that would be nice. I have 4 drives and want to mirror disk 0 to disk 2 disk 1 to disk 3. I did that with raidctl but I had to break the mirror go do it. Then I could boot to the mirrored drive.

Any information would be greatly appreciated.

 

[maximusGeek]

Installation

Packages required:
SUNWmdu (Commands)
SUNWmdr (Drivers)
SUNWmdx (64-Bit Drivers)
SUNWmdg (Optional GUI)
SUNWmdnr (Optional log daemon config)
SUNWmdnu (Optional log daemon)
Once installed, the system must be rebooted and reconfigured to continue.
Environment

Mirroring requires 2 drives that are identically partitioned. A minimum of once slice (4MB) must be reserved for the Database State Replicas or metadbs. (When jumpstarting a system with partitions pre-configured, do not label the second disk) It was found that the optimal configuration was a 3 drive boot disk mirror. Both examples are shown in this document

Database State Replicas

Database State Replicas are the repositories of information on the state and configuration of each meta-device.
There must be a minimum of 3 replicas
50% must be present for Sun VM to operate
51% must be present to reboot
3 drive configuration, 1 replica per drive
2 drive configuration, 2 replicas per drive (avoid system panic)
Create replicas with defined disk slices:

The -a and -f options create the initial replica. The -a option attaches a new replica to the existing DB. Create the replica on each of the reserved disk slices, on each disk.

2 Drive Configuration:

# metadb -a -f /dev/dsk/c1t0d0s3
# metadb -a /dev/dsk/c1t0d0s6
# metadb -a /dev/dsk/c1t1d0s3
# metadb -a /dev/dsk/c1t1d0s6

3Drive Configuration:

# metadb -a -f /dev/dsk/c1t0d0s3
# metadb -a /dev/dsk/c1t1d0s3
# metadb -a /dev/dsk/c1t2d0s3

Initializing Sub-mirrors and Creating Meta-devices

Each mirrored meta-device contains 2 or more sub-mirrors. The meta-device is what is mounted by the OS instead of the logical device.

Creating two sub-mirrors for / and 1-way mirror between the meta-device and the first sub-mirror

2 Drive Configuration:

# metainit -f d10 1 1 c1t0d0s0
d10: Concat/Stripe is setup
# metainit -f d20 1 1 c1t1d0s0
d20: Concat/Stripe is setup
# metainit d0 -m d10
d0: Mirror is setup

3 Drive Configuration:

# metainit -f d10 1 1 c1t0d0s0
d10: Concat/Stripe is setup
# metainit -f d20 1 1 c1t1d0s0
d20: Concat/Stripe is setup
# metainit -f d30 1 1 c1t2d0s0
d30: Concat/Stripe is setup
# metainit d0 -m d10
d0: Mirror is setup

Running the metaroot command sets up the /etc/vfstab and /etc/system file. This step must be run or the system will not boot!

# metaroot d0

Create sub-mirrors for the rest of file-systems (swap, var, etc.)

2 Drive Configuration:

# metainit -f d11 1 1 c1t0d0s1
d11: Concat/Stripe is setup
# metainit -f d21 1 1 c1t1d0s1
d21: Concat/Stripe is setup
# metainit d1 -m d11
d1: Mirror is setup

# metainit -f d14 1 1 c1t0d0s4
d14: Concat/Stripe is setup
# metainit -f d24 1 1 c1t1d0s4
d24: Concat/Stripe is setup
# metainit d4 -m d14
d4: Mirror is setup

# metainit -f d15 1 1 c1t0d0s5
d14: Concat/Stripe is setup
# metainit -f d25 1 1 c1t1d0s5
d24: Concat/Stripe is setup
# metainit d5 -m d15
d5: Mirror is setup

# metainit -f d17 1 1 c1t0d0s7
d16: Concat/Stripe is setup
# metainit -f d27 1 1 c1t1d0s7
d26: Concat/Stripe is setup
# metainit d7 -m d17
d7: Mirror is setup

3 Drive Configuration:

# metainit -f d11 1 1 c1t0d0s1
d11: Concat/Stripe is setup
# metainit -f d21 1 1 c1t1d0s1
d21: Concat/Stripe is setup
# metainit -f d31 1 1 c1t2d0s1
d31: Concat/Stripe is setup
# metainit d1 -m d11
d1: Mirror is setup

# metainit -f d14 1 1 c1t0d0s4
d14: Concat/Stripe is setup
# metainit -f d24 1 1 c1t1d0s4
d24: Concat/Stripe is setup
# metainit -f d34 1 1 c1t2d0s4
d34: Concat/Stripe is setup
# metainit d4 -m d14
d4: Mirror is setup

# metainit -f d15 1 1 c1t0d0s5
d15: Concat/Stripe is setup
# metainit -f d25 1 1 c1t1d0s5
d25: Concat/Stripe is setup
# metainit -f d35 1 1 c1t2d0s5
d35: Concat/Stripe is setup
# metainit d5 -m d15
d5: Mirror is setup

# metainit -f d16 1 1 c1t0d0s6
d16: Concat/Stripe is setup
# metainit -f d26 1 1 c1t1d0s6
d26: Concat/Stripe is setup
# metainit -f d36 1 1 c1t2d0s6
d36: Concat/Stripe is setup
# metainit d6 -m d16
d6: Mirror is setup

# metainit -f d17 1 1 c1t0d0s7
d17: Concat/Stripe is setup
# metainit -f d27 1 1 c1t1d0s7
d27: Concat/Stripe is setup
# metainit -f d37 1 1 c1t2d0s7
d37: Concat/Stripe is setup
# metainit d7 -m d17
d7: Mirror is setup

Update the /etc/vfstab

The / partition has all ready been updated by the metaroot command. The rest of the partitions must be manually configured in the /etc/vfstab.

2 Disk Configuration:

Before:

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s4 /dev/rdsk/c1t0d0s4 /var ufs 1 no -
/dev/dsk/c1t0d0s7 /dev/rdsk/c1t0d0s7 /export/home ufs 2 yes -
/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /oracle ufs 2 yes -
swap - /tmp tmpfs - yes -

After:

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no -
/dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 1 no -
/dev/md/dsk/d7 /dev/md/rdsk/d7 /export/home ufs 2 yes nousid
/dev/md/dsk/d5 /dev/md/rdsk/d5 /oracle ufs 2 yes -
swap - /tmp tmpfs - yes -

3 Disk Configuration:

Before:

#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no -
/dev/dsk/c1t0d0s4 /dev/rdsk/c1t0d0s4 /var ufs 1 no -
/dev/dsk/c1t0d0s7 /dev/rdsk/c1t0d0s7 /export/home ufs 2 yes nosuid
/dev/dsk/c1t3d0s0 /dev/rdsk/c1t3d0s0 /ora_backup02 ufs 2 yes nosuid
/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /oracle ufs 2 yes -
/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /webserver ufs 2 yes nosuid
swap - /tmp tmpfs - yes -

After:

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no -
/dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 1 no -
/dev/md/dsk/d7 /dev/md/rdsk/d7 /export/home ufs 2 yes nosuid
/dev/md/dsk/d5 /dev/md/rdsk/d5 /oracle ufs 2 yes -
/dev/md/dsk/d6 /dev/md/rdsk/d6 /webserver ufs 2 yes nosuid
/dev/dsk/c1t3d0s0 /dev/rdsk/c1t3d0s0 /ora_backup02 ufs 2 yes nosuid
swap - /tmp tmpfs - yes -

System Reboot

Now the system must be rebooted so it can run off the new meta-devices.

Attaching the Mirrors

The second half of the mirrors now must be attached. Once done, the mirrors will automatically begin synchronization.

2 Disk Configuration:

# metattach d0 d20
d0: submirror d20 is attached
# metattach d1 d21
d1: submirror d21 is attached
# metattach d4 d24
d4: submirror d24 is attached
# metattach d5 d25
d5: submirror d25 is attached
# metattach d7 d27
d7: submirror d27 is attached

3 Disk Configuration:

# metattach d0 d20
d0: submirror d20 is attached
# metattach d1 d21
d1: submirror d21 is attached
# metattach d4 d24
d4: submirror d24 is attached
# metattach d5 d25
d5: submirror d25 is attached
# metattach d6 d26
d6: submirror d26 is attached
# metattach d7 d27
d7: submirror d27 is attached

# metattach d0 d30
d0: submirror d30 is attached
# metattach d1 d31
d1: submirror d31 is attached
# metattach d4 d34
d4: submirror d24 is attached
# metattach d5 d35
d5: submirror d35 is attached
# metattach d6 d36
d6: submirror d36 is attached
# metattach d7 d37
d7: submirror d37 is attached


Status of the meta-devices is available via the metastat command. You should allow the meta-devices to finish syncing before you attempted to reboot the system again.

2 Disk Configuration:

# metastat d0
d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Resyncing
Resync in progress: 1 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 30731520 blocks (14 GB)

d10: Submirror of d0
State: Okay
Size: 30731520 blocks (14 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes


d20: Submirror of d0
State: Resyncing
Size: 30731520 blocks (14 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@w500000e012791fa0
c1t1d0 Yes id1,ssd@w500000e0127da6f0

3 Disk Configuration:

# metastat d0
d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Resyncing
Submirror 2: d30
State: Resyncing
Resync in progress: 3 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 24585216 blocks (11 GB)

d10: Submirror of d0
State: Okay
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes


d20: Submirror of d0
State: Resyncing
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


d30: Submirror of d0
State: Resyncing
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t2d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@w500000e012a19340
c1t1d0 Yes id1,ssd@w500000e012a19430
c1t2d0 Yes id1,ssd@w500000e012a215f0

Alternate Boot Device Configuration

In the event of a primary boot device failure, you will want to know the path the secondary boot device as well as set up an OBP alias to boot off the secondary device if the primary fails.

This configuration will last past the next system reset as it uses the system default boot devices for boot-disk.

2 Disk Configuration:

Ok setenv boot-device disk disk1

3 Disk Configuration:

Ok setenv boot-device disk disk1 disk2

This is an alternative configuration but note that it does not last past the next system reset.

Find the physical device path to the secondary boot drive

# ls -l /dev/dsk/c1t1d0s0
lrwxrwxrwx 1 root root 70 Mar 8 17:31 /dev/dsk/c1t1d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0127da6f1,0:a

Take the device path after /devices:

/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0127da6f1,0:a

Edit the string to change the major name (ssd in this case) to disk:

/pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e0127da6f1,0:a

Create alias:

Ok nvalias altdisk /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e0127da6f1,0:a

Ok nvstore
Define boot-device to try the altdisk as a secondary boot device:

Ok setenv boot-device disk altdisk
Ok nvstore

If changes are needed, use the devalias command

Ok devalias altboot /pci@8,600000/SUNW.... /disk@w500000e0127da6f1,0:a

Test the altboot disk by booting the system:

Ok boot altdisk


Boot Time Warnings

These errors show up for each device type that are not being used. These errors can be disregarded.

WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed


Notes:

spbuse00:

Mirror disk1: pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e0127da6f1,0:a
Mirror disk2:

spbuse01

Mirror disk1: /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e012a19431,0:a
Mirror disk2:


Failure Scenario

In the event of a disk failure these procedures should be followed to replace the failed disk. This is an example of disk c1t0d0 failing and the system successfully booting from the other mirror. Note that this process is IO intensive so system load of live systems should be considered.

1)Physically replace disk. Note that the state of d0 is “Needs maintenance”

2)Verify mirror status
# metastat d0
d0: Mirror
Submirror 0: d10
State: Needs maintenance
Submirror 1: d20
State: Okay
Submirror 2: d30
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 24585216 blocks (11 GB)

d10: Submirror of d0
State: Needs maintenance
Invoke: metareplace d0 c1t0d0s0 <new device>
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Maintenance Yes


d20: Submirror of d0
State: Okay
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


d30: Submirror of d0
State: Okay
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t2d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t1d0 Yes id1,ssd@w500000e012a19430
c1t2d0 Yes id1,ssd@w500000e012a215f0

3)Format replacement disk to match failed disk

4)Enable replacement disk with the metareplace command. –e option denotes a physical disk replacement.

# metareplace -e d0 c1t0d0s0
d0: device c1t0d0s0 is enabled
# metareplace -e d1 c1t0d0s1
d1: device c1t0d0s1 is enabled
# metareplace -e d4 c1t0d0s4
d4: device c1t0d0s4 is enabled
# metareplace -e d5 c1t0d0s5
d5: device c1t0d0s5 is enabled
# metareplace -e d6 c1t0d0s6
d6: device c1t0d0s6 is enabled
# metareplace -e d7 c1t0d0s7
d7: device c1t0d0s7 is enabled

5)Verify meta-devices

# metastat d0
d0: Mirror
Submirror 0: d10
State: Resyncing
Submirror 1: d20
State: Okay
Submirror 2: d30
State: Okay
Resync in progress: 11 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 24585216 blocks (11 GB)

d10: Submirror of d0
State: Resyncing
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Resyncing Yes


d20: Submirror of d0
State: Okay
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


d30: Submirror of d0
State: Okay
Size: 24585216 blocks (11 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t2d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@w500000e012a19340
c1t1d0 Yes id1,ssd@w500000e012a19430
c1t2d0 Yes id1,ssd@w500000e012a215f0

6)Verify meta-device replicas are intact.

Healthy Replicas:

# metadb -i
flags first blk block count
a m p luo 16 8192 /dev/dsk/c1t0d0s3
a p luo 16 8192 /dev/dsk/c1t1d0s3
a p luo 16 8192 /dev/dsk/c1t2d0s3
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Unhealthy Replicas:

root@spbuse01 # metadb -i
flags first blk block count
M p 16 unknown /dev/dsk/c1t0d0s3
a m p luo 16 8192 /dev/dsk/c1t1d0s3
a p luo 16 8192 /dev/dsk/c1t2d0s3
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

7)If a replica is in an error state, delete the replica and re-create.

# metadb -d /dev/dsk/c1t0d0s3
# metadb -a /dev/dsk/c1t0d0s3
# metadb -i
flags first blk block count
a u 16 8192 /dev/dsk/c1t0d0s3
a m p luo 16 8192 /dev/dsk/c1t1d0s3
a p luo 16 8192 /dev/dsk/c1t2d0s3
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

8)Once the meta-devices complete re-sync and replicas are healthy, the system can be rebooted off the primary boot device.

 

[dandan123]

If I remember right using raidctl one drive will appear to vanish. On a 440 you can only mirror one set of two drives, I'm assuming that when one drive fail the system will boot off the other drive using the same physical drive letter but I'm not sure as I haven't seen a drive failure on a system using raidctl.

 

[csgonan]

Thanks for the info. I used Disksuite to mirror the root drives on 2 servers and it went smoothly. One of the servers has a second (single-parition) drive and I didn't know if I needed metadb on it too, so I used raidctl to mirror it.

If one of the non-root raidctl mirror drives fails, will the server automatically use the second drive? What happens if there is a reboot and vfstab points to the failed drive? Is the raidctl volume defined in the hardware so it will come up using the non-failed disk?

Also I notice that the raidctl volume resyncs after each boot? What happens if writes come in while it is resyncing? Do they get buffered and writen after the resync is done?

Thanks