Veritas Volume Manager Problems

[100mbs]

OK Here is the problem.

I have a Disk Group called "ORA1" and this group is showing two drives that are bad.

Everytime i run the option to replace a drive for replacement i get the below errors.

Does anybody have any ideas here???

Thanks....




Replace a failed or removed disk
Menu: VolumeManager/Disk/ReplaceDisk

Use this menu operation to specify a replacement disk for a disk that you removed with the "Remove a disk for replacement" menu operation, or that failed during use. You will be prompted for a disk name to replace and a disk device to use as a replacement. You can choose an uninitialized disk, in which case the disk will be initialized, or you can choose a disk that you have already
initialized using the Add or initialize a disk menu operation.

Select a removed or failed disk [<disk>,list,q,?] list

Disk group: rootdg

DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE


Disk group: mediadg

DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE


Disk group: ora1

DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE

dm c3t4d1 - - - - REMOVED


Select a removed or failed disk [<disk>,list,q,?] c3t4d1

Select disk device to initialize [<address>,list,q,?] list

DEVICE DISK GROUP STATUS
c1t8d0 c1t8 rootdg online
c1t9d0 mediadg02 mediadg online
c1t10d0 c1t10d0 mediadg online
c1t11d0 c1t11d0 mediadg online
c1t12d0 c1t12d0 mediadg online
c1t13d0 c1t13 ora1 online
c2t0d0 c2t0 rootdg online
c2t1d0 mediadg01 mediadg online
c2t2d0 c2t2d0 mediadg online
c2t3d0 c2t3d0 mediadg online
c2t4d0 c2t4d0 mediadg online
c2t5d0 c2t5d0 ora1 online
c4t4d0 c4t4d0 ora1 online
c4t4d1 c4t4d1 ora1 online failing
c7t4d0 c3t4d0 ora1 online
c7t4d1 - - error

Select disk device to initialize [<address>,list,q,?] c4t4d1

The following device has already been added to a disk group. This disk will be ignored. Output format: [Device_Name,Disk_Access_Name,Disk_Group]

[c4t4d1,c4t4d1s2,ora1]

Hit RETURN to continue.

Select a different device? [y,n,q,?] (default: n) y

Select disk device to initialize [<address>,list,q,?] list

DEVICE DISK GROUP STATUS
c1t8d0 c1t8 rootdg online
c1t9d0 mediadg02 mediadg online
c1t10d0 c1t10d0 mediadg online
c1t11d0 c1t11d0 mediadg online
c1t12d0 c1t12d0 mediadg online
c1t13d0 c1t13 ora1 online
c2t0d0 c2t0 rootdg online
c2t1d0 mediadg01 mediadg online
c2t2d0 c2t2d0 mediadg online
c2t3d0 c2t3d0 mediadg online
c2t4d0 c2t4d0 mediadg online
c2t5d0 c2t5d0 ora1 online
c4t4d0 c4t4d0 ora1 online
c4t4d1 c4t4d1 ora1 online failing
c7t4d0 c3t4d0 ora1 online
c7t4d1 - - error

Select disk device to initialize [<address>,list,q,?] c7t4d1

This disk device does not appear to be valid. The disk may not be formatted (format(1M)), may have a corrupted VTOC, the device file for the disk may be missing or invalid, or the device may be turned-off or detached from the system. This disk will be ignored. Output format: [Device_Name,Disk_Access_Name]

[c7t4d1,c7t4d1s2]

Hit RETURN to continue.

 

[Annihilannic]

On the first occasion you attempted to use c4t4d1 as a replacement disk, however according to vxdisk list it is currently online and in use:

[tt]c4t4d1 c4t4d1 ora1 online failing[/tt]

The "failing" keyword just means that there have been I/O errors to that disk, but they may have been transient or cabling related, so the disk may be perfectly healthy.

On the second occasion you attempted to use c7t4d1:

[tt]c7t4d1 - - error[/tt]

The "error" flag means that the disk is not recognised as available for use by Veritas Volume Manager. If you just do a format /dev/rdsk/c7t4d1s2 and use the label option, then quit format the disk will be ready to use. You then need to run vxdctl enable to make Volume Manager rescan the disks and recognise that the disk is okay. You should now be able to go into vxdiskadm again to initialise c7t4d1 and use it as a replacement.

Annihilannic.

 

[100mbs]

Well I followed your advice and this is what i got after i ran the Add or Initialize disks.

When i did format it cmae up and said the disk is unformatted and then asked to label it and i said yes.





Continue with operation? [y,n,q,?] (default: y)

This disk device does not appear to be valid. The disk may not be formatted (format(1M)), may have a corrupted VTOC, the device file for the disk may be missing or invalid, or the device may be turned-off or detached from the system. This disk will be ignored. Output format: [Device_Name,Disk_Access_Name]

[c7t4d1,c7t4d1s2]

 

[dandan123]

As Annhilanic has pointed out you need to run the format command. This is not a veritas command, you need to become root and then type format /dev/rdsk/c7t4d1s2

Then select the label option and label the drive. Once that's done run vxdctl to make veritas recognise the disk.

Finally run vxdiskadm and do the replace.

 

[Annihilannic]

Hmm... what happens if you do a vxdisksetup -i c7t4d1? Does it succeed? If it does it should be listed as 'online' in vxdisk list.

Annihilannic.

 

[100mbs]

I ran through the format and label process then i ran vxdctl. Every time i run this command it keeps adding more slices to the vxdisk list but the drive never comes on line. See below.

Any other suggestions?


DEVICE DISK GROUP STATUS
c1t0d0 c1t0d0 rootdg online
c1t1d0 c1t1d0 rootdg online
c1t2d0 c1t2d0 rootdg online
c1t3d0 c1t3d0 rootdg online
c1t4d0 c1t4d0 rootdg online
c1t5d0 - - error
c1t5d0 - - error
c1t5d0 - - error
c2t0d0 c2t0d0 rootdg online
c2t1d0 c2t1d0 rootdg online failing
c2t2d0 c2t2d0 rootdg online
c2t3d0 c2t3d0 rootdg online
c2t4d0 c2t4d0 rootdg online
c2t5d0 c2t5d0 rootdg online
c5t1d0 c3t1d0-t0 orcl2-t3 online
c6t0d0 c5t0d0-t1 orcl2-t3 online

 

[Annihilannic]

Is this a different system? Because the vxdisk list output looks quite different.

What operating system exactly are you using? And what kind of hardware architecture?

Annihilannic.

 

[100mbs]

Yes this is a different system but the same problem. The other vxdisk list was from my test system and the post from today is my production server.

It i a SUN V880 runnin SOlaris 5.8.

 

[Annihilannic]

Did you try the vxdisksetup -i command I mentioned previously?

What is the output of prtvtoc /dev/rdsk/c1t5d0s2?

Annihilannic.

 

[100mbs]

I will try the vxdisksetup.


# prtvtoc /dev/rdsk/c1t5d0s2
* /dev/rdsk/c1t5d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 424 sectors/track
* 24 tracks/cylinder
* 10176 sectors/cylinder
* 14089 cylinders
* 14087 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 0 264576 264575
1 3 01 264576 264576 529151
2 5 01 0 143349312 143349311
6 4 00 529152 142820160 143349311

 

[100mbs]

This is what i get when running vxdisksetup:

# vxdisksetup -i c1t5d0
vxvm:vxdisk: ERROR: Device c1t5d0s2: online failed:
Device path not valid

 

[Annihilannic]

Hmm... it's a weird one alright. What kind of disks are they, in what kind of storage, and what controllers are used to connect to them?

What's the output of format < /dev/null?

And finally, if you do a vxdmpadm listctlr all and then a vxdmpadm getsubpaths ctlr=cN for each of the controllers listed by the previous command, what is the output?

Sorry about all the information requests... but so far I've seen nothing to explain the behaviour you are seeing... oh, one more command, vxdmpadm listenclosure all.

Annihilannic.

 

[100mbs]

This is an internal HDD in a SUN V880 and the drive is a Hitachi 72GB.

And thanks for trying to help. I am lost with this one.


# format < /dev/null
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000000876c9bbf,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000000876c79a5,0
2. c1t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000000876ca501,0
3. c1t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000000876c7996,0
4. c1t4d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000000876ca3be,0
5. c1t5d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e012013651,0
6. c2t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e01081e441,0
7. c2t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e01081cf81,0
8. c2t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e010822781,0
9. c2t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e01081e2d1,0
10. c2t4d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e01081cbc1,0
11. c2t5d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@9,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w500000e010825481,0
12. c5t1d0 <SUN-T300-0117 cyl 34901 alt 2 hd 224 sec 128>
/pci@9,600000/pci@1/SUNW,qlc@5/fp@0,0/ssd@w50020f23000079dc,0
13. c6t0d0 <SUN-T300-0117 cyl 34901 alt 2 hd 224 sec 128>
/pci@9,600000/pci@2/SUNW,qlc@5/fp@0,0/ssd@w50020f2300006a64,0
Specify disk (enter its number):





# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 SENA ENABLED SENA0
c2 SENA ENABLED SENA1
c5 T3 ENABLED T31
c6 T3 ENABLED T30




# vxdmpadm getsubpaths ctlr=c1
NAME STATE PATH-TYPE DMPNODENAME ENCLR-TYPE ENCLR-NAME
======================================================================
c1t0d0s2 ENABLED - c1t0d0s2 SENA SENA0
c1t3d0s2 ENABLED - c1t3d0s2 SENA SENA0
c1t1d0s2 ENABLED - c1t1d0s2 SENA SENA0
c1t2d0s2 ENABLED - c1t2d0s2 SENA SENA0
c1t4d0s2 ENABLED - c1t4d0s2 SENA SENA0
c1t5d0s2 DISABLED - c1t5d0s2 SENA SENA0
c1t5d0s2 ENABLED - c1t5d0s2 SENA SENA0

 

[100mbs]

Here is an update.

My server was rebooted over the weekend because of power problems.

When I do a "vxdisk list" This is what shows up.

How do i correct the c1t5d0s2 invalid issue and also move the bottom line back to where it was originally?




# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 sliced c1t0d0 rootdg online
c1t1d0s2 sliced c1t1d0 rootdg online
c1t2d0s2 sliced c1t2d0 rootdg online
c1t3d0s2 sliced c1t3d0 rootdg online
c1t4d0s2 sliced c1t4d0 rootdg online
c1t5d0s2 sliced - - online invalid
c2t0d0s2 sliced c2t0d0 rootdg online
c2t1d0s2 sliced c2t1d0 rootdg online failing
c2t2d0s2 sliced c2t2d0 rootdg online
c2t3d0s2 sliced c2t3d0 rootdg online
c2t4d0s2 sliced c2t4d0 rootdg online
c2t5d0s2 sliced c2t5d0 rootdg online
c5t1d0s2 sliced c3t1d0-t0 orcl2-t3 online
c6t0d0s2 sliced c5t0d0-t1 orcl2-t3 online
- - c1t5d0 rootdg removed was:c1t5d0s2

 

[dandan123]

c1t5d0s2 appears to have failed, to check further do -

echo | format ( and look for c1t5d0s2)

also -

iostat -En

If the drive has indeed failed then us vxdiskadm , option 4 to remove the drive, physically replace the drive, option 5 to replace the drive under Veritas.

After physically replacing the drive do a
vxdisk list

to see Veritas sees the drive, if not do a

vxdctl enable

 

[100mbs]

I just ran vxdisksetup -i c1t5d0 again and that worked now it shows it is online.

Now when i do a vxdiskadm and select;

14 Unrelocate subdisks back to a disk

Here is what i get for a result:

Move hot-relocated subdisks back to a disk
Menu: VolumeManager/Disk/UnrelocateDisk

Use this operation to move subdisks which were hot-relocated back
onto the original disk that has been replaced due to a disk failure.
This operation takes, as input, the original disk name. If the
failed drive was replaced with a disk using a different name, this
operation also provides an option to specify the new name.

Enter the original disk name [<disk>,list,q,?] list

c1t5d0

Enter the original disk name [<disk>,list,q,?] c1t5d0

Unrelocate to a new disk [y,n,q,?] (default: n)

Disk media name c1t5d0 does not appear to be valid.

Select a different disk ? [y,n,q,?] (default: n)

 

[Annihilannic]

I think there's a possibility that the disk is okay and that Veritas was just confused before the reboot... before replacing the disk I would attempt to test it using the analyze option in format, and if it's okay, just use vxdiskadm option 5 to try and replace it with itself, if you know what I mean.

Annihilannic.

 

[100mbs]

Annihilannic and everyone else who replied. By doing the vxdiskadm and option 5 worked.

I'm glad that is over.

 

[dandan123]

Thanks for keeping us posted.