Mirrored disk removed incorrectly

[normntwrk]

IBM came in and removed hdisk0 which was mirrored with hdisk1 (rootvg) without breaking the mirror and reducing the vg and now i can't do anything with it. There are a couple of posts in here with the exact same situation but I tried what they did and it just fails. I can't reducevg rootvg , can't remove the mirrored LV's using the PVID....nothing seems to work
# lsvg -p rootvg
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
rootvg:
PV ID PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
000b70dc5408433d removed 542 253 109..99..00..00..45
hdisk1 active 542 252 108..99..00..00..45
#

 

[normntwrk]

tried those also, get errors with both of those commands

 

[blarneyme]

syncvg rootvg

That should correct the stale partitions, then proceed to remove the incorrect pvid by removing all entries for hdisk0 from the ODM.

Then redefinevg and synclvodm.

 

[normntwrk]

# syncvg -v rootvg
0516-1296 lresynclv: Unable to completely resynchronize volume.
The logical volume has bad-block relocation policy turned off.
This may have caused the command to fail.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd6.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd8.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd4.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd2.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd9var.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd3.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
0516-934 /etc/syncvg: Unable to synchronize logical volume hd10opt.
0516-934 /etc/syncvg: Unable to synchronize logical volume loglv00.
0516-934 /etc/syncvg: Unable to synchronize logical volume fslv00.
0516-932 /etc/syncvg: Unable to synchronize volume group rootvg.

 

[blarneyme]

Did you unmirror rootvg? It will probably throw some errors but should still complete.

 

[Kareltje]

I just had the same thing, except it was me who made the mistake (as an ex-ibm'er).

search the odm for the missing pv (odmget CuAt|grep -p rootvg) and remove the missing disk from the odm (odmdelete -o CuAt -q value=thefullidfoundintheodmgetwithallthezeroes)
remove the lv copies on an lv basis, do not try it on the vg basis. (rmlvcopy hd8 1 00c28923988923cf)

And then mirror to the new disk

Hope this helps

 

[normntwrk]

unmirrored rootvg hdisk0 Ok
reducevg rootvg hdisk0 OK
odmdelete OK

# odmdelete -q value=000b70dc5408433d0000000000000000 -o CuAt
0518-307 odmdelete: 1 objects deleted.
# rmlvcopy hd8 1 000b70dc5408433d
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
0516-922 rmlvcopy: Unable to remove logical partition copies from
logical volume hd8.
#

 

[normntwrk]

ran syncvg after unmirror , got same errors as before

can't seem to get rid of that ghost PVID
still:

# lslv -l hd8
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
hd8:N/A
PVID COPIES IN BAND DISTRIBUTION
000b70dc5408433d 001:000:000 100% 000:000:001:000:000
hdisk1 001:000:000 100% 000:000:001:000:000
#

# lsvg -p rootvg
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
rootvg:
PV ID PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
000b70dc5408433d removed 542 253 109..99..00..00..45
hdisk1 active 542 252 108..99..00..00..45
# #
# lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
hd6 000b70dc5408433d rootvg 512MB 1 yes yes lv

 

[normntwrk]

I rmdev -dl hdisk0

cfgmgr

extendvg rootvg hdisk0 OK

mirrorvg rootvg hdisk0 some errors

this is the only thing it mirrored
# lslv -l hd5
hd5:N/A
PV COPIES IN BAND DISTRIBUTION
hdisk1 001:000:000 100% 001:000:000:000:000
hdisk0 001:000:000 100% 001:000:000:000:000

 

[Kareltje]

Check for any other references to 000b70dc5408433d in the odm.
What does lsvg -l rootvg show ?
(the others may still be mirrored to the phantom disk, the default for mirrorvg is -c 2)

 

[normntwrk]

# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 8 16 2 open/stale N/A
hd8 jfslog 1 2 2 open/stale N/A
hd4 jfs 1 2 2 open/stale /
hd2 jfs 33 66 2 open/stale /usr
hd9var jfs 1 2 2 open/stale /var
hd3 jfs 1 2 2 open/stale /tmp
hd1 jfs 1 2 2 open/stale /home
hd10opt jfs 2 4 2 open/stale /opt
loglv00 jfs2log 1 2 2 open/stale N/A
fslv00 jfs2 240 480 2 open/stale /work
#

 

[Kareltje]

Ok, so they probably all (except hd8) still are mirrored to the phantom disk.
Can you do a synclvodm and then retry the rmlvcopy for all ?

(My apologies for not having a clear and concise set of commands on how to solve this, it is a bit of trial and error. When you have these kinds of issues you don't retrace all your steps exactly, and you usually do not want to reproduce it just to get it documented)

 

[Kareltje]

Sorry, except hd5

 

[normntwrk]

# synclvodm -v rootvg
0516-510 synclvodm: Physical volume not found for physical volume
identifier 000b70dc5408433d0000000000000000.
0516-548 synclvodm: Partially successful with updating volume
group rootvg.
synclvodm: Logical volume hd5 updated.
synclvodm: Logical volume hd6 updated.
synclvodm: Logical volume hd8 updated.
synclvodm: Logical volume hd4 updated.
synclvodm: Logical volume hd2 updated.
synclvodm: Logical volume hd9var updated.
synclvodm: Logical volume hd3 updated.
synclvodm: Logical volume hd1 updated.
synclvodm: Logical volume hd10opt updated.
synclvodm: Logical volume loglv00 updated.
synclvodm: Logical volume fslv00 updated.
**********************************

# rmlvcopy hd6 1 000b70dc5408433d
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
0516-922 rmlvcopy: Unable to remove logical partition copies from
logical volume hd6.


# rmlvcopy hd4 1 000b70dc5408433d
0516-304 : Unable to find device id 000b70dc5408433d in the Device
Configuration Database.
0516-922 rmlvcopy: Unable to remove logical partition copies from
logical volume hd4.

 

[Kareltje]

Hmm, not nice.

I went through my shell history.
Just before the rmlvcopy commands I did :

synclvodm -v rootvg
reducevg -df rootvg 00c628eb9a217b3b (which did not work)
redefinevg -d hdisk3 rootvg
extendvg -f rootvg hdisk4 (did not work)
odmget CuAt|grep hd8
odmget CuDv|grep hd8

then all rmlvcopy's
reducevg -df rootvg 00c628eb9a217b3b (which did work)
extendvg rootvg hdisk4
mirrorvg -c2 rootvg hdisk3 hdisk4

So look very similar to your case, only difference maybe the redefinevg, but I cannot remember if that worked.
Is the odm totally clean of the phantom disk ?

 

[normntwrk]

I did a grep on all parts of the odm and didn't see anything related to the PVID

thanks for the tips, got other issues right now I'll see if I can get back to it later