hdisk0 defined missing, on same address as hdisk1

[finncrisp]

i have an old server with a mirrored rootvg ( with 2 HDs). i try to mimic this server to a new by using the mirrored disk.

i powered down the server. took one drive to another server and booted the new server up with this mirrored drive and a brand new drive.

then i ended up with a weired situation.
- diag shows 2 drives.
slot1 - hdisk2
slot2 - hdisk1
- But lsdev shows 3 drives. 2 disks(hdisk0/1) are on same scsi address. one is defined.
# lsdev -Cc disk
hdisk0 Defined 06-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive
- And rootvg shows a missing PV,
# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 missing 546 450 102..93..37..109..109
hdisk1 active 546 101 06..00..00..00..95
hdisk2 active 546 538 110..101..109..109..109

- somehow i can lspv all 3 PVs and even the missing disk/PV (hdisk0)has contents.

bash-4.2# lspv -l hdisk0
hdisk0:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
hd2 20 20 00..00..20..00..00 /usr
hd4 2 2 00..00..02..00..00 /
hd3 3 3 00..00..03..00..00 /tmp
hd9var 12 12 00..00..12..00..00 /var
hd10opt 10 10 00..00..10..00..00 /opt
hd1 24 24 00..00..24..00..00 /home
fslv004 8 8 08..00..00..00..00 /mc
hd8 1 1 00..00..01..00..00 N/A
hd6 16 16 00..16..00..00..00 N/A
bash-4.2# lspv -l hdisk1
hdisk1:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
fslv00 200 200 40..00..37..109..14 /trs
fslv01 40 40 00..40..00..00..00 N/A
fslv02 112 112 60..52..00..00..00 /backup
hd11admin 1 1 01..00..00..00..00 /admin
loglv00 1 1 00..01..00..00..00 N/A
hd2 20 20 00..00..20..00..00 /usr
hd4 2 2 00..00..02..00..00 /
hd3 3 3 00..00..03..00..00 /tmp
hd9var 12 12 00..00..12..00..00 /var
hd10opt 10 10 00..00..10..00..00 /opt
hd1 24 24 00..00..24..00..00 /home
livedump 1 1 01..00..00..00..00 /var/adm/ras/livedump
fwdump 1 1 01..00..00..00..00 /var/adm/ras/platform
hd5 1 1 01..00..00..00..00 N/A
hd8 1 1 00..00..01..00..00 N/A
hd6 16 16 00..16..00..00..00 N/A
bash-4.2# lspv -l hdisk2
hdisk2:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
fslv004 8 8 00..08..00..00..00 /mc

- the mount points lists from rootvg is a combination of all 3 PVs. although i only have 2 disks.
bash-4.2# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 16 32 2 open/syncd N/A
hd8 jfs2log 1 2 2 open/stale N/A
hd4 jfs2 2 4 2 open/stale /
hd2 jfs2 20 40 2 open/stale /usr
hd9var jfs2 12 24 2 open/stale /var
hd3 jfs2 3 6 2 open/stale /tmp
hd1 jfs2 24 48 2 open/stale /home
hd10opt jfs2 10 20 2 open/stale /opt
loglv00 jfslog 1 1 1 closed/syncd N/A
fslv00 jfs2 200 200 1 open/syncd /trs
fslv01 jfs2 40 40 1 closed/syncd N/A
fslv02 jfs2 112 112 1 open/syncd /backup
hd11admin jfs2 1 1 1 open/syncd /admin
livedump jfs2 1 1 1 open/syncd /var/adm/ras/livedump
fwdump jfs2 1 1 1 open/syncd /var/adm/ras/platform
fslv004 jfs 8 16 2 closed/stale /mc

- I have tried to remove the hdisk0 from rootvg. but i can not.
# reducevg -d -f rootvg hdisk0
0516-076 lreducelv: Cannot remove last good copy of stale partition.
Resynchronize the partitions with syncvg and try again.
0516-912 rmlv: Unable to remove logical volume fslv004.
0516-884 reducevg: Unable to remove physical volume hdisk0.


How can i resolve this duplicated/missing drives? i feel it is PVID or something messed up in OS.

thanks a lot for any suggestions.

 

[finncrisp]

just a bit more about the server. it is a p5-505 rackmount server with 2 internal scsi drives.

 

[DukeSSD]

You'd be much better off using a supported method, like a mksysb migration.

You'll probably need IBM support to learn how to hack out all the old stuff from the ODM to make it behave correctly but they won't help because you didn't use a supported method of migration.

Good luck.

 

[NoLogic001]

Hi,

What you are seeing makes sense. Remember that although hdisk0 is still in the ODM, it is physically not present. What you are seeing is a "ghost" of disk.

The two active disks now are hdisk1 and hdisk2.

This is what I would do.

1.) rmdev -Rdl hdisk0 (remove all traces from the ODM)
2.) rendev -l hdisk2 -n hdisk0 (rename hdisk2 to hdisk0)
3.) extendvg –f rootvg hdisk0 (phase 1 of creating a mirrored disk)
4.) mirrorvg -S rootvg hdisk0 (phase 2 of creating a mirrored disk)
5.) bosboot -a -d /dev/hdisk0 (creates boot sector on disk0)
6.) bosboot -a -d /dev/hdisk1 (creates boot sector on hdisk1)
7.) bootlist -m normal -o (confirm boot sectors)


You might also see Defined sas/vscsi/fcs adapters with the same issues, you are welcome to remove/ rename them to fit your standard.

Regards

 

[finncrisp]

Hello DukeSSD and NoLogic001,

Thanks a lot for your help.

Yes. i am a newbie on AIX. i can feel something in this box remembering the old disk. But i didnt know the ODM thing before. yes. then it totally makes sense.

this is a new server. when i first got it, i booted it up with an AIX7.1 CD and went through the installation. i thought i just tested the hard drive working.

then i tried to boot up the box with a disk from an old server and tried to mirror the old disk. so i can see all the problems.

i will try NoLogic001's strategy today and let you know if that works.

Thanks a lot guys! Appreciate very much.

 

[finncrisp]

sorry i am having problem un-mirror the rootvg right now.

rootvg has 3 disks.
bash-4.2# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 missing 546 458 110..93..37..109..109
hdisk1 active 546 102 06..01..00..00..95
hdisk2 active 546 190 66..00..00..15..109

"unmirrorvg" command run fine.
bash-4.2# unmirrorvg rootvg hdisk2
0516-1804 chvg: The quorum change takes effect immediately.
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
bosboot of system to reinitialize boot records. Then, user must modify
bootlist to just include: hdisk1.

-- current boot list has both 2 drives
bash-4.2# bootlist -m normal -o
hdisk1 blv=hd5 pathid=0
hdisk2 blv=hd5 pathid=0

-- Set the boot list to only hdisk1. and Verify.
# chpv -c hdisk2
# bootlist -m normal hdisk1 blv=hd5
# bootlist -m normal -o
hdisk1 blv=hd5 pathid=0

-- run bosboot command to reflect the changes
# bosboot -ad /dev/ipldevice

bosboot: Boot image is 55324 512 byte blocks.

But "reducevg" failed.
bash-4.2# reducevg rootvg hdisk2
0516-084 ldeletepv: Cannot remove or delete physical volume
without losing quorum.
0516-884 reducevg: Unable to remove physical volume hdisk2.

- this PV has no LV. but belongs to rootvg.
# lspv hdisk2
PHYSICAL VOLUME: hdisk2 VOLUME GROUP: rootvg
PV IDENTIFIER: 0001d2ba7c0d1fcc VG IDENTIFIER 0002fdaa0000d300000001444c5c5d77
PV STATE: active
STALE PARTITIONS: 0 ALLOCATABLE: yes
PP SIZE: 256 megabyte(s) LOGICAL VOLUMES: 0
TOTAL PPs: 546 (139776 megabytes) VG DESCRIPTORS: 1
FREE PPs: 546 (139776 megabytes) HOT SPARE: no
USED PPs: 0 (0 megabytes) MAX REQUEST: 256 kilobytes
FREE DISTRIBUTION: 110..109..109..109..109
USED DISTRIBUTION: 00..00..00..00..00
MIRROR POOL: None


-- i tried using PVID with "reducevg". still failed.
bash-4.2# reducevg -d -f rootvg 0001d2ba7c0d1fcc
0516-084 ldeletepv: Cannot remove or delete physical volume
without losing quorum.
0516-884 reducevg: Unable to remove physical volume 0001d2ba7c0d1fcc.

So should i do this command against rootvg in this case?
# ldeletepv -v VGID -p PVID


thanks in advance,

 

[finncrisp]

i am getting same error with ldeletepv command :

bash-4.2# ldeletepv -g 0002fdaa0000d300000001444c5c5d77 -p 0001d2ba7c0d1fcc
0516-084 ldeletepv: Cannot remove or delete physical volume
without losing quorum.

:<

 

[NoLogic001]

Hi,

is there is still a VG on hdisk2?

1.) lspv

2.) lspv -l hdisk2

If the VG is not present on hdisk2 you are welcome to continue with deleting / renaming of the disk.

 

[finncrisp]

it is OK now. i "rmdev -Rdl hdisk0" and reboot the server. the "ghost" hdisk0 is gone. And i have 2 good looking disks, hdisk1 and hdisk2.

and i gave up deleting the hdisk2 off the rootvg, instead i added it back to mirror. and run "mirrorvg".

everything looks fine now.

thanks everyone for helping out! have learn a great lesson.