Disk Issues within a mirrored rootvg AIX 4.3.3_09

[dominicdiddy]

Hi,

I am currently having issue in that I have been appointed as System Manager for a 2 week period to address some issues that have become apparent on an RS/6000 Model 144P.

It seems that at some stage, a previous admin has swapped out a disk without updating the system correctly. These are the current outputs. I need to recreate the rootvg mirror group and presumably update the ODM. Please see details below

# lsdev -Cc disk
hdisk0 Available 10-60-00-4,0 16 Bit LVD SCSI Disk Drive
hdisk1 Defined 10-60-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 10-61-00-0,0 16 Bit LVD SCSI Disk Drive
hdisk3 Available 10-61-00-1,0 16 Bit LVD SCSI Disk Drive
hdisk4 Available 10-61-00-2,0 16 Bit LVD SCSI Disk Drive
hdisk5 Defined 10-60-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk6 Available 10-60-00-5,0 Other SCSI Disk Drive
# lspv
hdisk0 0006bb1fbf7c0307 rootvg
hdisk2 0006bb1f015e8334 unifyvg
hdisk3 0006bb1f015ecc81 unifyvg
hdisk4 0006bb1fa2471205 unifyvg
hdisk6 none None
# lsvg rootvg
VOLUME GROUP: rootvg VG IDENTIFIER: 0006bb1f01363a44
VG STATE: active PP SIZE: 16 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 1084 (17344 megabytes)
MAX LVs: 256 FREE PPs: 450 (7200 megabytes)
LVs: 11 USED PPs: 634 (10144 megabytes)
OPEN LVs: 9 QUORUM: 1
TOTAL PVs: 2 VG DESCRIPTORS: 3
STALE PVs: 1 STALE PPs: 201
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per PV: 1016 MAX PVs: 32
# slvg -l rootvg
ksh: slvg: not found.
# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/stale N/A
hd6 paging 64 128 2 open/stale N/A
hd8 jfslog 1 2 2 open/stale N/A
hd4 jfs 32 64 2 open/stale /
hd2 jfs 125 250 2 open/stale /usr
hd9var jfs 31 62 2 open/stale /var
hd3 jfs 31 62 2 open/stale /tmp
hd1 jfs 26 52 2 open/stale /home
locallv jfs 2 4 2 open/stale /usr/local
hd7 sysdump 4 4 1 open/syncd N/A
hd71 sysdump 4 4 1 closed/syncd N/A
# lspv hdisk1
PHYSICAL VOLUME: hdisk1 VOLUME GROUP: rootvg
PV IDENTIFIER: 0006bb1f01615543 VG IDENTIFIER 0006bb1f01363a44
PV STATE: missing
STALE PARTITIONS: 201 ALLOCATABLE: yes
PP SIZE: 16 megabyte(s) LOGICAL VOLUMES: 10
TOTAL PPs: 542 (8672 megabytes) VG DESCRIPTORS: 1
FREE PPs: 225 (3600 megabytes)
USED PPs: 317 (5072 megabytes)
FREE DISTRIBUTION: 107..00..00..09..109
USED DISTRIBUTION: 02..108..108..99..00
# lspv hdisk0
PHYSICAL VOLUME: hdisk0 VOLUME GROUP: rootvg
PV IDENTIFIER: 0006bb1fbf7c0307 VG IDENTIFIER 0006bb1f01363a44
PV STATE: active
STALE PARTITIONS: 0 ALLOCATABLE: yes
PP SIZE: 16 megabyte(s) LOGICAL VOLUMES: 10
TOTAL PPs: 542 (8672 megabytes) VG DESCRIPTORS: 2
FREE PPs: 225 (3600 megabytes)
USED PPs: 317 (5072 megabytes)
FREE DISTRIBUTION: 107..00..00..09..109
USED DISTRIBUTION: 02..108..108..99..00
# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 542 225 107..00..00..09..109
hdisk1 missing 542 225 107..00..00..09..109


Please help!

Thanks

D

 

[mrn]

Firstly try and make hdisk1 available.

mkdev -l hdisk1

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[mrn]

Then take a look at section 5 in the following document

http://www.redbooks.ibm.com/redbooks/pdfs/sg246183.pdf

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[dominicdiddy]

Hi Mike,

Thanks for your response,

before running command, should there not be concern that hdisk1, hdisk5, and hdisk6 all point to same scsi device

# lsdev -Cc disk
hdisk0 Available 10-60-00-4,0 16 Bit LVD SCSI Disk Drive
hdisk1 Defined 10-60-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 10-61-00-0,0 16 Bit LVD SCSI Disk Drive
hdisk3 Available 10-61-00-1,0 16 Bit LVD SCSI Disk Drive
hdisk4 Available 10-61-00-2,0 16 Bit LVD SCSI Disk Drive
hdisk5 Defined 10-60-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk6 Available 10-60-00-5,0 Other SCSI Disk Drive
# lspv hdisk1
PHYSICAL VOLUME: hdisk1 VOLUME GROUP: rootvg
PV IDENTIFIER: 0006bb1f01615543 VG IDENTIFIER 0006bb1f01363a44
PV STATE: missing
STALE PARTITIONS: 201 ALLOCATABLE: yes
PP SIZE: 16 megabyte(s) LOGICAL VOLUMES: 10
TOTAL PPs: 542 (8672 megabytes) VG DESCRIPTORS: 1
FREE PPs: 225 (3600 megabytes)
USED PPs: 317 (5072 megabytes)
FREE DISTRIBUTION: 107..00..00..09..109
USED DISTRIBUTION: 02..108..108..99..00
# lspv -l hdisk1
hdisk1:
LV NAME LPs PPs DISTRIBUTION MOUNT POINT
hd5 1 1 01..00..00..00..00 N/A
locallv 2 2 01..01..00..00..00 /usr/local
hd6 64 64 00..64..00..00..00 N/A
hd71 4 4 00..04..00..00..00 N/A
hd4 32 32 00..05..01..26..00 /
hd2 125 125 00..21..41..63..00 /usr
hd3 31 31 00..13..18..00..00 /tmp
hd8 1 1 00..00..01..00..00 N/A
hd9var 31 31 00..00..31..00..00 /var
hd1 26 26 00..00..16..10..00 /home
# lspv hdisk5
0516-320 : Physical volume 0000000000000000 is not assigned to
a volume group.
# lspv hdisk6
0516-320 : Physical volume 0000000000000000 is not assigned to
a volume group.

Cheers
D

 

[dominicdiddy]

just for additional info it seems that I would need to reduce the vg before updating with new disk?

# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 active 542 225 107..00..00..09..109
hdisk1 missing 542 225 107..00..00..09..109


This is roughly what I plan... A 2nd opinion would be greatly appreciated!!!

1. Reduce volume group to remove hdisk1 from rootvg (do not need to rmdev hdisk1 because it has already been replaced and appears defined...?)
2. make the device file for the new hdisk1
3. extendvg to include new hdisk1 in rootvg
4. disable quorum
5. mklvcopy of each hd1 - 10
6. syncvg -v rootvg
7. bosboot ---
8. bootlist ---
9. shutdown -Fr
10. cross fingers (not both hands)
11. Hopefully go to pub

Does this sound right? Any advice, 2nd opinions greatly appreciated - particularly regarding the 3 hdisk entries with same scsi addresses.

Thanks

D




My only concern is that hdisk1,5 & 6 all have the same scsi addresses - presumably they are therefore the same disk Do I need to delete the entries (either remove the record of the device file rmpv, or edit via odm)or am I ahead of myself and just need to resync the rootvg...

What joy!

D

 

[mrn]

You missed the most important part. Do a mksysb before you start.

Can you supply the output for the following on all disks

lsattr -l hdisk0 -E

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[dominicdiddy]

absolutely right! two mksysb's are in order for this one!


# lsattr -l hdisk0 -E
pvid 0006bb1fbf7c03070000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 9100 Size in Megabytes False
# lsattr -l hdisk1 -E
pvid 0006bb1f016155430000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 9100 Size in Megabytes False
# lsattr -l hdisk2 -E
pvid 0006bb1f015e83340000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 18200 Size in Megabytes False
# lsattr -l hdisk3 -E
pvid 0006bb1f015ecc810000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 18200 Size in Megabytes False
# lsattr -l hdisk4 -E
pvid 0006bb1fa24712050000000000000000 Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 18200 Size in Megabytes False
# lsattr -l hdisk5 -E
pvid none Physical volume identifier False
queue_depth 3 Queue DEPTH False
size_in_mb 9100 Size in Megabytes False
# lsattr -l hdisk6 -E
pvid none Physical volume identifier False
queue_depth 1 Queue DEPTH True
q_type none Queuing TYPE True
q_err yes Use QERR bit True
clr_q no Device CLEARS its Queue on error True
rw_timeout 30 READ/WRITE time out value True
start_timeout 60 START unit time out value True
reassign_to 120 REASSIGN time out value True
#

 

[mrn]

I'd be tempted to remove 1,5 & 6 then reboot before I started after reducing the rootvg.

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[dominicdiddy]

Guys,

Thanks for all the help - following the above steps, we completed the work (and went to the pub afterwards...!). The problem was with a rootvg mirror issue and a failed disk (probably not helped by a predecessor incorrectly removing a disk several times...)