Unable to bread mirror on /usr

[linedancer888]

I have a bad disk in rootvg, hdisk0, and am unable to break the mirror for /usr
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd2 jfs 62 124 2 open/stale /usr

I've tried to resync, but it's unable to resynchronize. What else can I try?

 

[p5wizard]

post an lslv -m hd2 output please?

rmlvcopy hd2 1 hdisk0

should remove the stale mirror on hdisk0, but I'd like to verify that the other mirror is on an other disk...

also after that, I bet there are other LVs mirrored on that drive, you need to remove those too, and then replace the disk (look up the correct procedure)



HTH,

p5wizard

 

[linedancer888]

Here's the o/p of the lslv
[root]raymond:/home/klehr# lslv -m hd2
hd2:/usr
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0218 hdisk0 0220 hdisk1 0296 hdisk1
0002 0219 hdisk0 0221 hdisk1 0297 hdisk1
0003 0220 hdisk0 0222 hdisk1 0298 hdisk1
0004 0221 hdisk0 0223 hdisk1 0299 hdisk1
0005 0222 hdisk0 0224 hdisk1 0300 hdisk1
0006 0227 hdisk0 0225 hdisk1 0301 hdisk1
0007 0228 hdisk0 0226 hdisk1 0302 hdisk1
0008 0229 hdisk0 0227 hdisk1 0303 hdisk1
0009 0230 hdisk0 0228 hdisk1 0304 hdisk1
0010 0231 hdisk0 0229 hdisk1 0305 hdisk1
0011 0232 hdisk0 0230 hdisk1 0306 hdisk1
0012 0233 hdisk0 0231 hdisk1 0307 hdisk1
0013 0234 hdisk0 0232 hdisk1 0308 hdisk1
0014 0235 hdisk0 0233 hdisk1 0309 hdisk1
0015 0236 hdisk0 0234 hdisk1 0310 hdisk1
0016 0237 hdisk0 0235 hdisk1 0311 hdisk1
0017 0238 hdisk0 0236 hdisk1 0312 hdisk1
0018 0239 hdisk0 0237 hdisk1 0313 hdisk1
0019 0240 hdisk0 0238 hdisk1 0314 hdisk1
0020 0241 hdisk0 0239 hdisk1 0315 hdisk1
0021 0242 hdisk0 0240 hdisk1 0316 hdisk1
0022 0243 hdisk0 0241 hdisk1 0317 hdisk1
0023 0244 hdisk0 0242 hdisk1 0318 hdisk1
0024 0245 hdisk0 0243 hdisk1 0319 hdisk1
0025 0246 hdisk0 0244 hdisk1 0320 hdisk1
0026 0247 hdisk0 0245 hdisk1 0321 hdisk1
0027 0248 hdisk0 0246 hdisk1 0322 hdisk1
0028 0249 hdisk0 0247 hdisk1 0323 hdisk1
0029 0250 hdisk0 0248 hdisk1 0324 hdisk1
0030 0251 hdisk0 0249 hdisk1 0325 hdisk1
0031 0320 hdisk0 0250 hdisk1 0326 hdisk1
0032 0338 hdisk0 0251 hdisk1 0327 hdisk1
0033 0339 hdisk0 0252 hdisk1 0328 hdisk1
0034 0340 hdisk0 0253 hdisk1 0329 hdisk1
0035 0342 hdisk0 0254 hdisk1 0330 hdisk1
0036 0343 hdisk0 0255 hdisk1 0331 hdisk1
0037 0344 hdisk0 0256 hdisk1 0332 hdisk1
0038 0345 hdisk0 0257 hdisk1 0333 hdisk1
0039 0346 hdisk0 0258 hdisk1 0334 hdisk1
0040 0347 hdisk0 0259 hdisk1 0335 hdisk1
0041 0348 hdisk0 0260 hdisk1 0336 hdisk1
0042 0349 hdisk0 0261 hdisk1 0337 hdisk1
0043 0350 hdisk0 0262 hdisk1 0338 hdisk1
0044 0351 hdisk0 0263 hdisk1 0339 hdisk1
0045 0437 hdisk0 0264 hdisk1 0340 hdisk1
0046 0438 hdisk0 0265 hdisk1 0341 hdisk1
0047 0439 hdisk0 0266 hdisk1 0342 hdisk1
0048 0440 hdisk0 0267 hdisk1 0343 hdisk1
0049 0441 hdisk0 0268 hdisk1 0344 hdisk1
0050 0442 hdisk0 0269 hdisk1 0345 hdisk1
0051 0443 hdisk0 0270 hdisk1 0346 hdisk1
0052 0444 hdisk0 0271 hdisk1 0347 hdisk1
0053 0445 hdisk0 0272 hdisk1 0348 hdisk1
0054 0446 hdisk0 0273 hdisk1 0349 hdisk1
0055 0447 hdisk0 0274 hdisk1 0350 hdisk1
0056 0448 hdisk0 0275 hdisk1 0351 hdisk1
0057 0449 hdisk0 0276 hdisk1 0352 hdisk1
0058 0450 hdisk0 0277 hdisk1 0353 hdisk1
0059 0451 hdisk0 0278 hdisk1 0354 hdisk1
0060 0452 hdisk0 0279 hdisk1 0355 hdisk1
0061 0453 hdisk0 0280 hdisk1 0356 hdisk1
0062 0454 hdisk0 0281 hdisk1 0357 hdisk1

I did remove all the other copies
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 64 64 1 open/syncd N/A
hd8 jfslog 1 1 1 open/syncd N/A
hd4 jfs 3 3 1 open/syncd /
hd2 jfs 62 186 2 open/stale /usr
hd9var jfs 3 3 1 open/syncd /var
hd3 jfs 8 8 1 open/syncd /tmp
hd1 jfs 3 3 1 open/syncd /home


I tried everything I know to remove this last one, but everything has been unsuccessful. rmlvcopy , syncvg, migratepv all haven't worked.

 

[xmubeta]

Do you get any errpt message?

 

[linedancer888]

yes, we've been receiving errors on this drive since May, which is why we're replacing it.

LABEL: DISK_ERR1
IDENTIFIER: 21F54B38

Date/Time: Wed Jun 29 15:36:22
Sequence Number: 112824
Machine Id: 000200954C00
Node Id: raymond
Class: H
Type: PERM
Resource Name: hdisk0
Resource Class: disk
Resource Type: scsd
Location: 30-58-00-8,0
VPD:
Manufacturer................IBM
Machine Type and Model......DCHS04U
FRU Number..................76H2697
ROS Level and ID............36333633
Serial Number...............6822A24E
EC Level....................488638
Part Number.................93G3159
Device Specific.(Z0)........000002029F00003E
Device Specific.(Z1)........RAMR6063
Device Specific.(Z2)........09RI
Device Specific.(Z3)........98072
Device Specific.(Z4)........0001
Device Specific.(Z5)........22
Device Specific.(Z6)........E29611

Description
DISK OPERATION ERROR

Probable Causes
MEDIA

User Causes
MEDIA DEFECTIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Failure Causes
MEDIA
DISK DRIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A08 0000 2800 003B 10CF 0000 0100 0000 0102 0000 F000 0300 3B10 CF18 0000 0000
110B 0080 002F 0000 01D2 0000 0944 007C 007C 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 09BF 000E D440

 

[hirschaj]

You have 3 copies and 2 of them are on hdisk1. I would get down to just 2 copies and then try to break the mirror. Also, providing the exact error message will usually lead to a much quicker answer.


Jim Hirschauer

http://www.aixexpert.com

 

[linedancer888]

How do I remove that 3rd copy and why does the lspv show 2 copies?
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd2 jfs 62 186 2 open/stale /usr

 

[linedancer888]

I did remove the 3rd copy, but am still unable to break the mirror.
hd2:/usr
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0218 hdisk0 0220 hdisk1
0002 0219 hdisk0 0221 hdisk1
0003 0220 hdisk0 0222 hdisk1
0004 0221 hdisk0 0223 hdisk1
0005 0222 hdisk0 0224 hdisk1
0006 0227 hdisk0 0225 hdisk1
0007 0228 hdisk0 0226 hdisk1
0008 0229 hdisk0 0227 hdisk1
0009 0230 hdisk0 0228 hdisk1
0010 0231 hdisk0 0229 hdisk1
0011 0232 hdisk0 0230 hdisk1
0012 0233 hdisk0 0231 hdisk1
0013 0234 hdisk0 0232 hdisk1
0014 0235 hdisk0 0233 hdisk1
0015 0236 hdisk0 0234 hdisk1
0016 0237 hdisk0 0235 hdisk1
0017 0238 hdisk0 0236 hdisk1
0018 0239 hdisk0 0237 hdisk1
0019 0240 hdisk0 0238 hdisk1
0020 0241 hdisk0 0239 hdisk1
0021 0242 hdisk0 0240 hdisk1
0022 0243 hdisk0 0241 hdisk1
0023 0244 hdisk0 0242 hdisk1
0024 0245 hdisk0 0243 hdisk1
0025 0246 hdisk0 0244 hdisk1
0026 0247 hdisk0 0245 hdisk1
0027 0248 hdisk0 0246 hdisk1
0028 0249 hdisk0 0247 hdisk1
0029 0250 hdisk0 0248 hdisk1
0030 0251 hdisk0 0249 hdisk1
0031 0320 hdisk0 0250 hdisk1
0032 0338 hdisk0 0251 hdisk1
0033 0339 hdisk0 0252 hdisk1
0034 0340 hdisk0 0253 hdisk1
0035 0342 hdisk0 0254 hdisk1
0036 0343 hdisk0 0255 hdisk1
0037 0344 hdisk0 0256 hdisk1
0038 0345 hdisk0 0257 hdisk1
0039 0346 hdisk0 0258 hdisk1
0040 0347 hdisk0 0259 hdisk1
0041 0348 hdisk0 0260 hdisk1
0042 0349 hdisk0 0261 hdisk1
0043 0350 hdisk0 0262 hdisk1
0044 0351 hdisk0 0263 hdisk1
0045 0437 hdisk0 0264 hdisk1
0046 0438 hdisk0 0265 hdisk1
0047 0439 hdisk0 0266 hdisk1
0048 0440 hdisk0 0267 hdisk1
0049 0441 hdisk0 0268 hdisk1
0050 0442 hdisk0 0269 hdisk1
0051 0443 hdisk0 0270 hdisk1
0052 0444 hdisk0 0271 hdisk1
0053 0445 hdisk0 0272 hdisk1
0054 0446 hdisk0 0273 hdisk1
0055 0447 hdisk0 0274 hdisk1
0056 0448 hdisk0 0275 hdisk1
0057 0449 hdisk0 0276 hdisk1
0058 0450 hdisk0 0277 hdisk1
0059 0451 hdisk0 0278 hdisk1
0060 0452 hdisk0 0279 hdisk1
0061 0453 hdisk0 0280 hdisk1
0062 0454 hdisk0 0281 hdisk1

rmlvcopy hd2 1 hdisk0
0516-076 lreducelv: Cannot remove last good copy of stale partition.
Resynchronize the partitions with syncvg and try again.
0516-922 rmlvcopy: Unable to remove logical partition copies from
logical volume hd2.


syncvg -f -l hd2
0516-068 lresynclv: Unable to completely resynchronize volume. Run
diagnostics if neccessary.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd2.

 

[p5wizard]

Sounds like you had stale copies on hdisk1 to begin with...

perform following commands:

lspv hdisk0
lspv hdisk1

If you have "STALE PARTITIONS:" value other than 0 for hdisk1 then you could be in for trouble...

But if you can still access all files in /usr (stale partitions have no JFS data on them, all free blocks), then you can probably still make an mksysb (or if you have a recent enough mksysb) ==> reinstall (from mksysb) with new disk installed.


HTH,

p5wizard

 

[hirschaj]

Your rmlvcopy command is wrong. You need to specify the disk you want to remove, not the good disk. Change hdisk0 to hdisk1 and you should be fine.


Jim Hirschauer

http://www.aixexpert.com

 

[hirschaj]

Sorry about that last post, for some reason I had it in my head that hdisk1 was bad not hdisk0, please disregard :-(


Jim Hirschauer

http://www.aixexpert.com

 

[linedancer888]

hdisk1 does have a stale partition
lspv hdisk1
PHYSICAL VOLUME: hdisk1 VOLUME GROUP: rootvg
PV IDENTIFIER: 00020095b742fc38 VG IDENTIFIER 0002009516150f71
PV STATE: active
STALE PARTITIONS: 1 ALLOCATABLE: yes
PP SIZE: 8 megabyte(s) LOGICAL VOLUMES: 8
TOTAL PPs: 537 (4296 megabytes) VG DESCRIPTORS: 1
FREE PPs: 392 (3136 megabytes)
USED PPs: 145 (1160 megabytes)
FREE DISTRIBUTION: 107..43..27..107..108
USED DISTRIBUTION: 01..64..80..00..00

I did create a mksysb this morning, and this is a test box as of last week.

 

[hirschaj]

One other thing you can try is not specifying the hdisk at all, "rmlvcopy hd2 1". The OS will only keep the good copy, then you can use migratepv to move all good PP's over to the good disk if any are left on the bad disk.


Jim Hirschauer

http://www.aixexpert.com

 

[linedancer888]

Thanks Jim, I was able to break the mirror by not specifying the disk, but unable to migrate.

[root]raymond:/home/klehr# rmlvcopy hd2 1
[root]raymond:/home/klehr# lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 1 1 closed/syncd N/A
hd6 paging 64 64 1 open/syncd N/A
hd8 jfslog 1 1 1 open/syncd N/A
hd4 jfs 3 3 1 open/syncd /
hd2 jfs 62 62 1 open/syncd /usr
hd9var jfs 3 3 1 open/syncd /var
hd3 jfs 8 8 1 open/syncd /tmp
hd1 jfs 3 3 1 open/syncd /home

It looks like there is enough space on hdisk1
[root]raymond:/usr# lspv hdisk0
PHYSICAL VOLUME: hdisk0 VOLUME GROUP: rootvg
PV IDENTIFIER: 000200951a246649 VG IDENTIFIER 0002009516150f71
PV STATE: active
STALE PARTITIONS: 0 ALLOCATABLE: yes
PP SIZE: 8 megabyte(s) LOGICAL VOLUMES: 1
TOTAL PPs: 537 (4296 megabytes) VG DESCRIPTORS: 2
FREE PPs: 475 (3800 megabytes)
USED PPs: 62 (496 megabytes)
FREE DISTRIBUTION: 108..107..76..94..90
USED DISTRIBUTION: 00..00..31..13..18
[root]raymond:/usr# lspv hdisk1
PHYSICAL VOLUME: hdisk1 VOLUME GROUP: rootvg
PV IDENTIFIER: 00020095b742fc38 VG IDENTIFIER 0002009516150f71
PV STATE: active
STALE PARTITIONS: 0 ALLOCATABLE: yes
PP SIZE: 8 megabyte(s) LOGICAL VOLUMES: 7
TOTAL PPs: 537 (4296 megabytes) VG DESCRIPTORS: 1
FREE PPs: 454 (3632 megabytes)
USED PPs: 83 (664 megabytes)
FREE DISTRIBUTION: 107..43..89..107..108
USED DISTRIBUTION: 01..64..18..00..00

[root]# migratepv -l hd2 hdisk0 hdisk1
0516-015 lmigratelv: The migrate was done but the partition
could not be resynced.
0516-812 migratepv: Warning, migratepv did not completely succeed;
all physical partitions have not been moved off the PV.

 

[hirschaj]

Please show output from "lslv -l hd2"


Jim Hirschauer

http://www.aixexpert.com

 

[linedancer888]

lslv -l hd2
hd2:/usr
PV COPIES IN BAND DISTRIBUTION
hdisk0 062:000:000 50% 000:000:031:013:018
hdisk1 062:000:000 100% 000:000:062:000:000

 

[hirschaj]

That is very strange, you should only have 62 PP's total now but it is showing double that still. Getting this disk out could be kind of risky, since you have a mksysb already you might want to try a reboot to clean up the system, it seems to be confused. Maybe it will show the proper information after it reboots (if it reboots).


Jim Hirschauer

http://www.aixexpert.com

 

[Yegolev]

I hope I have followed along correctly. I'd try a syncvg to un-stale things... need good data on hdisk1, you know. lsvg -l rootvg will tell you which lv is stale, then you do syncvg -l lvname. After that you should be able to remove the copies from hdisk0, assuming it works. Good luck.

IBM Certified -- AIX 4.3 Obfuscation