client rootvg issues in dual VIO configuration

[unprophete]

LPAR gets two virtual hdisk's (VIO servers serves local [not shared SAN] drives - one from each VIOS).
LPAR's rootvg is mirrored on these drives.
When one VIO is halted, LPAR is still working with one disk missing in the rootvg and stale PPs.
Now when the VIOS is up again, "varyonvg rootvg" fails - can't activate the "missing disks" again and do the syncvg.


# varyonvg rootvg
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd8.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd4.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd2.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd9var.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd3.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd1.
0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume hd10opt.
0516-932 /usr/sbin/syncvg: Unable to synchronize volume group rootvg.

# lsvg -p rootvg
rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk0 missing 159 31 16..00..00..00..15
hdisk1 active 159 31 16..00..00..00..15

Right now, the only way to fix this online was:

# unmirrorvg rootvg hdisk0
# chpv -c hdisk0
# reducevg rootvg hdisk0
# rmdev -l vscsi0 -R
# cfgmgr
# extendvg -f rootvg hdisk0
# mirrorvg -m -s rootvg hdisk0
# varyonvg rootvg
# bosboot -a

Any other ideas? why the "varyonvg rootvg" sometimes helps in such case and sometimes as in example above not?

 

[mrn]

Have you tried enabling the path on the lpar?

lspath

chpath -l hdisk1 -s enable

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[p5wizard]

What is your AIX level on the client side?

Have you checked this?

HTH,

p5wizard

 

[unprophete]

client LPAR:
# oslevel -s
5300-11-03-1013

VIOS:
$ ioslevel
2.1.2.13-FP-22.1 SP-02


I didn't think to disable/enable the path to the disk - it was all the time in enabled state even when disks became not reachable (missing) after VIOS was halted (?).
lspath before and after VIOS was halted showed:

# lspath
Enabled hdisk0 vscsi0
Enabled hdisk1 vscsi1

Why the path didn't enter Failed state - I have no idea.

On clinet LPAR:
devices.vdevice.IBM.v-scsi.rte
5.3.11.0 C F Virtual SCSI Client Support

on VIOS:
devices.vdevice.IBM.v-scsi.rte
6.1.4.1 C F Virtual SCSI Client Support

 

[unprophete]

I will update devices.vdevice.IBM.v-scsi.rte to 5.3.10.1 and then check how the stuff works.

 

[unprophete]

sorry - my mistake. my system is/was already on the required level devices.vdevice.IBM.v-scsi.rte 5.3.11.0 ...

so the system shold not be affcted by the problem described on link P5wizard sent

 

[p5wizard]

OK, then. Did you try a [tt]cfgmgr[/tt] on client LPAR after the VIOS is rebooted. You're serving local disks on VIOS to client, so one path per disk only. Multipath I/O doesn't come into play (it won't change a path status if it is the last/only path of a disk).

You reboot one VIOS => disks that are served by that VIOS are declared MISSING by LVM on the client LPAR and are probably set to "Defined" in the ODM device database.

So you have to rediscover the disk on the client LPAR:
[tt]cfgmgr[/tt]
And then
[tt]varyonvg rootvg[/tt]


HTH,

p5wizard

 

[unprophete]

Yes, I tried cfgmgr but it failed... The error was:

# cfgmgr
cfgmgr: 0514-609 Unable to save the base customized information
on /dev/ipldevice.

Probably the missing disks was the disk system was booted from (was first on the bootlist)....

I guess if the second VIOS would be halted there would be a chanse that cfgmgr would worked (becasue hdisk1 would be missing - not the one system was boot from).

 

[MoreFeo]

You can try to hard link /dev/ipldevice to the other disk (to the raw device /dev/rhdiskX) and then run cfgmgr again.

 

[juredd1]

I am not intending to highjack this thread but thought I would ask how do you varyon the rootvg VG? If the system is up doesn't the rootvg VG already have to be varied on.

I don't know everything related to AIX that is why I am on this forum and asking this question. Does varyonvg do some repair to an already varied on VG? I did man on varyonvg but it was not clear to me.

Not sure if the commands you supplied in the first post were run or it it was just what you found to do but if you have reduced the disk from the VG can't you just remove the disk completely

# rmdev -dl hdisk0

If that is successful than just cfgmgr to bring it back.

Sorry if these had been run and what the results were, I just did not see any output.

I found this related link as well.

http://www.rootvg.net/content/view/174/89/

 

[p5wizard]

@juredd1

A varyonvg action on a VG that is already varyon-ed just reinstates any PVs that were declared MISSING and fires of a syncvg for mirrored LVs (if it works, that is ;-))


HTH,

p5wizard