DMX integration with sun cluster + power path

[gubacs]

Hello,

I have a serious problem with the following sun cluster - EMC configuration:
Hardware:
2 x Sun Fire V490 with sun fibre channel adapter (x6768a similar with qlogic 2342)
EMC DMX 801

We connected directly the sun servers to DMX box. We would like to use the following software architecture:
Solaris 9 09/04
SAN foundation package
original QLC driver stack on the HBAs.
Veritas Volume Manager 4.0 MP1
Veritas File systems 4.0 MP1
Power Path 4.3.0 or 4.3.1
Time Finder..
Sun cluster 3.1 update 3

Both the local SUN and "EMC partner" officially said that this configuration is supported by both company!!!

But when I installed these softwares I got the following problems:
After installed Solaris, SFS, QLC, sun cluster 3.1 packages and patches I took out from the cluster install mode to normal mode. I configured quorum disk to EMC disks. It looked everything okay. After I installed vxvm 4.0 via scvxinstall and encapsule boot disk. It was okay. I put on the power path 4.3.0. I use native device all level. After reboot the vxvm doesn't work properly. The system goes to maintenance mode because vxvm can't start special volumes (ex. var, swap volumes). The error message is the following:
VXVM INFO V-5-2-3247 starting special volumes ( swapvol rootvol var ORCL rootdg_17vol )...
/dev/vx/dsk/bootdg/swapvol: No such device or address
...
The /var file system (/dev/vx/rdsk/bootdg/var ) is being checked.
Can't open /dev/vx/rdsk/bootdg/var
/dev/vx/rdsk/bootdg/var: CAN'T CHECK FILE SYSTEM.
...
Entering system MAintenance mode


After I removed power path, unencapsulated the boot disk, and encapsulated again. It was good until the next reboot... After the error turned out again.

The suppliers can't tell me any useful technical information.
I think they would like to obsolate their qualification, because the customer has already paid...

Does anybody any idea?

Br,
Gyorgy

 

[Chacalinc]

are the rootvg in the internal disks on the 2 servers?

 

[gubacs]

Yes, I use internal disks for boot.

 

[Chacalinc]

hmmm... that's quite weird, powerpath should be compatible with vxvm... may be the versions (both) are shocking with the DMP feature.

Did you see the compatibility matrix for powerpath and Volume manager? or did you try disabling DMP of vxvm?

 

[gubacs]

Yes I read it. vxvm 4.0MP1 - PP 4.3.[0,1] is supported by EMC & SUN (they wrote down in a qualification letter for my customer, although I feel that they started slip out of their statement...)

If you use sun cluster 3.1 with PP then you have to enable vxvm dmp feature (document: Sun EIS checklist for sun cluster 3.1 u3).

Tomorrow I will try to disable dmp,and maybe change the hba firmware and driver stack from QLC (ssd) to QLA (sd)(in most of cases QLA works well although sun doesn't like it)


G,

 

[nachgedacht]

hmmm, i found this in the emc support matrix, did you set this in /etc/system ?

30. For systems configured with Sun native HBAs and SAN driver, the following applies: Must add "set fcp:ssfcp_enable_auto_configuration=1" (do not include quote marks) in /etc/system to fix the following problems: 1. VxVM rootdg can not be imported during boot 2. SVM Volume Manager can not be started after system boot 3. PowerPath 4.x error message "All paths to (array type) (array name) vol(volume number) are dead"

maybe this solve your problem

by

 

[gubacs]

Hello,

Thanks,
I've solved the problem. There were some bug, one was in the scvxinstall script, other was in the box's microcode. The scvxinstall script changed the vx minor number badly. I think Sun issued a fix for this, and EMC issued a new microcode too.

 

[Chacalinc]

gubacs: thanks for post the solution, could you post the versions (vxvm and microcode) with problems?
nachgedacht was pointing out the same thing, I think.

Cheers.

 

[gubacs]

Hello,
The final solution is:
-Solaris 09/04
-Sun Cluster 3.1 u3,
scinstall -pv:
Sun Cluster 3.1u3 for Solaris 9 sparc
SUNWscr: 3.1.0,REV=2003.03.25.13.14, 113801-12 117949-08 115364-09 115571-03 117949-09
SUNWscu: 3.1.0,REV=2003.03.25.13.14, 113801-12 117949-08 115364-09 115571-03 117949-09
SUNWscsck: 3.1.0,REV=2003.09.10.18.59, 115953-03 115953-04
SUNWscnm: 3.1.0,REV=2004.10.02.17.53
SUNWscdev: 3.1.0,REV=2003.03.25.13.14, 113801-12 117949-08 117949-09
SUNWscgds: 3.1.0,REV=2003.03.25.13.14, 115061-01
SUNWscman: 3.1.0,REV=2003.03.25.13.14, 113801-12 117949-08 117949-09
SUNWscsal: 3.1.0,REV=2003.03.25.13.14, 113801-12 117949-08 117949-09
SUNWscsam: 3.1.0,REV=2003.03.25.13.14, 113801-12
SUNWscvm: 3.1.0,REV=2003.03.25.13.14, 113801-12
SUNWmdm: 3.1.0,REV=2003.03.25.13.14, 115053-03
SUNWscmasa: 3.1.0,REV=2004.10.02.17.53
SUNWscva: 3.1.0,REV=2003.03.25.13.14, 115055-01
SUNWscspm: 3.1.0,REV=2004.10.02.17.53
SUNWscspmu: 3.1.0,REV=2004.10.02.17.53
SUNWscspmr: 3.1.0,REV=2004.10.02.17.53
SUNWscrsm: 3.1.0,REV=2003.09.10.18.59, 113801-12 117949-08 117949-09
SUNWcsc: 3.1.0,REV=2004.08.11.14.11
SUNWcscspm: 3.1.0,REV=2004.08.11.14.11
SUNWcscspmu: 3.1.0,REV=2004.08.11.14.11
SUNWdsc: 3.1.0,REV=2004.08.11.14.04
SUNWdscspm: 3.1.0,REV=2004.08.11.14.04
SUNWdscspmu: 3.1.0,REV=2004.08.11.14.04
SUNWesc: 3.1.0,REV=2004.08.11.14.07
SUNWescspm: 3.1.0,REV=2004.08.11.14.07
SUNWescspmu: 3.1.0,REV=2004.08.11.14.07
SUNWfsc: 3.1.0,REV=2004.08.11.14.01
SUNWfscspm: 3.1.0,REV=2004.08.11.14.01
SUNWfscspmu: 3.1.0,REV=2004.08.11.14.01
SUNWhsc: 3.1.0,REV=2004.08.11.14.17
SUNWhscspm: 3.1.0,REV=2004.08.11.14.17
SUNWhscspmu: 3.1.0,REV=2004.08.11.14.17
SUNWjsc: 3.1.0,REV=2004.08.11.14.18
SUNWjscman: 3.1.0,REV=2004.08.11.14.18
SUNWjscspm: 3.1.0,REV=2004.08.11.14.18
SUNWjscspmu: 3.1.0,REV=2004.08.11.14.18
SUNWksc: 3.1.0,REV=2004.08.11.14.10
SUNWkscspm: 3.1.0,REV=2004.08.11.14.10
SUNWkscspmu: 3.1.0,REV=2004.08.11.14.10
SUNWscor: 3.1.0,REV=2003.04.02.15.55, 115077-06 116040-04
SUNWscsmb: 3.1.0,REV=03.14.03,16:39:12, 116390-05 118023-02

box microcode:
from 5670.81.76
to 5670.83.78
and plus a patch for other problem with 27105 patch number.

veritas vxvm 4.0 mp1 with 115217-03 patch.
Maybe there are newer patches from above, which solve the following bug:
When you run the scvxinstall script (come with sun cluster), which first install vxvm packages, after reminor the veritas device minor numbers. So the program did it incorrect way, and if you compare the ls -l /dev/vx/rootdg/*, and vxprint -va vol_name then you should check the major/minor numbers which have to equal! (from vxvm 4.0 the orig vxvm did reminoring and after scvxinstall too... )
There are 3 solution:
1. you should install vxvm without scvxinstall, and reminor manually
2. After the script make two reboot, you should stop the process, boot with -x and comment out S*vxinstall under /etc/rc2.d/S*scvxinstall.
3. Look the sunsolve, if they've done patch or other workaround for this bug.

Other:
If you use Time Finder, use separate quorum disk which there isn't under time finder control. Because sometimes it will be scsi reservation conflict and TF won't be work correctly in generally under restore process (BCV ->STD).

Anyway I threw out leadville driver stack to Qlogic 4.15.03 driver stack, because our customer didn't accept about 40-50 sec waiting time in qlc driver probe timeout and 1 minute in power path ssd_timeout when one path failed. Under qlogic (QLA) driver these timeout values will be 0 and it works correctly

br,
Gubacs