HACMP: Problem starting gsclvmd. varyonvg -c error

[mdraja]

Hi

I have created a disk heartbeat volumegroup. However, when I try to vary it on in concurrent mode, I receive the following error:

# varyonvg -c hbeatvg
srcsrqt failed errno : SRC_NSVR
Subsystem [gsclvmd] is not active
tellclvmd: request failed rc = -9036 [SRC_NSVR]
0516-1334 varyonvg: The command /usr/sbin/tellclvmd returned an error.

I tried starting gsclvmd, but nothing happens

I am using HACMP 5.3 on AIX 5.3 TL5. Incidentally, bos.clvm is installed (only thing that i have found on google pointing to a solution, which wasn't so in my case)

Any help greatly appreciated

 

[khalidaaa]

Thought this won't help much but have you tried to test the heartbeat when you configured it?

dhb_read - how to test a heartbeat device in an IBM AIX HACMP cluster 
Description

To test an IBM AIX HACMP heartbeat device:

1) Set the first node in receive mode (-r):

# /usr/sbin/rsct/bin/dhb_read -p device_name -r

2) Set the second node to transmit mode (-t):

# /usr/sbin/rsct/bin/dhb_read -p device_name -t  
Example

dhb_read -p hdisk1 -r; dhb_read -p hdisk1 -t

What errors you get in the /tmp/hacmp.out or errpt?

Regards,
Khalid

 

[alexia32]

Hi,

I have the same problem as you with varyonvg -c vgname it failed also on the second node.
I am running AIX 5.3 TL4CSP but do you have hacmp running ? because you need it to be started to be able to do the varyonvg -c

Cheers
Al

 

[mdraja]

Thanks for the feedback - unfortunately, the first thing it moans about is that "ERROR: Devices are not in Enhanced Concurrent Mode VG" when I try to start the cluster services. I found that this was related to the gsclvmd problem and hence, the inability to vayonvg the vg in concurrent mode

 

[khalidaaa]

Did you try stopping the cluster on all nodes and then start it on that node?

 

[mdraja]

Yes I did - I've narrowed it down to the following. When I remove the heartbeat disk as a communications device, restart cluster services, I can varyonvg my disk and sync the two nodes (cluster services work fine). However, when I add it back in as a communication device, this seems to screw up the cluster services and complains that the devices are not in enhanced concurrent mode, even though lspv shows the vg to be in concurrent mode.

 

[khalidaaa]

did you test the communication after adding the disk using dhb_read? (it should be done while the cluster is down)

what is your cluster configuration before adding and after adding the disk? cldisp

What's your procedure in adding the disk?

Regards,
Khalid

 

[mdraja]

Thanks Khalid

Using the transmit and read options from the dhb_read, everyting looks good:

root@purplehacmp# /usr/sbin/rsct/bin/dhb_read -p hdisk5 -r
DHB CLASSIC MODE
First node byte offset: 61440
Second node byte offset: 62976
Handshaking byte offset: 65024
Test byte offset: 64512

Receive Mode:
Waiting for response . . .
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Link operating normally
Secondary node: /usr/sbin/rsct/bin/dhb_read -p hdisk5 -t
root@orange# /usr/sbin/rsct/bin/dhb_read -p hdisk5 -t
DHB CLASSIC MODE
First node byte offset: 61440
Second node byte offset: 62976
Handshaking byte offset: 65024
Test byte offset: 64512

Transmit Mode:
Magic number = 0x87654321
Detected remote utility in receive mode. Waiting for response . . .
Magic number = 0x87654321
Magic number = 0x87654321
Link operating normally

I added the disk through normal smitty mkvg and also through the hacmp menus

Before adding the disk, the cldisp is:

Application server [] does not exist.
Cluster: firstcluster
Cluster services: active
State of cluster: up
Substate: stable

#############
APPLICATIONS
#############
Cluster firstcluster provides the following applications:
#############
TOPOLOGY
#############
firstcluster consists of the following nodes: orange purplehacmp
orange
Network interfaces:
orange {up}
with IP address: 10.40.44.85
on interface: en0
on network: net_ether_01 {up}
purplehacmp
Network interfaces:
purplehacmp {up}
with IP address: 10.40.44.84
on interface: en0
on network: net_ether_01 {up}

however after adding the disk, I get the following errors:

WARNING: Unable to locate interface 'hdisk5' on node 'orange'.

WARNING: An unknown problem occurred while executing the ip_of_interface subroutine

how come it is looking for the heartbeat disk 'hdisk5' as an interface, when it should be a device?

 

[khalidaaa]

Just a question did you run

Discover HACMP-related Information from Configured Nodes

and then

Extended Verification and Synchronization

on the node that you've added the disk on?

Did you add the hbdisk in the cluster? (simtty cm_config_hacmp_networks_menu_dmn)(eg: net_diskhb_01)

Did you configure the disk communication interface? (smitty cm_config_hacmp_communication_interfaces_devi
ces_menu_dmn)

having said that you've added the disk smitty mkvg, Did you import the vg on the other nodes? (I believe you did as you can using the heartbeat test right?)

Regards,
Khalid

 

[mdraja]

Thanks again for all the assistance Khalid

Yes to all the questions unfortunately

I suppose that this is all minor as I can always configure this witout a disk heartbeat, but its been bugging me why the disk won't allow this

 

[THOR01]

I had an issue where the <cluster.es.clvm.rte> file had a problem. IBM asked if I had created this node from a mksysb/alt_dsk install. We did create it from that so they stated that that file "sometimes" get hosed up when created from mksysb/alt_dsk install and that I needed to remove the file and reinstall it. It fixed the issues I was haveing creating a HB-disk