HACMP problems after mksysb restore.

[jprabaker]

Since restoring from mksysb via NIM, we have problems starting cluster services on the 2nd node.

The OS is 5200-04 & HACMP is 5.2. The cluster has 2 cascading w/o failback resource groups, one prioritised to each node.

When attempting a clstart, I get a failure with the following messgae:

------------------
Starting Cluster Services on node: tru0056
This may take a few minutes. Please wait...
tru0056: start_cluster: Starting HACMP
tru0056: Changes detected on Node .
tru0056: Mar 31 2005 14:44:57 Starting execution of /usr/es/sbin/cluster/etc/rc.cluster
tru0056: with parameters: -boot -N -i
tru0056:
tru0056: Mar 31 2005 14:44:58
tru0056: rc.cluster: Error: Changes have been made to the Cluster Topology or Resource
tru0056: configuration. The Cluster Configuration must be synchronized before
tru0056: starting Cluster Services.
tru0056: cl_rsh had exit code = 1, see cspoc.log and/or clcomd.log for more information
------------------

I've tried running a sync from the running node but to no avail. The output from clcomd.log is as follows:

-------------------
Fri Apr 1 09:33:07 2005: Trying to establish connection to node tru0056
Fri Apr 1 09:33:07 2005: Trying to establish connection to node tru0055
Fri Apr 1 09:33:07 2005: Connection to node tru0056, success, 10.2.44.56->10.2.44.56
Fri Apr 1 09:33:07 2005: CONNECTION: ACCEPTED: tru0056: 10.2.44.56->10.2.44.56
Fri Apr 1 09:33:07 2005: COLLVER: ACCEPTED: tru0056: 10.2.44.56->10.2.44.56
Fri Apr 1 09:33:08 2005: Connection to node tru0055, success, 10.2.44.56->10.2.44.60
Fri Apr 1 09:33:09 2005: RSH: ACCEPTED: tru0056: 10.2.44.56->10.2.44.56
Fri Apr 1 09:33:09 2005: RSH: Command='/usr/es/sbin/cluster/cspoc/cexec start_cluster -boot -N -b -i',pid=15564
Fri Apr 1 09:33:09 2005: Connection Secmode = Standard
Fri Apr 1 09:33:09 2005: Connection to node tru0055: closed
Fri Apr 1 09:33:09 2005: Trying to establish connection to node tru0055
Fri Apr 1 09:33:09 2005: Connection to node tru0055, success, 10.2.44.56->10.2.44.60
Fri Apr 1 09:33:11 2005: RSH:COMPLETED: exit code = 1, pid=15564
Fri Apr 1 09:33:11 2005: Connection to node tru0056: closed
Fri Apr 1 09:33:11 2005: CONNECTION: CLOSED: tru0056: 10.2.44.56->10.2.44.56
Fri Apr 1 09:33:41 2005: Connection to node tru0055: closed
---------------

And the cspoc.log output is:

--------------
04/01/05 09:33:07 [========== C_SPOC COMMAND LINE ==========]
04/01/05 09:33:07 /usr/es/sbin/cluster/sbin/cl_rc.cluster -cspoc -n tru0056 -boot -N -b -i
04/01/05 09:33:08 cl_rc.cluster:
04/01/05 09:33:08 cl_rc.cluster: Starting Cluster Services on node: tru0056
04/01/05 09:33:08 cl_rc.cluster: This may take a few minutes. Please wait...
04/01/05 09:33:11 tru0056: FAILED: start_cluster -boot -N -b -i
04/01/05 09:33:11 tru0056: start_cluster: Starting HACMP
04/01/05 09:33:11 tru0056: Changes detected on Node .
04/01/05 09:33:11 tru0056: Apr 1 2005 09:33:09 Starting execution of /usr/es/sbin/cluster/etc/rc.cluster
04/01/05 09:33:11 tru0056: with parameters: -boot -N -b -i
04/01/05 09:33:11 tru0056:
04/01/05 09:33:11 tru0056: Apr 1 2005 09:33:11
04/01/05 09:33:11 tru0056: rc.cluster: Error: Changes have been made to the Cluster Topology or Resource
04/01/05 09:33:11 tru0056: configuration. The Cluster Configuration must be synchronized before
04/01/05 09:33:11 tru0056: starting Cluster Services.
04/01/05 09:33:11 tru0056: RETURN_CODE=1
04/01/05 09:33:11 tru0056: cl_rsh had exit code = 1, see cspoc.log and/or clcomd.log for more information
------------

Any suggestion would be gratrefully received.


Thanks,
John

 

[glj]

Hi John,i think that your topolgie Configuration is not
correct.
Try this:
- smitty hacmp
- extend Configuration
- Extended Verification and Synchronization
- set logging to verbose
No you can see more information during synch

good luck
glj

 

[DSMARWAY]

hi ,

it looks as if some changes on server 1 are different to that of server B , is your mksysbs created at the same time for both servers ?

As suggested in the log , run a synchronisation of the cluster
, first run it it emulate mode and note down and where possible fix the errors

it may also be a good idea to draw your cluster on paper as it should be so that you can go via , the setup starting from
1. cluster topology ( i.e. node names , ip addresses , serial heartbeat )
2. cluster resources
3. cluster application resources
4. stop and start script

check and make sure all HACMP start/stop scripts are the
same on both servers

HTH