Hi all,
My server (LPAR on P5 570 machine) gets shutdowned by itself and below is the errpt -a for the incident.
There is a cluster HACMP running on the system and this is the primary node of the cluster. So i came today and i found out that the standby node is having the shared (oravg from SAN) volume.
Any help would be appreciated.
Regards,
Khalid
My server (LPAR on P5 570 machine) gets shutdowned by itself and below is the errpt -a for the incident.
There is a cluster HACMP running on the system and this is the primary node of the cluster. So i came today and i found out that the standby node is having the shared (oravg from SAN) volume.
Any help would be appreciated.
Code:
---------------------------------------------------------------------------
LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6
Date/Time: Tue Jun 6 02:13:39 SAUST 2006
Sequence Number: 403
Machine Id: 00C5C1EB4C00
Node Id: localhost
Class: S
Type: TEMP
Resource Name: SYSPROC
Description
SYSTEM SHUTDOWN BY USER
Probable Causes
SYSTEM SHUTDOWN
Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
1
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE
Date/Time: Tue Jun 6 15:02:08 SAUST 2006
Sequence Number: 402
Machine Id: 00C5C1EB4C00
Node Id: localhost
Class: O
Type: TEMP
Resource Name: errdemon
Description
ERROR LOGGING TURNED ON
Probable Causes
ERRDEMON STARTED AUTOMATICALLY
User Causes
/USR/LIB/ERRDEMON COMMAND
Recommended Actions
NONE
---------------------------------------------------------------------------
LABEL: TS_NIM_ERROR_STUCK_
IDENTIFIER: 864D2CE3
Date/Time: Tue Jun 6 02:13:36 SAUST 2006
Sequence Number: 401
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: S
Type: PERM
Resource Name: topsvcs
Description
NIM thread blocked
Probable Causes
A thread in a Topology Services Network Interface Module (NIM) process
was blocked
Topology Services NIM process cannot get timely access to CPU
User Causes
Excessive memory consumption is causing high memory contention
Excessive disk I/O is causing high memory contention
Recommended Actions
Examine I/O and memory activity on the system
Reduce load on the system
Tune virtual memory parameters
Call IBM Service if problem persists
Failure Causes
Excessive virtual memory activity prevents NIM from making progress
Excessive disk I/O traffic is interfering with paging I/O
Recommended Actions
Examine I/O and memory activity on the system
Reduce load on the system
Tune virtual memory parameters
Call IBM Service if problem persists
Detail Data
DETECTING MODULE
rsct,nim_control.C,1.39.1.2,5492
ERROR ID
6XnGH40Ue9V2/LWT/T4U1/0...................
REFERENCE CODE
Thread which was blocked
send thread
Interval in seconds during which process was blocked
35
Interface name
rhdisk1
---------------------------------------------------------------------------
LABEL: OPMSG
IDENTIFIER: AA8AB241
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 400
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: O
Type: TEMP
Resource Name: OPERATOR
Description
OPERATOR NOTIFICATION
User Causes
ERRLOGGER COMMAND
Recommended Actions
REVIEW DETAILED DATA
Detail Data
MESSAGE FROM ERRLOGGER COMMAND
clexit.rc : Unexpected termination of clstrmgrES
---------------------------------------------------------------------------
LABEL: SRC_RSTRT
IDENTIFIER: BA431EB7
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 399
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: S
Type: PERM
Resource Name: SRC
Description
SOFTWARE PROGRAM ERROR
Probable Causes
APPLICATION PROGRAM
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
VERIFY SUBSYSTEM RESTARTED AUTOMATICALLY
Detail Data
SYMPTOM CODE
0
SOFTWARE ERROR CODE
-9035
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'217'
FAILING MODULE
emsvcs
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 398
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: S
Type: PERM
Resource Name: SRC
Description
SOFTWARE PROGRAM ERROR
Probable Causes
APPLICATION PROGRAM
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED
Detail Data
SYMPTOM CODE
1024
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
clstrmgrES
---------------------------------------------------------------------------
LABEL: HA002_ER
IDENTIFIER: 12081DC6
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 397
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: S
Type: PERM
Resource Name: haemd
Description
SOFTWARE PROGRAM ERROR
Probable Causes
SUBSYSTEM
Failure Causes
SUBSYSTEM
Recommended Actions
REPORT DETAILED DATA
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
DETECTING MODULE
LPP=PSSP,Fn=emd_gsi.c,SID=1.4.1.36,L#=1361,
DIAGNOSTIC EXPLANATION
haemd: 2521-032 Cannot dispatch group services (1).
---------------------------------------------------------------------------
LABEL: SRC_SVKO
IDENTIFIER: BC3BE5A3
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 396
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: S
Type: PERM
Resource Name: SRC
Description
SOFTWARE PROGRAM ERROR
Probable Causes
APPLICATION PROGRAM
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
MANUALLY RESTART SUBSYSTEM IF NEEDED
Detail Data
SYMPTOM CODE
2560
SOFTWARE ERROR CODE
-9017
ERROR CODE
0
DETECTING MODULE
'srchevn.c'@line:'350'
FAILING MODULE
grpsvcs
---------------------------------------------------------------------------
LABEL: GS_DOM_MERGE_ER
IDENTIFIER: 9DEC29E1
Date/Time: Tue Jun 6 02:13:33 SAUST 2006
Sequence Number: 395
Machine Id: 00C5C1EB4C00
Node Id: s2oraplp
Class: O
Type: PERM
Resource Name: grpsvcs
Description
Group Services daemon exit to merge domains
Probable Causes
Network between two node groups has repaired
Failure Causes
Network communication has been blocked.
Topology Services has been partitioned.
Recommended Actions
Check the network connection.
Check the Topology Services.
Verify that Group Services daemon has been restarted
Call IBM Service if problem persists
Detail Data
DETECTING MODULE
RSCT,NS.C,1.107.1.35,4370
ERROR ID
6Vb0vR0Re9V2/iRM/T4U1/0...................
REFERENCE CODE
DIAGNOSTIC EXPLANATION
The master requests to dissolve my domain because of the merge with other domain 1.9
---------------------------------------------------------------------------Regards,
Khalid