systemdump 888 102 700 0c0 2

[arvibm]

Hi,

Today our one server got hanged with LED Code 888 102 700 0c0.I have rebooted the server and analysed system dump to check for reason of system dump but could not be able to fine exact cause of problem.Pls help me to find the reason of systemdump.attaching the errorlog and dumpanalysis .

errorlog(errpt -a)

LABEL: JFS_FS_FRAGMENTED
IDENTIFIER: 5DFED6F1

Date/Time: Sat Sep 6 22:14:02 IST
Sequence Number: 65787
Machine Id: 0050F60A4C00
Node Id: ibm1
Class: O
Type: INFO
Resource Name: SYSPFS

Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

Probable Causes
FILE SYSTEM FREE SPACE FRAGMENTED

Recommended Actions
CONSOLIDATE FREE SPACE USING DEFRAGFS UTILITY

Detail Data
MAJOR/MINOR DEVICE NUMBER
0000 000F
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/prodarch_lv, /prodarch
---------------------------------------------------------------------------
LABEL: CASD_UP_IN
IDENTIFIER: C092AFE4

Date/Time: Sat Sep 6 22:05:09 IST
Sequence Number: 65786
Machine Id: 0050F60A4C00
Node Id: ibm1
Class: O
Type: INFO
Resource Name: ctcasd
Description
ctcasd Daemon Started

Probable Causes
ctcasd Daemon Started

Failure Causes
None

Recommended Actions
None required

Detail Data
DETECTING MODULE
rsct.core.sec,ctcas_main.c,1.13,295
ERROR ID
6YzeY.1RZdKz.Z0t0J.kj8....................
REFERENCE CODE
LABEL: RMCD_INFO_0_ST
IDENTIFIER: A6DF45AA

Date/Time: Sat Sep 6 22:05:09 IST
Sequence Number: 65785
Machine Id: 0050F60A4C00
Node Id: ibm1
Class: O
Type: INFO
Resource Name: RMCdaemon

Description
The daemon is started.

Probable Causes
The Resource Monitoring and Control daemon has been started.

User Causes
The startsrc -s ctrmc command has been executed or
the rmcctrl -s command has been executed.

Recommended Actions
Confirm that the daemon should be started.

Detail Data
DETECTING MODULE
RSCT,rmcd.c,1.34,196
ERROR ID
6eKora0RZdKz.7gf0J.kj8....................
REFERENCE CODE

---------------------------------------------------------------------------
LABEL: DUMP_STATS
IDENTIFIER: C0AA5338

Date/Time: Sat Sep 6 22:03:27 IST
Sequence Number: 65783
Machine Id: 0050F60A4C00
Node Id: localhost
Class: S
Type: UNKN
Resource Name: SYSDUMP
Description
SYSTEM DUMP

Probable Causes
UNEXPECTED SYSTEM HALT

User Causes
SYSTEM DUMP REQUESTED BY USER

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Failure Causes
UNEXPECTED SYSTEM HALT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
DUMP DEVICE
/dev/lg_dumplv
DUMP SIZE
228281856
TIME
Sat Sep 6 21:03:55 2003
DUMP TYPE (1 = PRIMARY, 2 = SECONDARY)
1
DUMP STATUS
0
ERROR CODE
0
FILE NAME

PROCESSOR ID
2
LABEL: JFS_META_CORRUPTION
IDENTIFIER: 684A365B

Date/Time: Sat Sep 6 21:03:51 IST
Sequence Number: 65782
Machine Id: 0050F60A4C00
Node Id: localhost
Class: U
Type: UNKN
Resource Name: SYSPFS
Resource Class: NONE
Resource Type: NONE
Location: NONE
VPD:

Description
FILE SYSTEM CORRUPTION

Probable Causes
INVALID FILE SYSTEM CONTROL DATA

Recommended Actions
PERFORM FULL FILE SYSTEM RECOVERY USING FSCK UTILITY
OBTAIN DUMP
CHECK ERROR LOG FOR ADDITIONAL RELATED ENTRIES

Failure Causes
ADAPTER HARDWARE OR MICROCODE
DISK DRIVE HARDWARE OR MICROCODE
SOFTWARE PROGRAM
STORAGE CABLE LOOSE, DEFECTIVE, OR UNTERMINATED

Recommended Actions
CHECK CABLES AND THEIR CONNECTIONS
INSTALL LATEST ADAPTER AND DRIVE MICROCODE
INSTALL LATEST STORAGE DEVICE DRIVERS
IF PROBLEM PERSISTS, CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
FILE NAME
v_mapsubs.c
LINE NO.
LINE NO.
416
MAJOR/MINOR DEVICE NUMBER
0000 000F
ADDITIONAL INFORMATION
4A46 5345 3F5A 8AFF 0000 001B 0000 002E 0003 1215 0000 0000 0000 0000 0000 000C
F100 009E 1809 F318 0000 41EE 0000 0000 000C 2C2D 0000 0000 0000 0581 0000 0000
0012 0E72 0000 00F6 0000 3E41 000A 6D31 0000 0000 0000 0000 0000 0000 0000 0000
---------------------------------------------------------------------------
LABEL: ERRLOG_ON
IDENTIFIER: 9DBCFDEE

Date/Time: Sat Sep 6 22:04:14 IST
Sequence Number: 65781
Machine Id: 0050F60A4C00
Node Id: localhost
Class: O
Type: TEMP
Resource Name: errdemon

Description
ERROR LOGGING TURNED ON

Probable Causes
ERRDEMON STARTED AUTOMATICALLY

User Causes
/USR/LIB/ERRDEMON COMMAND

Recommended Actions
NONE

---------------------------------------------------------------------------
LABEL: JFS_FS_FRAGMENTED
IDENTIFIER: 5DFED6F1

Date/Time: Sat Sep 6 15:40:46 IST
Sequence Number: 65780
Machine Id: 0050F60A4C00
Node Id: ibm1
Class: O
Type: INFO
Resource Name: SYSPFS

Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

Probable Causes
FILE SYSTEM FREE SPACE FRAGMENTED

Recommended Actions
CONSOLIDATE FREE SPACE USING DEFRAGFS UTILITY

Detail Data
MAJOR/MINOR DEVICE NUMBER
0000 000F
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/prodarch_lv, /prodarch


System dump analysis(using kdb)
------------------------------

# sysdumpdev -L
0453-039

Device name: /dev/lg_dumplv
Major device number: 10
Minor device number: 10
Size: 228281856 bytes
Uncompressed Size: 1528693647 bytes
Date/Time: Sat Sep 6 21:03:55 IST 2003
Dump status: 0
dump completed successfully
#
#
# kdb
The specified kernel file is a 64-bit kernel
Preserving 951886 bytes of symbol table
First symbol __mulh
START END <name>
0000000000003500 0000000002403068 _system_configuration+000020
F00000002FF3A600 F00000002FFCF8C8 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F100008780000000 F100008790000000 pvproc+000000
F100008790000000 F100008794000000 pvthread+000000
F100000040000000 F10000004028D4E8 vmmdseg+000000
F1000013B0000000 F1000083B5400000 vmmswpft+000000
F100000BB0000000 F1000013B0000000 vmmswhat+000000
F100000050000000 F100000060000000 ptaseg+000000
F100000070000000 F1000000B0000000 ameseg+000000
F100009E10000000 F100009E20000000 KERN_heap+000000
F1000089C0000000 F1000089D0000000 MBUF_heap+000000
F100009C00000000 F100009C10000000 lkwseg+000000
PFT:
id....................0007
raddr.....0000000010000000 eaddr.....0000000010000000
size..............10000000 align.............10000000
valid..1 ros....0 holes..0 io.....0 seg....1 wimg...2

PVT:
id....................0008
raddr.....0000000002081000 eaddr.....0000000000000000
size..............01C00000 align.............00001000
valid..1 ros....0 holes..0 io.....0 seg....1 wimg...2
(0)> dla
No deadlock found
(0)> vfs3
invalid command
(0)> vfs 3
GFS DATA TYPE FLAGS

3 KERN_heap+A0CE530 0052B1C8 F100009E3A479078 JFS2 DEVMOUNT
... /dev/hd9var mounted over /var

vfs_next..... F100009E1A0CE410 vfs_gfs...... 000000000052B1C8
vfs_mntd..... F100009E41E12020 vfs_mntdover. F100009E1BE11E40
vfs_vnodes... F100009E511950E0 vfs_count.... 00000001
vfs_data..... F100009E3A479078 vfs_number... 00000004
vfs_mdata.... F100009E1A0CD980 vfs_bsize.... 00001000
vfs_lock@.... F100009E1A0CE580 vfs_lock..... 0000000000000000
vmt_revision. 00000001 vmt_length... 0000007C vmt_vfsnumber 00000004
vmt_fsid..... 8000000A00000006 0000000000000000
vmt_time..... 3F5A98F5 vmt_flags.... 00000004
vmt_gfstype.. 00000000
vfs_type..... 00000000 vfs_ops...... j2_vfsops
@vmt_data.... F100009E1A0CD9AC
OBJECT .... /dev/hd9var
STUB ...... /var
HOST ...... -
HOSTNAME .. -
INFO ...... 80 00 00 0A 00 00 00 03
(0)> more (^C to quit) ? Interrupted
(0)> trace -k
expected address
(0)> status
CPU TID TSLOT PID PSLOT PROC_NAME
0 13E03B 159 D2014 105 kdb_64
1 6007 3 6006 3 wait
2 8009 4 8008 4 wait
3 A00B 5 A00A 5 wait
(0)>
(0)>
(0)> thread
SLOT NAME STATE TID PRI RQ CPUID CL WCHAN

pvthread+006500 202 NONE 1940FB 03C ?? 07C08 0

NAME................
WTYPE............... WZOMB
.................tid :00000000001940FB ......tsleep :FFFFFFFFFFFFFFFF
...............flags :00000000 ..............flags2 :00000000
DATA.........pvprocp :0000000000000000
LINKS.....prevthread :F100008790007780 <pvthread+007780>
..........nextthread :F100008790006500 <pvthread+006500>
DISPATCH.......synch :FFFFFFFFFFFFFFFF
SCHEDULER...affinity :00000000 .................pri :0000003C
.............boosted :00000000 ...............wchan :0000000000000000
...............state :00000000 ...............wtype :0000000C
CHECKPOINT......vtid :00000000 .............chkfile :0000000000000000
LOCK........ lock_d @ F100008790006510 0000000000000000
PROCFS......procfsvn :0000000000000000
THREAD.......threadp :0000000000000000 ........size :00000080

.................tid :0000000000000000 ......stackp :0000000000000000
.................scp :0000000000000000 .......ulock :0000000000000000
...............uchan :0000000000000000 ....userdata :0000000000000000
..................cv :0000000000000000 .......flags :0000000000000000
..............atomic :0000000000000000 ......flags2 :0000000000000000
DATA...........procp :0000000000000000
...........pvthreadp :0000000000000000
...............userp :0000000000000000
............uthreadp :0000000000000000
SLEEP/LOCK......usid :0000000000000000 ......wchan1 :0000000000000000
..............wchan2 :0000000000000000 ......swchan :0000000000000000
...........eventlist :0000000000000000 ......result :00000000
.............polevel :00000000 ..............pevent :0000000000000000
..............wevent :0000000000000000 .......slist :0000000000000000
...........wchan1sid :0000000000000000 wchan1offset :00000000
...........lockcount :00000000 ..........adsp_flags :E821
DISPATCH.......ticks :00000000 ...............prior :0000000000000000
................next :0000000000000000 ......dispct :7C34FBA64C00012C
...............fpuct :7C3FFAA678211F24
MISC........graphics :0000000000000000 ...pmcontext :00007FF800000000
...........lockowner :0000000000000000 ..kthreadseg :0000000000000000
..........time_start :7C34FAA67821C000 ..........wlm_charge :0
SIGNAL........sigproc:00000003 ..............cursig :000000A6
......(pending) sig :[3] 3821FF0080600184
......................[2] E8210000F861FFF8
......................[1] 642107C86021E800
......................[0] 7C3FFAA678211F24 QUIT ABRT KILL BUS SEGV SYS PIPE
...................... STOP TTOU WINCH PWR USR1 USR2 VTALRM
...................... MIGRATE ALRM1 CPUFAIL GRANT RETRACT SOUND SAK
............sigmask :[3] 07C8DC1807D07CF0
......................[2] 07C8800007D0772C
......................[1] 4E80002007C8D91C
......................[0] 7C6803A68040018C QUIT ILL FPE KILL IO PROF VTALRM
...................... MIGRATE ALRM1 CPUFAIL GRANT RETRACT SOUND SAK
SCHEDULER......cpuid :00007C08 ..............scpuid :000002A6
.........affinity_ts :7C0004AC ..............policy :00000000
.................cpu :00000100 .............lockpri :00000001
.............wakepri :00000090 ................time :0000007C
.............sav_pri :00000008 ...........run_queue :600000007C3243A6
................cpu2 :0000F801
.............suspend :00000000 .............fsflags :00000000
..........norun_secs :00000080
CHECKPOINT..chkerror :0000 ............chkblock :00000000
PROCFS.......whystop :0000004E ............whatstop :00000000
............pri_band :F8 ........allowed_cpus :642107C86021E000
.......prefunnel_cpu :00000000
......threadcontrolp :0000000000000000
PVTHREAD...pvthreadp :0000000000000000 ........size :00000080
(0)>
(0)>
(0)>
(0)> thread -r
SLOT NAME STATE TID PRI RQ CPUID CL WCHAN

pvthread+000100 2>wait RUN 004005 0FF 0 00000 0
pvthread+000180 3!wait RUN 006007 0FF 1 00001 0
pvthread+000200 4!wait RUN 008009 0FF 2 00002 0
pvthread+000280 5!wait RUN 00A00B 0FF 3 00003 0
pvthread+004F80 159!kdb_64 RUN 13E03B 03C 0 65
(0)>
(0)> trace -h
Channel 0 (@ 00242258)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0000 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 00242288
Channel 1 (@ 00242488)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0001 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 002424B8
Channel 2 (@ 002426B8)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0002 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 002426E8
Channel 3 (@ 002428E8)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0003 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 00242918
Channel 4 (@ 00242B18)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0004 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 00242B48
Channel 5 (@ 00242D48)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0005 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 00242D78
Channel 6 (@ 00242F78)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0006 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 00242FA8
Channel 7 (@ 002431A8)
state.......... 00000000 lockword....... FFFFFFFFFFFFFFFF
logio tid...... 00000000 buf ptr........ 00000000
#bufs.......... 00000000 #bufs finished. 00000000
i/o err........ 00000000 trconcount..... 0007 fileflags...... 0000
mode........... 0000 channel........ 0000 events....... @ 002431D8
(0)>
(0)>
(0)> vmlog
Most recent VMM errorlog entry
Error id =
Exception DSISR/ISISR = 0000000000000000
Exception srval = 0000000000000000
Exception virt addr = 0000000000000000
Exception value = 00000000
(0)>
(0)>
(0)>
(0)> dev
Slot address F100009E14074080
MAJOR: 001
open: .syopen (004CB3A8)
close: .nulldev (000AA0A0)
read: .syread (004CB11C)
write: .sywrite (004CAE8C)
ioctl: .syioctl (004CAA44)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .syselect (004CA9B0)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074100
MAJOR: 002
open: .nulldev (000AA0A0)
close: .nulldev (000AA0A0)
read: .mmread (004CA1B8)
write: .mmwrite (004C9EF0)
ioctl: .nodev (000AA0C4)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000006A DEV_DEFINED DEV_MPSAFE DEV_MEM

Slot address F100009E14074180
MAJOR: 003
open: .mdopen (004C88A0)
close: .mdclose (004C8844)
read: .mdread (004C80B0)
write: .nodev (000AA0C4)
ioctl: .mdioctl (004C8128)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .mdinit (0057F6A0)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .mdmpx (004C8004)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 00000022 DEV_DEFINED

Slot address F100009E14074200
MAJOR: 004
open: .conopen (004CCA28)
close: .conclose (004CC8E0)
read: .conread (004CC1B0)
write: .conwrite (004CC2BC)
ioctl: .conioctl (004CBDF0)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .conselect (004CBF5C)
config: .conconfig (004CCD1C)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .conmpx (004CC80C)
revoke: .conrevoke (004CC07C)
dsdptr: 00000000
selptr: 00000000
opts: 0000002E DEV_DEFINED CONS_DEFINED DEV_MPSAFE

Slot address F100009E14074280
MAJOR: 005
open: .nodev (000AA0C4)
close: .nodev (000AA0C4)
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: .nodev (000AA0C4)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074300
MAJOR: 006
open: .erropen (0053F908)
close: .errclose (0053FE28)
read: .errread (0004E8EC)
write: .errwrite (0053FC28)
ioctl: .errioctl (0053FA34)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074380
MAJOR: 007
open: .dmpopen (004C926C)
close: .dmpclose (004C9244)
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: .dmpioctl (004C9694)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .dmpdump (004C8A40)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074400
MAJOR: 008
open: .auditopen (004CE498)
close: .auditclose (004CE3C8)
read: .auditread (004CDD18)
write: .nodev (000AA0C4)
ioctl: .auditioctl (004CDF70)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .auditmpx (004CE0C8)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 00000002 DEV_DEFINED

Slot address F100009E14074480
MAJOR: 009
open: .nodev (000AA0C4)
close: .nodev (000AA0C4)
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: .nodev (000AA0C4)
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 00000002 DEV_DEFINED

Slot address F100009E14074500
MAJOR: 00A
open: 0251177C
close: 02511144
read: 02510874
write: 0251079C
ioctl: 0250F208
strategy: 025210A0
ttys: 00000000
select: .nodev (000AA0C4)
config: 02506068
print: .nodev (000AA0C4)
dump: 02529640
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E1BDFA000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074580
MAJOR: 00B
open: 0245D73C
close: 0245D658
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 0245CEE4
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074600
MAJOR: 00C
open: 0246CA1C
close: 0246C97C
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 0246C3A0
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074680
MAJOR: 00D
open: 0255A474
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074700
MAJOR: 00E
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074780
MAJOR: 00F
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074800
MAJOR: 010
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074880
MAJOR: 011
open: 024940EC
close: 02497044
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 0249545C
strategy: 0249C618
ttys: 00000000
select: .nodev (000AA0C4)
config: 024965E4
print: .nodev (000AA0C4)
dump: 024A8398
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074900
MAJOR: 012
open: 025B8094
close: 025B7E7C
read: 025B7D74
write: 025B7C6C
ioctl: 025B7190
strategy: 025BF1E0
ttys: 00000000
select: .nodev (000AA0C4)
config: 025B8ABC
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074980
MAJOR: 013
open: 025E6B10
close: 025E6A44
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 025E6E54
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 025E5658
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E1BE03000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074A00
MAJOR: 014
open: 0261A104
close: 02619FF0
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 02619EB0
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 02614704
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: 0261A228
revoke: .nodev (000AA0C4)
dsdptr: F100009E14071980
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074A80
MAJOR: 015
open: 02622810
close: 026226DC
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 02622328
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 0261DBFC
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E14071980
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074B00
MAJOR: 016
open: 0262B238
close: 0262AA18
read: 02629B70
write: 02628DDC
ioctl: 0262748C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 0262C084
print: .nodev (000AA0C4)
dump: 02630D18
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074B80
MAJOR: 017
open: 024D36D0
close: 024D3148
read: 024D3038
write: 024D2F28
ioctl: 024D187C
strategy: 024E1AE8
ttys: 00000000
select: .nodev (000AA0C4)
config: 024D4564
print: .nodev (000AA0C4)
dump: 024E146C
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074C00
MAJOR: 018
open: 024B9220
close: 024B8178
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 024B7BC4
strategy: 024C4900
ttys: 00000000
select: .nodev (000AA0C4)
config: 024B9C0C
print: .nodev (000AA0C4)
dump: 024BD38C
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074C80
MAJOR: 019
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: 0266C74C
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074D00
MAJOR: 01A
open: 026244E4
close: 02624258
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 02623FF4
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 02624A5C
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074D80
MAJOR: 01B
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074E00
MAJOR: 01C
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074E80
MAJOR: 01D
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14074F00
MAJOR: 01E
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075000
MAJOR: 020
open: 0267B90C
close: 0267B6D4
read: .nodev (000AA0C4)
write: .nodev (000AA0C4)
ioctl: 026921FC
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 0267AE40
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: 0267B630
revoke: .nodev (000AA0C4)
dsdptr: F100009E8AD8CA00
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075080
MAJOR: 021
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075100
MAJOR: 022
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075180
MAJOR: 023
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075200
MAJOR: 024
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075280
MAJOR: 025
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075300
MAJOR: 026
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075380
MAJOR: 027
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075400
MAJOR: 028
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075480
MAJOR: 029
open: 02555B80
close: 02554D2C
read: 0255453C
write: 0255433C
ioctl: 0255283C
strategy: .nodev (000AA0C4)
ttys: 00000000
select: 02551FE0
config: .nodev (000AA0C4)
print: .nodev (000AA0C4)
dump: .nodev (000AA0C4)
mpx: .nodev (000AA0C4)
revoke: 02551EA0
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075500
MAJOR: 02A
open: 024EB148
close: 024E9CF0
read: 024E9590
write: 024E9270
ioctl: 024E6F10
strategy: 024F8054
ttys: 00000000
select: .nodev (000AA0C4)
config: 024EBF50
print: .nodev (000AA0C4)
dump: 024F7B58
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075580
MAJOR: 02B
open: 02644D90
close: 026442C4
read: 02643488
write: 02642724
ioctl: 02639BB4
strategy: .nodev (000AA0C4)
ttys: 00000000
select: .nodev (000AA0C4)
config: 02646380
print: .nodev (000AA0C4)
dump: 0264A634
mpx: 02639B0C
revoke: .nodev (000AA0C4)
dsdptr: 00000000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075600
MAJOR: 02C
open: 0251177C
close: 02511144
read: 02510874
write: 0251079C
ioctl: 0250F208
strategy: 025210A0
ttys: 00000000
select: .nodev (000AA0C4)
config: 02506068
print: .nodev (000AA0C4)
dump: 02529640
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E8ACEA000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075700
MAJOR: 02E
open: 0251177C
close: 02511144
read: 02510874
write: 0251079C
ioctl: 0250F208
strategy: 025210A0
ttys: 00000000
select: .nodev (000AA0C4)
config: 02506068
print: .nodev (000AA0C4)
dump: 02529640
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E8AF38000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

Slot address F100009E14075780
MAJOR: 02F
open: 0251177C
close: 02511144
read: 02510874
write: 0251079C
ioctl: 0250F208
strategy: 025210A0
ttys: 00000000
select: .nodev (000AA0C4)
config: 02506068
print: .nodev (000AA0C4)
dump: 02529640
mpx: .nodev (000AA0C4)
revoke: .nodev (000AA0C4)
dsdptr: F100009E8AF36000
selptr: 00000000
opts: 0000002A DEV_DEFINED DEV_MPSAFE

(0)>
(0)>
(0)>
(0)>
(0)> [2] + Stopped (SIGTSTP) kdb
#
#
#


Pls help me to find out the exact reason of system dump.


Regards

arvibm

 

[unixtechie]

You have more useless info in your dump analysis than what you really need. It shows cpu0 which is kdb_64 which I wonder about, because kdb replaces crash from 4.3 to 5L; you have nothing in the stack trace to show the kernel extensions; you didn't run od proglog 8a; you didn't run vmmerrlog; you didn't run on tslot 159.

From the errpt it says meta data corruption which could cause a system crash. That is the best guess without full info.

Since you are on AIX5, why not call IBM and have them look at it?

 

[arvibm]

Hi unixtechie ,

pls tell me How to resolve the issue of meta data corruption.i tried the commands with kdb but the command dosen't work.

(0)> vmmerrlog
invalid command
(0)> vmlog
Most recent VMM errorlog entry
Error id =
Exception DSISR/ISISR = 0000000000000000
Exception srval = 0000000000000000
Exception virt addr = 0000000000000000
Exception value = 00000000

(0)> tslot
invalid command
(0)> tslot 159
invalid command
(0)> prodlog
invalid command


Regards

arvibm

 

[MoshiachNow]

Hi,

I would also look at output of the following:

1.diag ->diagnostics rooutines ->problem determination
This will analyse the eerpt for HW errors,that can be assotiated with the system dump.

2.errpt
Just lists all events around the time of dump,gives a better idea.

3.If it IS a meta data corruption (if it's a Fs corruption - try fixing it with fsck)- the way I used to look for it was either copying the whole FS data to another FS and wtaching on which file it will crash,or - running a full backup (Legato,etc) and checking on which folder/file it failed.
Then you can delete the problematic file/folder.

4.within the kdb run and post:
thread -r


&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[arvibm]

Hi,

I have used diag to check for HW errors but it gives no troubles found.

Here is output of errpt:-

# errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
5DFED6F1 0906221403 I O SYSPFS UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
C092AFE4 0906220503 I O ctcasd ctcasd Daemon Started
A6DF45AA 0906220503 I O RMCdaemon The daemon is started.
C0AA5338 0906220303 U S SYSDUMP SYSTEM DUMP
684A365B 0906210303 U U SYSPFS FILE SYSTEM CORRUPTION
9DBCFDEE 0906220403 T O errdemon ERROR LOGGING TURNED ON
5DFED6F1 0906154003 I O SYSPFS UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
1F0B7B49 0905222703 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1F0B7B49 0904234003 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1F0B7B49 0904233903 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1F0B7B49 0903225403 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1F0B7B49 0903021503 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED

#




(0)> thread -r
SLOT NAME STATE TID PRI RQ CPUID CL WCHAN

pvthread+000100 2>wait RUN 004005 0FF 0 00000 0
pvthread+000180 3>wait RUN 006007 0FF 1 00001 0
pvthread+000200 4>wait RUN 008009 0FF 2 00002 0
pvthread+000280 5>wait RUN 00A00B 0FF 3 00003 0
pvthread+005900 178*kdb_64 RUN 1640E7 03D 0 65
pvthread+010D80 539!oracle RUN 436023 072 1 65
pvthread+012180 579!oracle RUN 486059 072 2 65
pvthread+012200 580!oracle RUN 488099 072 3 65

Here is the output of dump device details:-
# sysdumpdev -l
primary /dev/lg_dumplv
secondary /dev/sysdumpnull
copy directory /var/adm/ras
forced copy flag TRUE
always allow dump TRUE
dump compression ON


Now, the server is running perfectly normal and all the database and application are running without any problem but i just want to know what wrong happened at that time.

waiting for ur reply...

Regards

arvibm

 

[arvibm]

Hi
Here is output of diag>diagnostic routine>proble determination:-

TESTING COMPLETE on Mon Sep 8 09:53:57 IST 2003 801010

No trouble was found.

The resources tested were:

[TOP]
- sysplanar0 00-00 System Planar
- oppanel 00-00 Operator panel
- mem0 00-00 Memory
- proc0 00-00 Processor
- L2cache0 00-00 L2 Cache
- proc1 00-01 Processor
- proc2 00-02 Processor
- proc3 00-03 Processor
- fd0 01-D1-00-00 Diskette Drive
- tty0 01-S1-00-00 Serial Port
- tty3 01-S2-00-00 Serial Port
- tty1 01-S3-00-00 Serial Port
[MORE...22]


Regards

arvibm,

 

[MoshiachNow]

HI,

The picture looks pretty clear - during Oracle operations one of the FS got corrupted :

Date/Time: Sat Sep 6 21:03:51 IST

Description
FILE SYSTEM CORRUPTION

Probable Causes
INVALID FILE SYSTEM CONTROL DATA

Recommended Actions
PERFORM FULL FILE SYSTEM RECOVERY USING FSCK UTILITY
OBTAIN DUMP
CHECK ERROR LOG FOR ADDITIONAL RELATED ENTRIES

Failure Causes
ADAPTER HARDWARE OR MICROCODE
DISK DRIVE HARDWARE OR MICROCODE
SOFTWARE PROGRAM
STORAGE CABLE LOOSE, DEFECTIVE, OR UNTERMINATED
======================
Since no scsi/hdisk error show close to the time of the dump - possibly it's only a logical FS corruption - you will fix it by running &quot;fsck -y <FSname>&quot;

The filesystem to be fixed can be derived from:

MAJOR/MINOR DEVICE NUMBER
0000 000F

000F means &quot;15&quot; in decimal - .
Run &quot;echo vfs | kdb&quot;

You will get something like :
13 316F2AAC 006161F0 13E3B080 1357BA00 13F74A00 300823C0 JFS DEVMOUNT
... /dev/SharedItemslv mounted over /scitex/version/SharedItems
14 316F2AE0 006161F0 14812600 13F52F80 15205C00 30082428 JFS DEVMOUNT
... /dev/TantalusMsgDBlv mounted over /scitex/version/TantalusMsgDB
15 316F2B14 006161F0 151E9D00 1492A500 151E9D00 30082490 JFS DEVMOUNT
... /dev/ExposeSpoollv mounted over /scitex/version/ExposeSpool
16 316F2B48 006161F0 13AF3680 1311C100 1329AF00 30082560 JFS DEVMOUNT
... /dev/slv3.0 mounted over /dataVolumes/condor2.3.0
17 316F2B7C 006161F0 13AF3800 13C0B580 14C96500 30082630 JFS DEVMOUNT
... /dev/slv1.0 mounted over /dataVolumes/condor2.1.0
=====================
The filesystem numbered &quot;15&quot; (/dev/ExposeSpoollv in this case) will be the corrupted one.




&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[arvibm]

Hi levw ,

The command echo vfs | kdb has pointed towards logical volume maxtest which is of no use. i have remove the maxtest volume.hope i will not face this problem in future.
do u have any documents or redbooks for dump analysis.pls let me know where i can find and learn to examine the system dump.ur post is helpful get a star for it.

Thanx and Regards

arvibm

 

[MoshiachNow]

Thanks.

Please refer to FAQ:

faq52-1546

http://www.tek-tips.com/faqs.cfm?spid=52&sfid=1546

and:

http://www-1.ibm.com/support/docview.wss?uid=isg1hintsTips0442

&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[unixtechie]

That is incorrect. If it is Meta Data corruption an fsck will not fix the problem because it is at a lower level. The LV would need to be recreated or do a cplv to recreate it.

 

[MoshiachNow]

unixtechie - that's what I wrote above,assuming it's NOT a simple FS corruption:

&quot;If it IS a meta data corruption (if it's a Fs corruption - try fixing it with fsck)- the way I used to look for it was either copying the whole FS data to another FS and wtaching on which file it will crash,or - running a full backup (Legato,etc) and checking on which folder/file it failed.
Then you can delete the problematic file/folder.&quot;


&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[MoshiachNow]

P.S. This if from IBM book

http://www-1.ibm.com/support/docvie...intsTips0313&loc=en_US&cs=utf-8&cc=US&lang=en

========================================
Error Description: JFS_META_CORRUPTION Error ID: 684A365B
This error is logged when filesystem corruption is detected. Examine the system error report for any errors on the storage subsystem. Ensure that the latest drivers and microcode are installed for the storage subsystem components. Unmount the affected filesystem and run fsck against it. If a system crash occurred when this error was logged, the dump should be examined to help determine the cause. Contact your appropriate service representative if you need further assistance.






&quot;Long live king Moshiach !&quot;

http://www.7for70.com/

 

[unixtechie]

Gee, I WAS going to let it go, however, when I have had metadata corruption on systems, AIX SUPPORT had me recreate LV's - NOT run an fsck.

And the key here is &quot;If a system crash occurred when this error was logged, the dump should be examined to help determine the cause&quot; which is what happened.

Besides, this WAS your response for fixing it: &quot;Since no scsi/hdisk error show close to the time of the dump - possibly it's only a logical FS corruption - you will fix it by running &quot;fsck -y <FSname>&quot;&quot; Which is incorrect.

Kinda touchy about being wrong, huh?

 

[bi]

arvibm,

I would be concerned about the /prodarch filesystem filling up so frequently (twice in one day) -- that could cause filesystem corruption.

By the filesystem's name, I am assuming it is where the archived logs from a database go. I would recommend that you either make that filesystem larger or have your DBAs more frequently run whatever routine they need to to empty out the archives more frequently.

 

[arvibm]

Hi bi,

U r correct the filesystem is for stroing archive logs.I have discussed the problem with DBA and we decide to increase the size of filesystem to avoid corruption problems.I have removed that unwanted corrupted filesystem (maxtest) now.and i hope i will not face the dump problem in future.Thanx to u all for such a nice fruitful discussion which we have on system dump.with all of ur efforts the thread is quite helpful for resolving the problems of systemdump.

Thanx to all of u again...

Regards

arvibm