core dump help

[ponetguy2]

I have apache tomcat running on Solaris 9. I've been getting core dumps through
out the day. How do I analyze these core dumps from tomcat
(/var/core/core.httpd.8719.hostname.60001.60001.1139818065)?

We do not have a Sun contract to assist is anaylyzing this core dump.
Any help will be appreciated.

 

[iambosley]

You can download a free Solaris dump anaylsis tool called
SCAT.... Try url :

http://www.sun.com/download/products.xml?id=3fce7df0

Roger Bosley

 

[ponetguy2]

GREAT!!! thanx

After digging through the logs, I found that the core dumps were caused by improper ABORT signals. Plus coreadm configuration on this server is a little out of whack. I will disable coreadm process with command: coreadm -d process

All the core dumps is eating up alot of disk space and I'm afraid it might bring down my server.

 

[kHz]

Run mdb or gdb against the core file and then analyze the output. Solaris Crash Analysis Tool (SCAT).

 

[ponetguy2]

I ran mdb on one of the core files and I don't know what to make out of it. Please help me by explaining what
this output means.


# mdb core.httpd.4460.hostname.60001.60001.1138830459
mdb: warning: failed to infer pathname to executable; symbol table will not be available
Loading modules: [ libc.so.1 ld.so.1 ]
> ::status
debugging core file of httpd (32-bit) from hostname
initial argv: bin/httpd -d /export/home/stronghold3.0 -f conf/httpd.conf
threading model: single-threaded
status: process terminated by SIGABRT (Abort)
> ::regs
%g0 = 0x0000000000000000 %l0 = 0xff1bc000
%g1 = 0x0000000000000025 %l1 = 0x00000000
%g2 = 0x000000000000008c %l2 = 0x00000000
%g3 = 0x0000000000000067 %l3 = 0x00000000
%g4 = 0x0000000000464846 %l4 = 0x00000000
%g5 = 0x0000000000000000 %l5 = 0x00000000
%g6 = 0x0000000000000000 %l6 = 0x00000000
%g7 = 0x0000000000000000 %l7 = 0x00000000
%o0 = 0x0000000000000000 %i0 = 0x00000000
%o1 = 0x0000000000000006 %i1 = 0x00003520
%o2 = 0x00000000ffbfe7c8 %i2 = 0x00004015
%o3 = 0xa572d6fd61681a06 %i3 = 0x0021f6ec
%o4 = 0x0000000000464844 %i4 = 0x00000000
%o5 = 0xfc9d3168b316848d %i5 = 0x00003520
%o6 = 0x00000000ffbfe768 %i6 = 0xffbfe7f8
%o7 = 0x00000000ff136c98 libc.so.1`abort+0xc0 %i7 = 0x0021d414

%psr = 0xfe401000 impl=0xf ver=0xe icc=nZvc
ec=0 ef=4096 pil=0 s=0 ps=0 et=0 cwp=0x0
%y = 0x00000000
%pc = 0xff1a01a0 libc.so.1`_kill+8
%npc = 0xff1a01a4 libc.so.1`_kill+0xc
%sp = 0xffbfe768
%fp = 0xffbfe7f8

%wim = 0x00000000
%tbr = 0x00000000

 

[kHz]

You need the stack trace from mdb. Try using ::stack

 

[kHz]

There is a tool (MDeBug) on this link that will analyze kernel cores and application cores:

http://access1.sun.com/tools/MDeBug/README.html

 

[kHz]

Post the output after running the MDeBug script (option 2 - application core). You should be able to identify something from the stack.

 

[kHz]

And a side note. Sun wouldn't be able to help you analyze a third party application core, that isn't their software, like tomcat.

 

[ponetguy2]

I probably will not be able to install MDeBug on our web server until the weekend. This
is a live production box, and we are a little paranoid with installing additional
packages on this server. However, I will try to convince my boss.

Here is the stack output from mdb:

# mdb core.httpd.24158.hostname.60001.60001.1139157484
mdb: warning: failed to infer pathname to executable; symbol table will not be available
Loading modules: [ libc.so.1 ld.so.1 ]
> ::stack
libc.so.1`_kill+8(0, 1590, 4015, 21f6ec, 0, 1590)
0x21d414(452d80, 496008, 4015, 496008, 225e64, 0)
0x1fd944(461040, 17, fec4c000, 4000, 3a6010, 3)
0x1fd894(461040, 17, fec4c000, 4000, 2, 1c7dff)
0x1fd610(4000, 17, fec48000, 4000, 16, ff00)
0x1fbe08(461040, fec48000, 8000, 1fbd1c, 73736c00, 73736c00)
0x202bc4(461040, fec48000, 8000, 0, 0, 0)
0xae3f0(344050, fec48000, 8000, ae3a8, 69746500, 69746500)
0x13bbc4(ffbff310, 30a040, 30d088, 0, 6170706c, 64656661)
0x13a638(298ca8, ffbff32c, 344050, fec48000, 8000, 0)
0x105128(344050, fec48000, 8000, 6feb6f, a, 1c7dfd)
0x1078d4(344050, fec48000, 8000, ffbff498, 2, 1c7dff)
0x1067b8(344050, fec48000, 8000, 3814f5, 16, ff00)
0x10690c(344050, fec48000, 8000, 11df48, 29eb68, 0)
0x106ff0(344050, fec48000, 8000, 0, 0, 0)
0x120940(fec40000, 398840, 0, 3b418, 112800, 0)
0x113254(398840, 112bc8, ffffffff, 0, 6170706c, 64656661)
0x108944(398840, 0, 2a0000, 16, 4, fed40178)
0x1250d8(398840, 1, 4, fed40158, 398840, 1)
0x125158(398840, c8, 398840, ffbff8c0, ffbff8d0, 2)
0x118cc4(2, 116e20, 116c00, 118f84, 0, 0)
0x119044(2f9040, 2, 43e62782, 6e, ff1bf2a8, 0)
0x11958c(0, ffbffb04, 0, 2f9040, 29dd70, 27c980)
0x119d0c(5, ffbffc34, 1, 0, ffbffd16, 0)
0x11a67c(5, ffbffc34, ffbffc4c, 2f2264, 0, 0)
0x5db18(0, 0, 0, 0, 0, 0)

 

[ponetguy2]

sorry khz. mdb is installed and i misunderstood your post. i did'nt know that mdebug is a script and i do have the required packages installed


# pkginfo | grep -i mdb
system FJSVmdb Fujitsu Platform Modular Debugger
system FJSVmdbx Fujitsu Platform Modular Debugger (64-bit)
system SUNWmdb Modular Debugger
system SUNWmdbdm Modular Debugger Demo Source
system SUNWmdbx Modular Debugger (64-bit)

 

[kHz]

You will get a lot more information after running the MDeBug script, but at the top of the stack is libc.so.1_kill which makes me wonder if that is the library causing your problems. Not entirely certain, but it is suspect.

Do you run Veritas Netbackup or Tivoli Storage Manager where you can look at archived backups for libc.so.1 and see if it has changed from the last time Tomcat wasn't producing cores?

 

[kHz]

Or something is calling libc.so.1 and causing an exception at the address shown.

 

[ponetguy2]

Welcome to the MDeBug Session
******************************

Select one of the following:
1. Run MDeBug against a Kernel Crash dump
2. Run MDeBug against an Application core
3. Exit
Enter your selection:2
Enter the binary name which generated the core:

I ran mdebug script on the directory where the core dump resides. However, I'm a little confused.
What is the binary name? httpd generated the core dump. Does this mean that httpd is the binary name?

 

[kHz]

You will need to provide the full path name of the binary that caused the core because it uses that binary to gather information from the core.

 

[ponetguy2]

Do you run Veritas Netbackup or Tivoli Storage Manager where you can look at archived backups for libc.so.1 and see if it has changed from the last time Tomcat wasn't producing cores?

No. I only have apache tomcat and SDS running on this box.