Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations SkipVought on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Help with core dump

Status
Not open for further replies.
SAIMike

SAIMike

IS-IT--Management
Jun 18, 2005
8
0
0
US
Everyone, i am running aix 4.3.3 and received the following core dump sporadically but always at the same time. Can anyone figure out what is causing this.
---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:18:43

Sequence Number: 10510

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

6616

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

sh_mbslen 40

sh_mbslen 48

io_sync 54

xec_switc 540

sh_exec 2E4

comsubst 664

getch D80

copyto 270

mac_expan A4

mac_trim 18

env_setli 84

xec_switc 2E4

sh_exec 2E4

xec_switc E98

sh_exec 2E4

xec_switc F80

sh_exec 2E4

exfile 60C

main A34

__start 8C



Symptom Data

REPORTABLE

1

INTERNAL ERROR

1

SYMPTOM CODE

PIDS/5765c3403 LVLS/430 PCSS/SPI2 FLDS/ksh SIG/11 FLDS/sh_mbslen VALU/40 FLDS/io_sync

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:18:39

Sequence Number: 10509

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

14162

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

p_flush B8

p_flush 9C

p_setout F0

sh_failst 28

sh_failms 34

sh_wcstom 58

p_flush DC

emacs_rea 84

io_readbu 140

env_readl 1D8

b_read 140

xec_built 19C

xec_switc 3EC

sh_exec 2E4

xec_switc E98

sh_exec 2E4

xec_switc F80

sh_exec 2E4

exfile 60C

main A34

__start 8C



Symptom Data

REPORTABLE

1

INTERNAL ERROR

1

SYMPTOM CODE

PIDS/5765c3403 LVLS/430 PCSS/SPI2 FLDS/ksh SIG/11 FLDS/p_flush VALU/b8 FLDS/p_setout

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:18:13

Sequence Number: 10508

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35112

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:17:43

Sequence Number: 10507

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35106

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:17:12

Sequence Number: 10506

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35100

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:16:41

Sequence Number: 10505

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35094

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:16:10

Sequence Number: 10504

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35088

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:15:40

Sequence Number: 10503

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35082

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:15:09

Sequence Number: 10502

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35076

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:14:37

Sequence Number: 10501

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35326

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:14:07

Sequence Number: 10500

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35320

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

---------------------------------------------------------------------------

LABEL: CORE_DUMP

IDENTIFIER: C60BB505



Date/Time: Thu Jan 5 04:13:35

Sequence Number: 10499

Machine Id: 000275784C00

Node Id: sysb

Class: S

Type: PERM

Resource Name: SYSPROC



Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED



Probable Causes

SOFTWARE PROGRAM



User Causes

USER GENERATED SIGNAL



Recommended Actions

CORRECT THEN RETRY



Failure Causes

SOFTWARE PROGRAM



Recommended Actions

RERUN THE APPLICATION PROGRAM

IF PROBLEM PERSISTS THEN DO THE FOLLOWING

CONTACT APPROPRIATE SERVICE REPRESENTATIVE



Detail Data

SIGNAL NUMBER

11

USER'S PROCESS ID:

35314

FILE SYSTEM SERIAL NUMBER

9

INODE NUMBER

141345

PROGRAM NAME

ksh

ADDITIONAL INFORMATION

Unable to generate symptom string.

Too many stack elements.

any help would be great!

 
Sort by date Sort by votes
Probably a cron job calling someting.
Try lquerypv -h core 6b0 64



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!
 
Upvote 0 Downvote
do i run this from the home directory?
 
Upvote 0 Downvote
run the command where the core file is located or use the entire path to the core file. if it is always at the same time, then look at your crontab as mentioned, because it might be a ksh script (program name) in cron. find what the filesystem (9) is that is indicated in the errpt.
 
Upvote 0 Downvote
since i am new to core dumps, would it matter if i did an errclear the day it happened and now now be able to find it via lquerypv -h core 6b0 64
Mike
 
Upvote 0 Downvote
Do you still see the core file on the machine? If so, go for it.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!
 
Upvote 0 Downvote
errclear clears the errpt but doesn't remove any core files.
 
Upvote 0 Downvote
If you don't know where the core file is, you have to identify the Serial Number for the filesystems and then look for the filesystem with serial number 9 in this case. you need a C program to fullstat every filesystem's mountpoint to get the FS serno. It is not visible from the shell AFAK.

However, this command can also identify the core file:

find / -name core -exec ls -i {} \;

Then look for the core file(s) with the same inode number(s) as mentioned in the error report.

As your core file is being generated by a ksh process and there is a mention of "too many stack elements", I would look for some infinite looping or recursion in a script which is fired off from cron (around 04:00 - 04:10).

Also in some entries of your error report, I see IO system calls in the stack trace, so an IO-error on disk may also be causing problems - but that would also be obvious in a full error report. Either way, I'd run a diag on the box, just to make sure there's nothing wrong with the hardware.


HTH,

p5wizard
 
Upvote 0 Downvote
thanks for that info on how to search for the core. i did find a problem with one of the cron jobs yesterday and today i had no errors.
I will check the I/O error.
Mike
 
Upvote 0 Downvote
You could write a program for all mounted filesystems and run the stat subroutine on each of them, then compare st_vfs field of the filesystem to the VFS serial number in the errpt.

Or you can run the 'crash' command with the vfs subcommand to list the mounted filesystems. Then compare the values in the number column of the VFS table with the VFS serial number in the errpt.

The filesystem serial number depends on the mount sequence and, therefore, may change at restart.
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

mohamedazrin
  • Locked
  • Question
Need some help with extend the PP
Replies
1
Views
128
Andjey
Andjey
Declan-Hughe-CS
  • Locked
  • Question
Curl - AIX 7.2 - Illegal instruction - core dumped
Replies
0
Views
121
Declan-Hughe-CS
Declan-Hughe-CS
Loic SARRAZIN
  • Locked
  • Question
Authentication order with NIS / ypbind
Replies
0
Views
53
Loic SARRAZIN
Loic SARRAZIN
Luqy
  • Locked
  • Question
SEA re-create with single VSwitch
Replies
0
Views
49
Luqy
Luqy
Wilder Mellotto
  • Locked
  • Question
SYSTEM DUMP 1
Replies
1
Views
47
DukeSSD
DukeSSD

Part and Inventory Search

Sponsor

Back
Top