Crash dump - Fujitsu

[dandan123]

A server that I'm trying to patch always panics after applying the patch cluster.

Once it panics and I bring it to the ok prompt trying to get a crash dump by typing sync produces this output -

No callback routine has been installed

It's Primepower 650 Fujitsu server.

Any ideas on what's hapenning ?

 

[Annihilannic]

You can't force a server to crash using 'sync' when it is no longer running an OS.

If it isn't dumping when it panics (possibly happening too early in the boot process, before it knows where your dump device is), then you're stuck... you'll need to boot off working media (CD-ROM/DVD?) and remove the patches again I think, otherwise Sun or Fujitsu may be able to recognise your problem based on the panic strings.

Annihilannic.

 

[dandan123]

I always detach the mirror before patching so recovery is not a problem.

I need a crash dump though as Sun is not able to diagnose without a crash dump.

 

[Annihilannic]

Well, if it doesn't dump when it crashes... you're stuck. You'll need to explain that to Sun, I think.

Unless dumps have been disabled or something... at what stage in the boot process does it happen? Do you have a console log we can see?

Annihilannic.

 

[dandan123]

I'll post the console log tomorrow.

Thank you for taking the trouble to post.

 

[dandan123]

unOS Release 5.10 Version Generic_137111-06 64-bit
Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
NOTICE: Emulex FC SCSI/IP lpfc 6.11cx2
NOTICE: lpfc0: Firmware Rev 1.91A5 (T2D1.91A5)
NOTICE: lpfc0: WWPN:10:00:00:00:c9:5e:5c:f7 WWNN:20:00:00:00:c9:5e:5c:f7 INTX:1
NOTICE: lpfc1: Firmware Rev 1.91A5 (T2D1.91A5)
NOTICE: lpfc1: WWPN:10:00:00:00:c9:5e:5b:47 WWNN:20:00:00:00:c9:5e:5b:47 INTX:1
WARNING: /pci@84,2000/fibre-channel@1/sd@e,0 (sd1):
Corrupt label; wrong magic number

WARNING: /pci@85,2000/fibre-channel@1/sd@f,0 (sd785):
Corrupt label; wrong magic number


SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error, VER: 1, SEVERITY: Major
EVENT-TIME: 0x4a747a2c.0x71ebe44 (0x26d27242c0)
PLATFORM: FJSV,GPUZC-M, CSN: -, HOSTNAME:
SOURCE: SunOS, REV: 5.10 Generic_137111-06
DESC: Errors have been detected that require a reboot to ensure system
integrity. See

http://www.sun.com/msg/SUNOS-8000-0G

for more information.
AUTO-RESPONSE: Solaris will attempt to save and diagnose the error telemetry
IMPACT: The system will sync files, save a crash dump if needed, and reboot
REC-ACTION: Save the error summary below in case telemetry cannot be saved


panic[cpu0]/thread=2a100889ca0: BERR Errors(s)

000002a1008888b0 FJSV,SPARC64-V:fjsv_cpu_sync_error+5ac (2a1008890b0, 10ff170604c, 4280804b, 80000000, 0, e0000000)
%l0-3: 0000000000000000 0000000000000001 000000000000006e 000000008c000000
%l4-7: 0000000000000000 0000000000000001 000000004280804b 0000000000000000
000002a100889000 unix:ktl0+48 (3000b2b8d80, 29ebfa0e04c, 100889384, 0, 100000000, 30000fcd728)
%l0-3: 0000000000000003 0000000000001400 0000004404001602 0000000001220344
%l4-7: 000003000b2b8d80 0000000000000000 0000000000000000 000002a1008890b0
000002a100889150 FJSVscf:_init+1791c (70409524, 30000fcd700, 30000fcd728, 0, c, 2a1008892bc)
%l0-3: 0000000000000002 0000000000000001 000003000b2b8d80 0000029ebfa0e000
%l4-7: 0000000000000003 0000000000000000 0000000000000200 0000000000000000
000002a100889200 FJSVscf:_init+157b0 (1, 70409524, 30000fcd700, 30000d5a230, 2a100889378, 2a100889384)
%l0-3: 000003000b0c08e0 000003000b2f0000 000002a1008892b0 0000000000000016
%l4-7: 00000000012b2000 0000000001265768 0000000001265400 0000000000000000
000002a1008892c0 FJSVscf:_init+20fc (30000fcd700, 0, 7044e300, 0, 70409ab8, 7044e388)
%l0-3: 0000030000d5a230 0000030000d5a230 0000000000000001 0000000000000000
%l4-7: 0000000070454ca8 0000000070409514 000000000000000e 0000000000000000
000002a1008894d0 genunix:devi_attach+ac (30000d5a230, 0, 30000013498, 0, 0, 7bead578)
%l0-3: 0000030000c53478 000000000000011f 0000000001888c28 0000000000007498
%l4-7: 00000000018ed000 0000030000013498 0000000000000003 0000000000000002
000002a1008895a0 genunix:attach_node+9c (30000d5a230, 1, 0, 0, 30000d5a298, 4)
%l0-3: 00000000fffeffff 00000000fffefc00 0000000040010000 0000000000000000
%l4-7: 0000000040010000 0000000000000003 0000000000000000 0000000000000000
000002a100889650 genunix:i_ndi_config_node+104 (30000d5a230, 114, 10d1954, 10, 1888c00, 10d1800)
%l0-3: 0000030000d5a650 0000000000000000 0000000000000000 0000000000000006
%l4-7: 0000000001888ee0 0000000000000000 0000030000d59950 0000000000000004
000002a100889700 genunix:i_ddi_attachchild+38 (30000d5a230, 2a100889ca0, 1888c00, 189b520, 0, 0)
%l0-3: 0000030000d5a6b8 0000000000000001 ffffffffffffffff 0000000000000003
%l4-7: 0000000001888ee0 0000000000000000 0000000000000000 0000000000000001
000002a1008897b0 genunix:devi_attach_node+84 (30000d5a230, 4040, 0, 30000d5a298, 4040, 0)
%l0-3: 0000030000d5a6b8 0000000000000000 0000030000d52d00 0000000000000002
%l4-7: 0000000000000001 000000000185d000 0000000000000002 0000000000000003
000002a100889860 genunix:config_immediate_children+d0 (30000d5a650, 30000d5a230, 11f, 0, 4, 70906f08)
%l0-3: ffffffffffffffff 0000000000004040 0000000000010000 0000030000d5a6b8
%l4-7: 000000000180e000 00000000feedface 0000030000d5a020 ffffffffffffffff
000002a100889920 genunix:devi_config_common+c0 (30000d5a650, 4, 11f, 11f, 0, 4040)
%l0-3: 00000000018e6400 000002a1007b1ca0 000000000000003d 0000000000000010
%l4-7: 0000000070906f08 0000000070906ff0 0000000000000009 0000000000000000
000002a1008899d0 genunix:mt_config_thread+60 (3000b0c0838, 0, 18462a0, 18462a0, 3000b2ccb80, 30000d5a650)
%l0-3: 0000000000000000 0000000000000000 00000000018ea400 000002a100899ca0
%l4-7: 000000000180e000 0000000000000000 000002a10001fca0 0000000000001d1d

syncing file systems... done
ereport.io.pci.rta ena=26d1e0180000001 detector=[ version=0 scheme="dev"
device-path="/pci@87,4000" ] pci-status=12a0 pci-command=146 pci-pa=0
pbm-afsr=0 pbm-afar=0 component="UNKNOWN"

ereport.io.pci.target-rta ena=26d1e0180000001 detector=[ version=0 scheme="dev"
device-path="/pci@87,4000/ebus@1" ] pci-pa=10ff170604c component=
"Invalid address"

ereport.cpu.SPARC64-V.berr ena=26d1e0180000001 detector=[ version=1 scheme=
"cpu" cpuid=0 cpumask=51 serial="0" ] pc=1047c54 tl=0 tt=32 privileged=1
flt-status=200 sfsr=4280804b sfar=10ff170604c raw-sfar=29ebfa0e04c resource=[
version=0 scheme="dev" device-path="UNKNOWN" devid="Invalid address" ]

skipping system dump - no dump device configured

 

[Annihilannic]

I think that's your answer right there at the bottom:

 skipping system dump - no dump device configured

Have you previously configured a dump device (usually using the dumpadm command)? If not, you can probably do so by booting from CD-ROM, mounting this disk and running it (see the -r option to work relative to an alternative root).

It's possible that you have already configured it and, as mentioned earlier, it's just too early in the boot process, or for some reason as a result of the problem you're running into the dump device isn't available anyway.

Annihilannic.

 

[dandan123]

dump device, savecore directory are both configured and have enough space to hold several crash dumps.

 

[Annihilannic]

In that case... Sun/Fujitsu are your only hope!

Out of curiosity... what is your general opinion of Solaris on Fujitsu? Happy with the combination? When my previous employer were considering them they certainly looked like a cost-effective alternative to pure Sun.

Annihilannic.

 

[dandan123]

We have some 650s and 850s and we've havent' had any major issues.

Of course all the current large Sun servers are made by Fujitsu -

Mx000 series, of which we have quite a few and they are pretty stable.