Server panics and reboots after scheduled reboot

[johngiggs]

On a HP-UX 10.20 server, for the past few weeks, after the regular weekly reboot, the server will panic and reboot itself again.

13:44 Sun Mar 21, 2004. Reboot:
01:38 Tue Mar 23 2004. Reboot after panic: , isr.ior = 0'0.0'0
10:55 Sun Mar 28, 2004. Reboot:
11:14 Sun Mar 28 2004. Reboot after panic: , isr.ior = 0'0.0'0
21:40 Sun Apr 4, 2004. Reboot:
21:59 Sun Apr 04 2004. Reboot after panic: , isr.ior = 0'0.0'0
11:15 Sun Apr 11, 2004. Reboot:
11:34 Sun Apr 11 2004. Reboot after panic: , isr.ior = 0'0.0'0
10:08 Sun Apr 18, 2004. Reboot:
10:26 Sun Apr 18 2004. Reboot after panic: , isr.ior = 0'0.0'0
19:47 Sun Apr 25, 2004. Reboot:
20:06 Sun Apr 25 2004. Reboot after panic: , isr.ior = 0'0.0'0
10:06 Sun May 2, 2004. Reboot:
10:24 Sun May 02 2004. Reboot after panic: , isr.ior = 0'0.0'0

The panic on 3/23 was legitimate, but later that day, one of the admins created a new RC script in the rc3.d directory. Ever since then this has been occurring. The new script is just supposed to start the syslog if it is not already running.

What is the HP-UX equivalent to prtdiag?

Does anyone have any ideas as to what could be causing the panic?

Thanks,

John

 

[comtec17]

Have you tried removing the rc script?

 

[pete91z]

Are there any files in /var/tombstones? These are sometimes generated when the system panics and may give a few clues to the cause.

 

[johngiggs]

comtec17,

I would have thought that trying to remove the rc script would be a good troubleshooting technique as well, but the admins are convinced that it is likely a hardware issue. The script isn't even working anyway, so I'm not sure why they have it there anyway. The syslog is not being started when the server comes up, so the script is useless.

pete91z,

/var/tombstones does not exist.

Thank you both for your help.

Has anyone else experienced problems with the syslog not being started upon the server rebooting?

Any other sugguestions / tips would be greatly appreciated.

Thanks,

John

 

[new2me]

by default there should be a directory /var/tomestones, check your sysinfo for error/warning msg. Once you have your tomestones directory back, the next panic reboot check /var/tomestone/ts** files. I believe in the file if the chassis codes are all zeros that indicated a hardware problem - verify with HP.

The startup of syslogd script should be in /sbin/init.d/syslogd.

 

[johngiggs]

new2me,

I don't have a /var/tombstones directory on any of my HP-UX machines. I am running 10.20 on all of them. The rc script was an attempt to start the syslog if it did not come up properly (which didn't happen very frequently until recently). They will probably just have HP analyze the crash file to determine the cause of the problem.

Thanks,

John

 

[new2me]

My opinion/suggestion create /var/tomestones - owner is root:root & chmod 755 - so HP will have something to look at to determind what the problem is.

Good-luck!

 

[unixqueen]

Did the /etc/rc.log record any useful info following the panic, I wouldn't have thought so, but was a crash directory created in /var/adm/crash, if so does the INDEX file tell you any more.
Can the dmesg or OLDsyslog.log tell you anything.

Why don't you post the rc3.d script for us to test.

 

[johngiggs]

It could be purely coincidental, but here is the rc script anyway...

if [ -s /var/adm/syslog/syslog.log ]
then
echo "OK" > /dev/null
else
cp /etc/syslog.conf.orig /etc/syslog.conf
/sbin/init.d/syslogd stop
sleep 5
/sbin/init.d/syslogd start
fi
if [ -s /etc/syslog.conf ]
then
echo "OK" > /dev/null
else
cp /etc/syslog.conf.orig /etc/syslog.conf
/sbin/init.d/syslogd stop
sleep 5
/sbin/init.d/syslogd start
fi
logger "TESTING SYSLOG PROCESS ON SERVER"
grep "TESTING SYSLOG PROCESS ON SERVER" /var/adm/syslog/syslog.log > /dev/null
if test $? -eq 0
then
exit
else
cp /etc/syslog.conf.orig /etc/syslog.conf
/sbin/init.d/syslogd stop
sleep 5
/sbin/init.d/syslogd start
sleep 300
logger "TESTING SYSLOG PROCESS ON SERVER"
grep "TESTING SYSLOG PROCESS ON SERVER" /var/adm/syslog/syslog.log > /dev/null
if test $? -eq 0
then
exit
else
cp /etc/syslog.conf.orig /etc/syslog.conf
/sbin/init.d/syslogd stop
sleep 5
/sbin/init.d/syslogd start
fi
fi

The crash logs were there yesterday, but apparently someone removed them. They may have been sent to HP for analysis. After reviewing the rc.log, it seems like the crash may have something to do with either Oracle or Unicenter TNG.

John

 

[cmoore4]

A bad rc script shouldn't cause a panic. It might make the system unbootable, but it shouldn't panic the kernel.

The /var/tombstones directory holds hardware tombstones. These get generated as a result of hardware failure (HPMCs, or High Priority Machine Checks).

In the case of a panic the dump should get saved in /var/adm/crash unless you've configured it otherwise.
HP's dump analysis tool, q4, will be able to read the dump but unless you know what you're looking for it won't help a lot.

Chris Moore