/sbin/init failed! - kernel panic

[roycrom]

OK, let me tell you what happened:
RHEL4 Dell poweredge 2850 with hardware mirrored pair of internal disks.
Partitions of /, /boot, /home and swap
One of the disks in the mirror went so we got a replacement.
Grub.conf says root=LABEL=/
and similar for /etc/fstab.
When the mirror has rebuilt, it sems all the fs labels have gone, so I thought OK, I'll boot with a live CD and alter grub.conf and fstab.
I boot up the liveCD and see I now have /dev/sda1, sda2, sda3 and sda4. Ah, that look good.
I modify grub.conf to reference /dev/sda2 as the root instead of LABEL=/ and do the same for ther relevant lines in fstab.

Sorted - it should boot now.

However, wen I try to boot up, it mounts the root filesystem then tries to run /sbin/init.

This fails and I get a kernel panic.

It actually finds /sbin/init, but cannot execute it.

I have tried booting into a live CD and going into a chroot by issuing,

chroot /mnt/sysimage /bin/bash

but I get cannot execute /bin/bash (or/bin/sh) if I leave the /bin/bash bit off.
So I have valid filesystems, all have had an fsck annd all are clean.
Why would things be non-executable?

Any ideas would be greatly appreciated.

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[Annihilannic]

When you are booted from the live CD, what's the output of the mount command? Any "noexec"s in there or anything?

Does strace -f chroot /mnt/sysimage /bin/bash give you any more clues about why it won't execute?

Annihilannic.

 

[roycrom]

I think I have traced this to a corruption with /lib somewhere, I'm still looking at it, I copied a lib folder from another machine and can chroot fine. I'm currently comparing permssions/sizes etc. between the one I copied on and the original - in the mean time - if you glean anything from the below output then please let me know!

Relevant output from mount command

/dev/sda2 on /mnt/sysimage type ext3 (rw)
/dev/sda4 on /mnt/sysimage/home type ext3 (rw)
/dev/sda1 on /mnt/sysimage/boot type ext3 (rw)

output from strace

execve("/usr/bin/chroot", ["chroot", "/mnt/sysimage", "/bin/bash"], [/* 22 vars */]) = 0
uname({sys="Linux", node="localhost.localdomain", ...}) = 0
brk(0)                                  = 0x8264000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/sse2/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xbff90690) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", 0xbff90690)     = -1 ENOENT (No such file or directory)
open("/lib/tls/sse2/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xbff90690)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0xbff90690)          = -1 ENOENT (No such file or directory)
open("/lib/i686/sse2/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xbff90690)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", 0xbff90690)         = -1 ENOENT (No such file or directory)
open("/lib/sse2/libselinux.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xbff90690)         = -1 ENOENT (No such file or directory)
open("/lib/libselinux.so.1", O_RDONLY)  = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260 \0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0555, st_size=54992, ...}) = 0
old_mmap(NULL, 56144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40015000
old_mmap(0x40022000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x40022000
close(3)                                = 0
open("/lib/libm.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\3202\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0555, st_size=136584, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40023000
old_mmap(NULL, 139424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40024000
old_mmap(0x40045000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x40045000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0(S\1\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0555, st_size=1116372, ...}) = 0
old_mmap(NULL, 1125684, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40047000
old_mmap(0x40154000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10d000) = 0x40154000
old_mmap(0x40158000, 7476, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40158000
close(3)                                = 0
mprotect(0x40154000, 4096, PROT_READ)   = 0
access("/etc/selinux/", F_OK)           = 0
brk(0)                                  = 0x8264000
brk(0x8285000)                          = 0x8285000
open("/etc/selinux/config", O_RDONLY)   = -1 ENOENT (No such file or directory)
open("/proc/mounts", O_RDONLY)          = 3
fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4015a000
read(3, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 552
close(3)                                = 0
munmap(0x4015a000, 4096)                = 0
chroot("/mnt/sysimage")                 = 0
chdir("/")                              = 0
execve("/bin/bash", ["/bin/bash"], [/* 22 vars */]) = -1 EACCES (Permission denied)
write(2, "chroot: ", 8chroot: )                 = 8
write(2, "cannot execute /bin/bash", 24cannot execute /bin/bash) = 24
write(2, ": Permission denied\n", 20: Permission denied
)   = 20
exit_group(1)                           = ?
Process 763 detached

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[Annihilannic]

execve("/bin/bash", ["/bin/bash"], [/* 22 vars */]) = -1 EACCES (Permission denied)

Well, it certainly sounds permissions related. What are the permissions on the /bin/bash binary itself? Does chroot /a /usr/bin/ldd -r -v /bin/bash show any failed library dependencies?

Annihilannic.

 

[roycrom]

Hi Annihilannic, thanks again for you response,

I've tracked this down to selinux, I don't know why or what, I haven't got that far in to it yet, but I changed /etc/selinux/config to make selinux permissive (just warn me instead of denying me), rebooted and things work again.

Very strange, maybe its to do with my devices using disk devices instead of disk label so selinux wasn't happy with that. I need to read up a bit more on swelinux before I really understand it, it could be some other reason altogether.

Thanks for the help

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[macd68]

The bottom line here is that selinux is not a good mandatory addition.
I really wish main line distributions would take it out of production.

Sysadmins can decide where and when to include it.

 

[roycrom]

I read the other day (i thought it was slashdot but can't find it there now, must be some other linux site I read) that an alternative to selinux is being developed that does the extra security in a much more concise and configurable way - more human shall we say?

selinux seems to have sneaked into many distros via the back door, and whilst the majority of people never even realise it's there, when something goes wrong, selinux sticks its big fat head in the way and decides to do whatever it can to flummox you at every turn!

My experience over the last few days anyway

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[Annihilannic]

I haven't encounterd selinux yet; which distributions are including it?

Annihilannic.

 

[roycrom]

Red Hat EL 4 for definite. Thats what I've had this problem on.

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[Annihilannic]

We have recently deployed some RHEL4 servers and SELinux appears to be installed, but disabled by default (in /etc/selinux/config). Did you specifically enable it? Or could it be down to some obscure choice made during the installation?

Annihilannic.

 

[roycrom]

Yes, there is a choice during the install,

Installed and active
Installed but disabled
Not installed

I thought it was installed and active by default - but I'm not 100% sure.

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom