Red Hat Linux ES4 keeps rebooting

[CarrahaG]

After a power cut, our Red Hat Linux ES4 does not complete the boot process and keeps rebooting itself.

We have two hard drives connected to SCSI channel A on the onboard Adaptec AIC-7899 SCSI adapter. There is no RAID. The drives are:

Fujitsu MAP3367NP (Main drive with MBR)
Seagate ST318417W


We got a chance to note some of the errors it gives before it reboots and starts the process over again.

Checking root filesystem
[/sbin/fsck.ext2 (1) -- /] fsck.ext2 - a /
WARNING: couldn't open /etc/fstat: no such file or directory
ext2fs_check_if_mount/:

The superblock could not read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device>

: No such file or directory while determining whether / is mounted.
fsck.ext2: Is a directory while trying to open /
/etc/rc.d/rc.sysinit : line 415: failure: command not found.

*** An error occurred during the filesystem check
*** Dropping you to a shell; the system will reboot
*** when you leave the shell
*** WARNING -- SELinux is active
*** Disabling security enforcement for system recovery
*** Run 'setenforce 1' to reenable

sulogin: cannot open password database!
Unmounting filesystems


We have tried the following:

Boot with the Red Hat Linux ES4 Disk 1. At the boot prompt typed "linux rescue".

After going through the boot process we get the message. "The rescue environment will now attempt to find your Linux installation and mount it under directory /mnt/sysimage."

We then press OK to continue.

We then get a subsequent message "Error mounting filesystem on sdb1: Invalid Argument".

We then press OK and get another message "You don't have any Linux partitions. Press return to get a shell. The system will reboot automatically when you exit from the shell".

We then get the command line prompt: "-/bin/sh-3.00#"

We then type "fdisk -l" and get the following listing:


Disk /dev/sda: 36.7 GB, 36748945408 bytes
255 heads, 63 sectors/track, 4467 cylinders
Units = cylinders of 16065 x 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 4467 35776755 8e Linux LVM


Disk /dev/sdb: 18.4 GB, 18400000000 bytes
255 heads, 63 sectors/track, 2237 cylinders
Units = cylinders of 16065 x 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 2237 57968671 c W95 FAT32 (LBA)



Can anyone tell us how we can repair our filesystem so that Linux can boot successfully?

 

[Annihilannic]

Sounds like you have a seriously corrupted root filesystem; do you have backups?

When you are booted in rescue mode, does fsck /dev/sda1 appear to work?

Annihilannic.

 

[CarrahaG]

Hi

When we run fsck /dev/sda1 we get the following:

fsck: 1.35 (28-Feb-2004)
WARNING: couldn't open /etc/fstab: no such file or directory
e2fsck: 1.35 (28-Feb-2004)
/boot: clean, 50/26104 files, 26754/104388 blocks

Regards,
Georges

 

[Annihilannic]

It sounds like /dev/sda1 is your /boot filesystem, which is apparently healthy. I guess your root filesystem must be in an LVM volume... I'm no Linux LVM wizard so I'll leave it to the other regulars to advise you.

Annihilannic.

 

[Kozusnik]

Can you run fdisk -l (that's a lower case L)?

I'd run fsck on each filesystem you can find with the output from above. The root fs may be /dev/sda5. If it's not mounting, then /etc is not available.

Maybe try...

fsck -V /dev/sda5

Mark

 

[CarrahaG]

Hi Kozusnik

We put the drive in another Linux box and ran the commands you gave us.

We ran the following:

[root@rs-svr1 etc]# fdisk -l




The output was as follows:

Disk /dev/sda: 36.7 GB, 36771581952 bytes
255 heads, 63 sectors/track, 4470 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 4470 35800852+ 8e Linux LVM

Disk /dev/sdb: 36.7 GB, 36748945408 bytes
255 heads, 63 sectors/track, 4467 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 13 104391 83 Linux
/dev/sdb2 14 4467 35776755 8e Linux LVM



Then we ran the command:

[root@rs-svr1 etc]# fsck -V /dev/sdb2


The result was as follows:

fsck 1.35 (28-Feb-2004)
[/sbin/fsck.ext2 (1) -- /dev/sdb2] fsck.ext2 /dev/sdb2
e2fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Filesystem has unexpected block size while trying to open /dev/sdb2

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>


We also ran the command

[root@rs-svr1 etc]# e2fsck -b 8193 /dev/sdb2


We were not successful.


Any other advice would be appreciated.

 

[Annihilannic]

I don't think you want to try and fsck an LVM partition directly or you could damage data.

Do you get any output from vgdisplay?

Can you see any logical volumes when you ls /dev/vg00? If one of them is called root, try running an fsck against that, i.e. fsck /dev/vg00/root.

Annihilannic.

 

[CarrahaG]

Hello

We ran the command:

[root@rs-svr1 /]# vgdisplay

The output was as follows:

--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 34.09 GB
PE Size 32.00 MB
Total PE 1091
Alloc PE / Size 1090 / 34.06 GB
Free PE / Size 1 / 32.00 MB
VG UUID q382vr-U8lM-1ZYe-rM97-gWo0-EgSA-e2cSKG

--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 34.12 GB
PE Size 32.00 MB
Total PE 1092
Alloc PE / Size 1091 / 34.09 GB
Free PE / Size 1 / 32.00 MB
VG UUID cb813R-Sw2F-RXRu-i78f-XSYq-VZ6L-WSLcWe


We then ran the command:

[root@rs-svr1 /]# ls /dev/vg00

The output was as follows:

ls: /dev/vg00: No such file or directory



We then ran the command:

[root@rs-svr1 /]# ls /dev/VolGroup00

The output was as follows:

LogVol00 LogVol01



After that we ran the command:

[root@rs-svr1 /]# fsck /dev/LogVol01


The output was as follows:

fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
fsck.ext2: No such file or directory while trying to open /dev/LogVol01

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>



We then ran:

[root@rs-svr1 /]# e2fsck -b 8193 /dev/LogVol01


The outout was as follows:


e2fsck 1.35 (28-Feb-2004)
e2fsck: No such file or directory while trying to open /dev/LogVol01

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>



And finally we then ran the command:

[root@rs-svr1 /]# e2fsck -b 8193 /dev/sdb2


The output was as follows:

e2fsck 1.35 (28-Feb-2004)
e2fsck: Bad magic number in super-block while trying to open /dev/sdb2

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

 

[Annihilannic]

Okay, how about the output of vgdisplay -v?

Why did you put these disks in a new system by the way? Do you suspect some fault with the original hardware? Does the new system contain any other disks (i.e. another installation of Linux)? The reason I ask is that it's strange (to me at least) to see two LVM Volume Groups with the same name on one system.

Annihilannic.

 

[CarrahaG]

Hi

We ran the command:

[root@rs-svr1 ~]# vgdisplay -v


The output was as follows:

Finding all volume groups
Finding volume group "VolGroup00"
Fixing up missing format1 size (34.12 GB) for PV /dev/sdb2
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 34.09 GB
PE Size 32.00 MB
Total PE 1091
Alloc PE / Size 1090 / 34.06 GB
Free PE / Size 1 / 32.00 MB
VG UUID q382vr-U8lM-1ZYe-rM97-gWo0-EgSA-e2cSKG

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID mmugP9-wZFP-Irbm-lsdm-b1DI-LYbi-OR9pVY
LV Write Access read/write
LV Status available
# open 1
LV Size 32.06 GB
Current LE 1026
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID a9236S-60vM-HqgT-58ch-43Vg-9eLs-NzHfJo
LV Write Access read/write
LV Status available
# open 1
LV Size 2.00 GB
Current LE 64
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:1

--- Physical volumes ---
PV Name /dev/sdb2
PV UUID y8R3zV-rlXR-c7gG-NnI1-I1wZ-EBPE-fUlCen
PV Status allocatable
Total PE / Free PE 1091 / 1

Archiving volume group "VolGroup00" metadata (seqno 3).
Archiving volume group "VolGroup00" metadata (seqno 3).
Creating volume group backup "/etc/lvm/backup/VolGroup00" (seqno 3).
Finding volume group "VolGroup00"
Fixing up missing format1 size (34.14 GB) for PV /dev/sda2
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 34.12 GB
PE Size 32.00 MB
Total PE 1092
Alloc PE / Size 1091 / 34.09 GB
Free PE / Size 1 / 32.00 MB
VG UUID cb813R-Sw2F-RXRu-i78f-XSYq-VZ6L-WSLcWe

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID C2ShX6-UzXX-V6VP-vhUi-KYkH-rMeV-npQslo
LV Write Access read/write
LV Status available
# open 1
LV Size 33.09 GB
Current LE 1059
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:0

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID d2NNu9-sIg4-CO93-p6XO-PP5h-b4RG-WAnszO
LV Write Access read/write
LV Status available
# open 1
LV Size 1.00 GB
Current LE 32
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:1

--- Physical volumes ---
PV Name /dev/sda2
PV UUID c638qH-8iY3-255h-QEWL-GG64-G7qf-sTTbQB
PV Status allocatable
Total PE / Free PE 1092 / 1

Archiving volume group "VolGroup00" metadata (seqno 3).
Archiving volume group "VolGroup00" metadata (seqno 3).
Creating volume group backup "/etc/lvm/backup/VolGroup00" (seqno 3).


We actually used the old server with a different drive and installed a fresh Linux on it. We then added the drive with the problem so we can repair it if possible.

 

[Annihilannic]

Okay, it makes sense now. You should rename the one that you are not booted off to avoid any confusion; I'm presuming it's the one on sdb:

vgrename Zcb813R-Sw2F-RXRu-i78f-XSYq-VZ6L-WSLcWe VolGroup00_tmp

Then perhaps run another vgscan and hopefully you should be able to access its volumes separately through /dev/VolGroup00_tmp.

Then you should be able to try fsck /dev/VolGroup00_tmp/LogVol01, for example.

Annihilannic.