On 9/24, I patched 4 RHL based servers with updates from RHL errata page, which included 1 checkpoint secure platform firewalls and 2 Adlex PLDs (RH 7.2 based) and one of our general use RHL 7.3 installations in our data center. Sometime earlier in the week, another admin team from another office patched a Checkpoint Secure Platform (RHL 7.2 based) firewall with the same patches, but downloaded from ftp://rpmfind.net/linux/redhat/updates/7.3/en/os/i386/openssh-server-3.1p1-14.i386.rpm.
Yesterday - 9/25 - the Adlex servers on my network and in my data center became unresponsive to the network and were rebooted. Neither recovered. On 9/24 the general purpose Linux 7.3 installation was rebooted and did not recover. The remote Checkpoint installation in the other data center was rebooted immediately on being patched and also did not recover. All showed the following error:
pivotroot: pivot.root(/sysroot,/sysroot/initrd) failed: 2
...
kernel panic: no init found. try pass init=option to kernel
even though init exists in /sbin, right where it should be.
Due to the fact that the same error occurred on 2 Adlex PLD's, one general use 7.3 installation (the 7.2 patches and 7.3 patches on the errata page have the same checksum, so I assume they are the same file) and 2 Checkpoint secure platform (7.2 based) installs in use at different data centers by different user bases and done by different admins with patches from different sources, so far I have to assume the OpenSSH patches were the source of my problems.
At our company, the Adlex servers are considered essential, critical systems for our
IT department, as well as being a favorite app of upper level management, and this caused quite a panic -- not to mention the firewalls cutting off an entire data center!
After several hours of troubleshooting I was able to recover the systems by creating a
/initrd directory at the root of the filesystem, setting ownership to root:root and
permissions to 755.
Has anyone else run into anything like this with these patches?
Yesterday - 9/25 - the Adlex servers on my network and in my data center became unresponsive to the network and were rebooted. Neither recovered. On 9/24 the general purpose Linux 7.3 installation was rebooted and did not recover. The remote Checkpoint installation in the other data center was rebooted immediately on being patched and also did not recover. All showed the following error:
pivotroot: pivot.root(/sysroot,/sysroot/initrd) failed: 2
...
kernel panic: no init found. try pass init=option to kernel
even though init exists in /sbin, right where it should be.
Due to the fact that the same error occurred on 2 Adlex PLD's, one general use 7.3 installation (the 7.2 patches and 7.3 patches on the errata page have the same checksum, so I assume they are the same file) and 2 Checkpoint secure platform (7.2 based) installs in use at different data centers by different user bases and done by different admins with patches from different sources, so far I have to assume the OpenSSH patches were the source of my problems.
At our company, the Adlex servers are considered essential, critical systems for our
IT department, as well as being a favorite app of upper level management, and this caused quite a panic -- not to mention the firewalls cutting off an entire data center!
After several hours of troubleshooting I was able to recover the systems by creating a
/initrd directory at the root of the filesystem, setting ownership to root:root and
permissions to 755.
Has anyone else run into anything like this with these patches?