Recover from failed glibc upgrades

I recently found a bug with the udpated glibc packages on Red Hat's updates.redhat.com FTP server for Red Hat 9.0. If you try to upgrade to the new glibc packages, the process may fail leaving your server in very bad shape. Any command that you run will return "Segmentation Fault", so nothing will work including logging in. Your server will be unbootable when you restart and will hang at something that says "init: no more processes left in this runlevel", because the root filesystem won't be able to mount.

After I fixed this problem, I tried upgrading glibc again on the same test box that it crashed on earlier, but it worked the second time, so this is an intermittent problem. I searched the web and found that I'm not the only one to have this problem, so thats why I turned this into an FAQ.

If this happens to you, here's how I got the server to boot again. You might have more problems to deal with afterwards, but getting the server back up and running is most important.

Boot off of the RH90 CD #1 or a boot.iso CD. Enter rescue mode by typing "linux rescue" at the boot: prompt. When it asks to mount the root filesytem, say Yes.

After your at the shell, find out which glibc packages are installed...

rpm -qa --root /mnt/sysimage | grep glibc

... then delete them all individually using something like this....

rpm -e glibc* --root /mnt/sysimage --nodeps

Next, lets re-install the glibc RPMs from the RH90 CD...

rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-2.3.2-5.i386.rpm --nodeps
rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-common-2.3.2-5.i386.rpm --nodeps
rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-devel-2.3.2-5.i386.rpm --nodeps
rpm -Uvh /mnt/sources/RedHat/RPMS/glibc-kernheaders-2.4-8.10.i386.rpm --nodeps

Type exit to reboot from rescue mode. At this point, your server should be able to boot again.

After the server boots up, you'll probably have to rebuild your /etc/ld.so.conf file. If the file is blank, or doesn't exist, add this to it...

/usr/kerberos/lib
/usr/X11R6/lib
/usr/lib/sane
/usr/lib/qt-3.1/lib

...and then run ldconfig afterwards.


ChrisP
RHCE, LPIC-1, CCNA, CNE, MCSE, +10 others