/lib/libc.so.1missing - recovery.

[PGRAYSON]

One of our development Sun machines has somehow lost /lib/libc.so.1, resulting in nothing bar the shell being able to run. The only way I can get a directory listing is by doing 'echo *' from the shell.

I need to be able to get this up and running again quickly.

The developer responsible for this machine has left the company, so I don't know which version of Solaris it was running at the time. We do have media for most versions.

I'm not too familliar with system recovery on Solaris, but have plenty experiences on other systems. Can simply I boot from the release media, mount the root filesystem and copy the missing file across, and if so, how?

 

[Annihilannic]

/lib is normally a symbolic link to /usr/lib. Are you sure that libc is missing and not the symlink, or perhaps /usr isn't mounted?

In any case, what you suggest should work... try to shut down to the 'ok' prompt using init 0, then boot cdrom -s to get to single user. Annihilannic.

 

[PGRAYSON]

Both appear to be gone, the only way I can list files is by typing 'echo *' at a prompt, which of course doesn't list symlinks.

 

[Annihilannic]

echo * should list symlinks, though you can't tell whether they *are* symlinks. Does echo /lib/* list anything?

There are also statically linked binaries of cp, ln, mv, rcp and tar in /usr/sbin/static should you need them.
Annihilannic.

 

[PGRAYSON]

I've discovered another machine in the building running the same version of the OS, so pulling libc.so.1 off via tar shouldn't be a problem. Unfortantly that machine isn't running either.

On most Unix systems I've used before, all of /bin and /sbin are usually statically linked. Why is solaris so different?

 

[Annihilannic]

Good question, no idea... I presumed everything in .../sbin would be statically linked, being under the impression that the 's' meant 'static', but that's obviously not the case. Annihilannic.

 

[PGRAYSON]

Got it working now. Someone knew which version of Solaris it was previously running. A couple of other libraries had also gone, leaving me to think that someone had done something malicious to this. The machine is behind a NAT firewall so is inacessible from the outside world - it must have been someone internally.

 

[dierkerj]

I would suggest that you have in fact been hacked, although I don't know the details yet. I say this because the exact same thing has happened here to two different hosts.

One thing I would check for is the presence of an unusual tar file in the root directory of the affected machine. In our case the file is named 2003_Feb_05_22_30_01.tar. It had not existed prior to the problems. It is not really a tar file, but appears to a text file sized at 1024 bytes that does not appear to contain anything (per cat and vi).

 

[PGRAYSON]

It is firewalled, so unlikely to have been hacked directly. However there is one user who accesses this via a VPN.