Certain root cmds and chkrootkit hang SSH

[thedaver]

I've got a rented box on a Virtual Linux server that has suddenly begun to hang the SSH terminal when I run them.

Commands "w", "ps ax", "lsof -Pni" all hang either immediately or partially through their output. The SSH session is not actually terminated - I can send CR's - but the output from the command is hung. In fact, I cannot CTRL-C or CTRL-Z the command as an interrupt.

I wondered if I've been hacked so I brought in chkrootkit, it hangs too, which I assume is related to whatever is causing the other commands to hang.

Suggestions welcome.

Internet Directory

http://dir.surfinbox.com/dir.php/Top

 

[xmb]

I would guess something aint right, anyone (badly) trojanned your binaries and stuff, but something exactly cant be told from the few infos you wrote.



. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[marsd]

Try looking at the op of lsmod, strace -f some simple binaries, and if these crash or block, import some fresh binaries into the box and see if these cause the same behaviors.
It does sound like a hack however. chkrootkit dying is
not a good sign..it doesn't do anything really special,
so it sounds like a design to obscure information gathering rather than flaky hardware.

 

[thedaver]

So far, no peculiar dates or symlinks on the binaries.

"ls" is working, my direct access to /proc is working.

lsmod shows nothing peculiar. Other services are working fine.

"top" and "killall" fail.
"free", "mount", and "df" work.

Rebooting has not helped. "ifconfig eth0" doesn't show any material traffic outgoing. nmap against the box doesn't show any unusual ports open.

Still doing the review...

marsd, lsmod and insmod look ok, owned by root.
I ran an "strace -f lsof" and got an endless list of replies like this:

[pid  2767] close(700)                  = -1 EBADF (Bad file descriptor)
[pid  2767] close(701)                  = -1 EBADF (Bad file descriptor)
[pid  2767] close(702)                  = -1 EBADF (Bad file descriptor)
[pid  2767] close(703)                  = -1 EBADF (Bad file descriptor)

I wonder if there's a filesystem corruption???




Internet Directory

http://dir.surfinbox.com/dir.php/Top

 

[thedaver]

After more straces I find that pid 3404 has gone bad/rogue in some way... I have tried using "kill -s 9 3404" as root with no evidence that the system has changed.

I went to the /proc folder and did this...

# rm -fr 3404
rm: cannot unlink `3404/fd/0': Operation not permitted
rm: cannot unlink `3404/fd/1': Operation not permitted
rm: cannot unlink `3404/fd/2': Operation not permitted
rm: cannot unlink `3404/fd/3': Operation not permitted
rm: cannot remove directory `3404/fd': Permission denied
rm: cannot unlink `3404/environ': Permission denied
rm: cannot unlink `3404/status': Permission denied
rm: cannot unlink `3404/cmdline': Permission denied
rm: cannot unlink `3404/stat': Permission denied
rm: cannot unlink `3404/statm': Permission denied
rm: cannot unlink `3404/cpu': Permission denied
rm: cannot unlink `3404/maps': Permission denied
rm: cannot unlink `3404/mem': Permission denied
rm: cannot unlink `3404/cwd': Permission denied
rm: cannot unlink `3404/root': Permission denied
rm: cannot unlink `3404/exe': Permission denied
rm: cannot unlink `3404/mounts': Permission denied
rm: cannot remove directory `3404': Operation not permitted

I don't know if this normal anyways..... but I'm surprised that root cannot kill something from /proc????




Internet Directory

http://dir.surfinbox.com/dir.php/Top

 

[xmb]

try insmod ip_conntrack and check /proc/net/ip_conntrack for any weird [ASSURED] marked connections.
what does strace -fF w return?

. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[thedaver]

OK, my software reboot hadn't taken. A cold reboot by the data center has resolved the issue. I'm now able to run chkrootkit with no problems. All services have returned to normal.

Apparently something in memory or on /proc (memory) got whacked, which messed up pid 3404 (I never found out what it was). Once the system restarted and cleaed up the pids from /proc, things seem normal.

I can't think of any other forensics to perform since functionality has returned and chkrootkit is happy. Other ideas?

Internet Directory

http://dir.surfinbox.com/dir.php/Top

 

[marsd]

Yeah, with a rootkit, you can assume nothing about the integrity of your system.
That lsof returns alot of nonexistent descriptors for
supposedly open file types is weird, but perhaps only
signifies a zombie, or broken process due to ??

I don't know. If it were me, I'd disconnect from
public/private interfaces(I'm sure you have), and start treating this as a hack even though I'm not confident
that it is. Working with flaked out hardware at a large
hosting facility I've seen totally weird scenarios like
this.

If it's worth it recovery and forensics is now key IMO.

 

[plamb]

The proc filesystem is a pseudo-filesystem which is used as an interface to kernel data
structures. It is commonly mounted at /proc. Most of it is read-only, but some files allow
kernel variables to be changed.
so why would you try to rm -fr asnd try to remove these items?? not a good thing to try.

 

[thedaver]

plamb,
The "attempt" was to try and clear a PID that wouldn't respond to kill -s 9. I couldn't use killall as it would hang when trying to read from all the PID files - it would hang on pid 3404. I had never done it, so I tried to physically remove the /proc pid 3404 entry directly, assuming that whatever damage it might do would be no worse than what was already in effect.

My reasoning may suck, but that's what I thought at the time.

Internet Directory

http://dir.surfinbox.com/dir.php/Top