Hi all, I have a serious problem

[phuctran]

Hi all,

I have a serious problem. I have a Linux Redhat 6.2 box. Recently, when I use IP sniffer to check, I see a connection to a chatroom (nobody logs in at that time, also it happens right after I restart the system).

1. It tries to connect to "Haarlem.NL.EU.UnderNet.Org 433 _lamer- Sysop- :Nickname is already in use.." with the nick "NICK Sysop-..UnderN".
2. Then that IRC server returns "Haarlem.NL.EU.UnderNet.Org 433 _lamer- Sysop- :Nickname is already in use.."
3. My server sends "S¨AWAY :Sug sule de moldoveni toata ziua.
ysop"
4. Then that IRC server returns "Haarlem.NL.EU.UnderNet.Org 306 _lamer- :You have been marked as being away."

After that, the process begins from step 1 again. It sends these info every 2-4 seconds.

The IP address of that IRC chatroom is 62.250.14.6.

Following is all the processes running on the Linux box (ps -Af). Could anyone find the program (you think) it is the cause.

One more thing, in windows they have autoexec.bat to run programs on booting. Where is that file (or directory) in Linux? I couldn't find anything like that in Linux (sc directory ..). I am not really good with Linux but have some experiences. Please just tell me anything you think of, if I don't undertand well I can look it up.


UID PID PPID C STIME TTY TIME CMD
root 1 0 0 10:31 ? 00:00:04 init [5]
root 2 1 0 10:31 ? 00:00:00 [kflushd]
root 3 1 0 10:31 ? 00:00:00 [kupdate]
root 4 1 0 10:31 ? 00:00:00 [kpiod]
root 5 1 0 10:31 ? 00:00:00 [kswapd]
root 6 1 0 10:31 ? 00:00:00 [mdrecoveryd]
bin 320 1 0 10:33 ? 00:00:00 portmap
root 335 1 0 10:33 ? 00:00:00 [lockd]
root 336 335 0 10:33 ? 00:00:00 [rpciod]
root 345 1 0 10:33 ? 00:00:00 rpc.statd
root 359 1 0 10:33 ? 00:00:00 /usr/sbin/apmd -p 10 -w 5 -W -s /etc/sysconfig/apm-scripts/suspend -r /etc/sysconfig/apm-scrip
root 410 1 0 10:33 ? 00:00:00 syslogd -m 0
root 419 1 0 10:33 ? 00:00:00 klogd
daemon 437 1 0 10:33 ? 00:00:00 /usr/sbin/atd
root 451 1 0 10:33 ? 00:00:00 crond
root 469 1 0 10:33 ? 00:00:00 inetd
root 483 1 0 10:33 ? 00:00:00 lpd
root 528 1 0 10:33 ? 00:00:00 sendmail: accepting connections on port 25
root 543 1 0 10:33 ttyS0 00:00:00 gpm -t ms
xfs 595 1 0 10:33 ? 00:00:00 xfs -droppriv -daemon -port -1
root 604 1 0 10:33 ? 00:00:24 httpd
root 655 1 0 10:34 tty1 00:00:00 /sbin/mingetty tty1
root 656 1 0 10:34 tty2 00:00:00 /sbin/mingetty tty2
root 657 1 0 10:34 tty3 00:00:00 /sbin/mingetty tty3
root 658 1 0 10:34 tty4 00:00:00 /sbin/mingetty tty4
root 659 1 0 10:34 tty5 00:00:00 /sbin/mingetty tty5
root 660 1 0 10:34 tty6 00:00:00 /sbin/mingetty tty6
root 661 1 0 10:34 ? 00:00:00 /usr/bin/kdm -nodaemon
root 737 661 0 10:41 ? 00:00:00 /etc/X11/X -auth /usr/X11R6/lib/X11/xdm/authdir/A:0-rz9vtb
root 741 661 0 10:41 ? 00:00:00 -:0

 

[ifincham]

Hi,



Looks like you have a problem there. I can't see anything really obvious from the process list. I would do (as root) :



cd /etc

grep -r 'Haarlem*' *



This should find any files within the /etc directory structure containing the string shown in the quotes. If you get nothing try it from the system root directory, i.e. search the entire filesystem.



Under Redhat the main initialisation scripts are as follows :



/etc/rc.d/rc.sysinit

/etc/rc.d/rc.serial (if present)

/etc/rc.d/rc

/etc/rc.d/rc.local



the 'rc' is called everytime there is a change in system runlevels, e.g. from 3 to 5 when you go into X. Under Sysv init, there are a whole load of scripts that are run from /etc/rc.d/init.d under symbolic links from runlevel related directories - e.g. runlevel 5 calls every symbolic link in /etc/rc.d/rc5.d . It can really get quite complicated. Incidentally, the last script run on normal start-up (before logon) is the rc.local script and that is the most obvious place to put customised routines.



Hope this is of some help, Iain

 

[marsd]

lsof is your friend here.
lsof | grep :
will give you all registered listeners and sock processes.
Check this and see if you get any more info.

There are so many problems (security wise) with the 2.2
kernels to 2.2.19 that you should really upgrade to 2.2.19,
or the 2.4 series.

Have you filtered this address?
2.2 kernel
ipchains -A -s 0/0 -d 62.250.14.6 -j DENY
Check for at and cron jobs as well -
man at ; man cron
(you can use the same script below with cron -l in place of ps -u for cronjobs, the same with at.)

Lastly check users:
#!/bin/sh
names=` awk ' FS=":" {
if ($1 ~ /^[a-zA-Z]/) {
print $1
}' /etc/passwd | sed '/root/d'`

for x in `echo $names`
do
echo -e "`ps -u $x` , $x is the owner."
done

If none of this works then you may have trouble.

 

[phuctran]

Ifincham,
I tried all your suggestions but could not find anything. I am in big trouble now (however, good thing is that the server is for developing only, so I just disconnect it). Could you please give me more suggestions. Thank you.

Marsd,

Following is the result from "lsof | grep :". Do you see anything wrong here. Thank you.

portmap 320 root 3u IPv4 364 UDP *:sunrpc
portmap 320 root 4u IPv4 365 TCP *:sunrpc (LISTEN)
rpc.statd 345 root 0u IPv4 405 UDP *:946
rpc.statd 345 root 1u IPv4 408 TCP *:948 (LISTEN)
inetd 469 root 4u IPv4 524 TCP *:ftp (LISTEN)
inetd 469 root 5u IPv4 525 TCP *:telnet (LISTEN)
inetd 469 root 6u IPv4 526 TCP *:login (LISTEN)
inetd 469 root 9u IPv4 527 TCP *op3 (LISTEN)
inetd 469 root 10u IPv4 528 TCP *:imap (LISTEN)
lpd 483 root 6u IPv4 546 TCP *rinter (LISTEN)
sendmail 528 root 4u IPv4 590 TCP *:smtp (LISTEN)
in.telnet 1058 root 0u IPv4 31601 TCP spock.com:telnet->SPOCK01:2072 (ESTABLISHED)
in.telnet 1058 root 1u IPv4 31601 TCP spock.com:telnet->SPOCK01:2072 (ESTABLISHED)
in.telnet 1058 root 2u IPv4 31601 TCP spock.com:telnet->SPOCK01:2072 (ESTABLISHED)
in.telnet 1218 root 0u IPv4 31778 TCP spock.com:telnet->SPOCK01:2097 (ESTABLISHED)
in.telnet 1218 root 1u IPv4 31778 TCP spock.com:telnet->SPOCK01:2097 (ESTABLISHED)
in.telnet 1218 root 2u IPv4 31778 TCP spock.com:telnet->SPOCK01:2097 (ESTABLISHED)
X 1631 root 0u IPv4 32241 TCP *:6000 (LISTEN)

** I changed the real address to spock.com

and X is (from ps -ef)
root 1631 661 0 13:56 ? 00:00:00 /etc/X11/X -auth /usr/X11R6/lib/X11/xdm/authdir/A:0-rz9vtb

 

[phuctran]

Hi all,

I use netstat -ae and get the following result. Could someone explain the second line (CONNECTED) for me. What is @0000001b?

Thank you.

Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 0 [ ACC ] STREAM LISTENING 518 /dev/printer
unix 0 [ ] STREAM CONNECTED 183 @0000001b
unix 3 [ ] DGRAM 438 /dev/log
unix 0 [ ] DGRAM 714
unix 0 [ ] DGRAM 666
unix 0 [ ] DGRAM 510

 

[marsd]

maybe the last is worth a try:
ps -aux | grep "@0000001b"
Did you check and see what user processes were registered?

 

[phuctran]

Marsd,

How can I check registered user processess?
Thanks.

 

[phuctran]

Marsd,

This is what I got from ps -aux | grep "@0000001b"

root 725 0.0 0.9 1360 508 pts/0 S 18:34 0:00 grep @0000001b

it does not appear when I use ps -ef. Do you know how to kill it completely?

 

[phuctran]

Marsd,

More info, the PID of
root 725 0.0 0.9 1360 508 pts/0 S 18:34 0:00 grep @0000001b
is not constant. If I run ps -aux| grep "@0000001b" many times then I won't see it sometimes; however, if I see it then it has different PID. I think that it creates a new PID whenever it sends info out to the chatroom.

Any idea?
Thank you.

 

[marsd]

No , you are seeing the pid of the grep process.
I do not see where this irc process is being called from locally.
I do not know what your network looks like but I would
filter and log traffic to this address and run tcpdump against traffic to this address to make sure that the source is local.
If it is and you still cannot pinpoint the process, then
you must assume that you have been rooted, and a rootkit
is possibly in place. Otherwise don't forward traffic and
filter this address.

 

[ifincham]

Hi again,

Hmmm, there are so many possibilities here. Obviously if you have a trojan / backdoor /worm operating then it must be initiated somewhere. Thats why my first approach would be to look through the init process to try to see where it kicks-off, i.e how it loads.

Its possible that a system binary (even the kernel itself) has been patched with some dodgy code and that would be why nothing unusual appears on the ps list, etc. Have you tried to run 'chkrootkit' on your system ? You can get it from

http://www.chkrootkit.org

- its a tool that searchs for various worms, etc. on your box. See also

http://www.cert.org/tech_tips/intruder_detection_checklist.html

and

http://www.cert.org/tech_tips/root_compromise.html

.

Have a look in /var/log/secure, /var/log/xfer and /var/log/messages for unusual activity although some of these progs delete their own log entries so you may find nothing much.

Marsd' suggestion to firewall the ip address is a good step (i.e. the ipchains above). Under kernel 2.4, you can use iptables to perform more sophisticated firewalling than is possible with ipchains, however - especially the 'stateful' parameters. For example, you can let packets out only if they correspond to an existing connex, etc.

(On your netstat output, I'm not absolutely certain what it means but '@0000001b' or similar as path is not unusual - it may just indicate that the socket concerned resulted from a child process rather than direct from a given binary - i.e. the path is not known.)


Hope this helps, Iain

 

[phuctran]

Iain and Marsd, Thank you very much for your help.

I guess the kernel or some binary files have been altered (but 90% I think the login file is altered so that it will get the information -- username and password -- to send to that chatroom). I think the best way is just kill the whole system and upgrade to Redhat 7.1. I don't have any data in that system, except using it for testing pop3 email on my program. (I haven't used it for a long time).

Oh, one more thing, that box is the only one Linux box in my network. The rest are NT boxes. Could there be any chances that the sniffer could get information travels from trusted NT boxes (like password ...). Anyway, I am going to change all passwords for NT boxes.

Thank you very much.

People have so much free time in their hands to do terible thing like that ...

 

[marsd]

phuctran,
I am still not 100% convinced that the linux box is the source of the net traffic you are seeing. You say that
you have NT boxes on the network, yes there is a very good
chance that an NT box is the source of the traffic.
I would carefully monitor my NT machines.
Most, not all,trojans and rootkits can be beat to some degree by using system tools like lsof and netstat and logging if only to the degree that you realize something is missing or strange from the output of these utilities.

If one machine is compromised there is a good chance that others are...There are innumerable tools and code floating
around for exploiting cifs/smb shares and ipc services.

You are running a number of insecure services(unencrypted) I would look into an apop arrangment for your pop mail, postfix or smail for mail, & ssh for access to your linux machine. These are just suggestions of course.

Another thing you could look into are replacing hubs with
switches, this might keep garden variety sniffers and script
kiddies under control. There are tools(dsniff and hunt) that
render this useless however and as a fix this is of little
value.