HTTP exploit within Elastix code...asterisk, guidance please

[cefinla]

Hello All-- First time posting hear.

I need some guidance with a security issue.

Hear is a note I took at the time we noticed the attack:

The attack occurred on our clients virtual machine (Debian), which we host on site. This VM is running our clients PBX and it appears the attacker used an HTTP exploit and weak Elastix code. The intruder created file in /var/

http://www/html/admin/elastix.php

(e.g., small shell program, see attached) directory. The attacker has root access to Apache with user privilege 'asterisk'.

This is a link to the PHP file which alerted us to this attack:

http://blog.protectedareas.us/public/bak/elastix.pdf

and it is attached.

We are operating on the assumption that the intruder wanted to gain access and place international phone calls. And we are working to correct this issue.

My question is what can we do to mitigated this type of security breach in the future?

In this case the PHP file was created with malicious intent and we were caught unaware. We only noticed this attack by chance when manual log files.

We can configure snort, or some other IDS, to alert us when files are created in /var/

http://www/html/admin/

and create other rules to alert us of suspicious activities and notify our team.

~Thanks

 

[Noway2]

Cefinla, welcome to tek-tips.

I am going to start by trying to recap the facts of your situation:
1 - this is a Debian based virtual machine being used as a PBX system.
2 - You are using Elastix as the Asterisk/pbx system.
3 - You found a modified file, which has been attached in PDF format.
4 - Logs indicated signs of the breach
5 - You believe that the intruder has root level privilege.

I am one of the security incident responders on a popular Linux forum and we strongly advocate a methodical, evidence based, approach to handling these types of situations. What we DO NOT recommend is either a wipe and re-install without an investigation OR leaving the system in operation during an investigation. The first thing that we recommend you do is isolate the system by either disconnecting the network cable or putting up a firewall to allow only SSH access from a trusted source. Do not reboot the machine and modify as little as absolutely possible. Your objective is to gather as much information as possible regarding the compromise and attempts to "clean" the system or keep it operational will only destroy the much needed evidence.

Once you have secured the machine, I recommend reviewing the CERT Intruder detection check list. Though it is dated, it still contains valuable information and outlines the steps necessary to perform in such an investigation. Here is a link.

You will want to follow the steps outlined in the procedure. Specifically, you will want to look for signs of changes to the user accounts, cron tabs, look for hidden files (especially in strange locations). The checklist that I linked to provides the commands on how to perform most of these functions. You will want to check for modified binary files, which you can do using the debsums utility.

You will want to examine the network connections and the process tree. *This is a big part of why you don't want to reboot or try to clean the system* I would recommend that you execute the following command set as root to gather this information:

'( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /tmp/log.txt'

This will generate a large file called log.txt in your temp directory containing this information. You will need to examine the output VERY carefully. You can post it as an attachement to this thread if you require help with the analysis.

Before you run this command, I would recommend that you first verify the integrity of the binaries and download clean copies from the Debian repositories if need be.

Next you will want to examine the log files very carefully. The tool logwatch can be very helpful in this regard. If you use it you will want to run it with the following switches "--detail High --service All --range All --archives --numeric --save /path/to/logwatch.log" switches (substitute "/path/to/logwatch.log").

Now lets ask the following:
1)What version of Debian were you running and how up to date was it on the patches.
2)What version of Elastix were you running, and how up to date was it. Most importantly, are there known vulnerabilities for this version that could explain your compromise?
3)Are you running any other sort of server software, e.g. SSH, etc.
4)What evidence do you have, if any, that indicates how root level privilege was obtained?
5) Are you currently running any sort of HIDS, e.g. Tripwire, Aide, Samhain, Ossec? Similarly, do you run Snort and did it give you alerts?

In order to mitigate this type of event and prevent it from happening again it is of paramount importance that you determine exactly how you were compromised. Until you know this, there is no way to effectively plug the hole they used to get in.

 

[cefinla]

Thanks for the through reply.

I will try to address the questions that I can answer now.

Your recap list is accurate, but I would like to clarify number 3. The file e.g., elastix.php, was created not modified...this file was created in /var/

http://www/html/admin/

directory.

The attacker was using some GET mechanism to exploit a known bug in elastix. The attacker cachet the /etc/amportal.conf here we have user/pass for the user "asteriskuser" using this username they can access to the ALL PBX.

The fallowing lines:

197.195.232.104 - asteriskuser [03/Apr/2012:07:36:16 -0700] "GET /admin/config.php HTTP/1.1" 200 20994
197.195.232.104 - asteriskuser [03/Apr/2012:07:36:20 -0700] "GET /admin/config.php?handler=file&module=dashboard&file=dashboard.css HTTP/1.1" 200 2389
197.195.232.104 - asteriskuser [03/Apr/2012:07:36:20 -0700] "GET /admin/common/script.js.php HTTP/1.1" 200 1111

Now they are inside the GUI interfaces with all the privileges.

Thanks for the link.

I appreciate the advice and it was very helpful.

I am working to answer the five questions at the end of your reply and I will post my findings later.

 

[cefinla]

Five questions:

1)What version of Debian were you running and how up to date was it on the patches.

A: Correction, we are running CentOS 5.4 (Final). Kernel Linux 2.6.24-12 SMP 64bit

2)What version of Elastix were you running, and how up to date was it. Most importantly, are there known vulnerabilities for this version that could explain your compromise?

A: Elastix version 1.5. 64bit We installed from the ISO CD.
The vulnerabilities are basically package vtigercrm.

Next this:

IP Adress - - [03/Apr/2012:07:35:33 -0700] "GET /vtigercrm/graph.php?module=../../../../../../../../../etc/amportal.conf%00 HTTP/1.1" 200 9843

Inside the file amportal.conf you have the username asteriskuser and the passwd. You can login to the GUI using that info. Here is the second what they did.

IP Address - asteriskuser [03/Apr/2012:07:36:16 -0700] "GET /admin/config.php HTTP/1.1" 200 20994

IP Address - asteriskuser [03/Apr/2012:07:36:20 -0700] "GET /admin/config.php?handler=file&module=dashboard&file=dashboard.css HTTP/1.1" 200 2389

Next the attacker use some mechanism for writing file to the folder /var/

http://www/html/admin

because they had owner and group permissions for the apache/asterisk.

3)Are you running any other sort of server software, e.g. SSH, etc.
A: We are running:

Apache - Open for internet
SSH - Close for internet
Asterisk - Close for internet
MySQL - Close for internet

Just the services http & https are open for remote access.

4)What evidence do you have, if any, that indicates how root level privilege was obtained?

A: We don't have any evidence they have root level privilege.

Q: But how could the attacker have created the elastix.php in /var/

http://www/html/admin/

directory?

5)Are you currently running any sort of HIDS, e.g. Tripwire, Aide, Samhain, Ossec? Similarly, do you run Snort and did it give you alerts?

A: No!

 

[Noway2]

Thank you for the clarification. It looks like you have obtained a pretty good understanding of how you were compromised. Good Work, if I may say so!

There are a couple of pieces of information in your last post that may be key to the situation:
1) CentOS 5.4 (Final). Kernel Linux 2.6.24-12 SMP 64bit. Unless you have been regularly updating, which given the release dates of other software you are running I suspect you have not, this linux version is pretty long in the tooth, being a 2009 release. Centos 5 is still active, but is currently at 5.8.

2) Elastix version 1.5. This too is about the same vintage (I cleared the page with the exact release, but it is from about the same time). The current version being 2.3. The change and fix log in subsequent versions is extremely long.

3) "GET /vtigercrm/graph.php?module=../../../../../../../../../etc/amportal.conf%00 HTTP/1.1" 200
I think that this part is key. Searching on this term brought up this following page:

http://www.ush.it/team/ush/hack-vtigercrm_504/vtigercrm_504.txt.

The page is based upon vtiger 5.0.4 (current version being 5.4). Your post doesn't mention the version, but I think that this version was from around your applications time frame. I am not sure of the release date but the document discusses and links to a full write up regarding a multitude of vulnerabilities including XSS and File Inclusion, which it looks like were used against you. I suspect that this ultimately was where you were "had".

4) You have done everything correct as far as what you allowed open to the internet and what you have not. It looks like a known vulnerability may have been exploited. Clearly the GET statement allows them to 'escape' the web file directory and it looks like it is somehow done through the module parameter (noting that it is a GET parameter) and it is apparently not filtered properly in the application.

As long as you are certain that this is how they gained entry, and it looks like the evidence you have is supporting it, my recommendations would be:

1 - rebuild your system with up to date components. This will close a gross number of known vulnerabilities related to web stack exploits.
2 - Install an intrusion prevention system, in your case a HIDS like Samhain, Aide, or Ossec that will provide real time monitoring and active response to threats.
3 - Implement a policy of maintenance and upgrades to keep your applications up to date.
4 - Install and use Snort and try to couple it with your HIDS. The advantage to Snort is that it will pick up SIP / VOIP traffic. Be sure to use the Emerging Threats rules which will help with these types of traffic conditions
***5 - Use SELinux with your Centos. SELinux, which works on context should have prevented a web user from being able to read a file in the /etc directory, which has a different context. This would have prevented them from being able to read the file and gain access to the Asterisks user information.

 

[cefinla]

Thanks for the kind complement.

One of our fixes is to stop deploying Elastix and go a different direction. However, we still need to follow your recommendations you have suggested in your reply. Thanks for the recommendations...now we just need to research our options and take the appropriate steps.

I appreciate your input...

 

[Noway2]

I am happy to have been of assistance.

Good luck with your new system and if you have any questions, please don't hesitate to ask.

 

[cefinla]

With regards to the suggestions to mitigate these attacks in the future...

My working knowledge of Snort is limited.

We would like to dedicate a server to Snort and configure Snort to monitor specific servers on our local network. Or, have Snort monitor IP address X on our LAN and monitor specific directories on the server X.

So, in this specific example we will install a Snort on xxx.xxx.10.12 and create a Snort rule to monitor for file creations on xxx.xxx.10.111 in direcory /var/

http://www/html/admin/

Is this possible or does Snort have to be installed and ruing on the server you want to monitor?

 

[Noway2]

Snort can be running on any server on the LAN. It is a little bit processor intensive, so if you have a moderate sized network, dedicating a host to Snort is probably a good idea. You shouldn't have to mess with the rules to customize for your desired IP address. I would suggest letting it run, see what kinds of alerts it generates from normal traffic and then customize the rule set to ignore the normal events. If you want to restrict the data to a particular host, or set of hots, you could wall it off with IPtables, though I think would be of best use to let it run against the majority of your traffic and then customize as required.

You can even run Snort on a non-configured NIC card (no IP address, no gateway, nothing) so that the machine will be effectively invisble to your LAN.

Personally, I use a SPAN port in the swich to capture the traffic on the desired nets and then pass this into a non-configured interface.

 

[cefinla]

Hello All--

We have made some good progress...we have now have installed Security Onion in a virtual environment, on our network. We are using Squert, Snorby, and Squil to monitor events as the occur.

Our sensor appears to only be monitoring traffic on our private network (subnet), 10.10.xxx.xxx

The next step is to configure our Snort sensor to monitor all traffic coming from our main switch e.g., monitor all traffic on our network.

We will need to configure Snort to watch the SPAN port on our main switch. Can anyone advise on how bet to achieve this goal? Do we need to add a network in the Snort config file? Or is there another way to best and easily achieve our goal to monitor all traffic on our network.

~Thanks

 

[Noway2]

Once you have the SPAN port monitoring all of the traffic, I would recommned bringing that line to a non configured NIC on the machine running snort. Then run an instance of snort configured to "listen" on that interface. For the purpose of running snort, the interface does not need to have an IP address or gateway assigned. This makes it much harder to detect its presence on the network.

 

[cefinla]

Today we tried to add a virtual ethernet device in Proxmox for vlan xxx and add IP address 66.xxx.xx.0/24 to the appropriate section in our snort config file (/etc/nsm/onion-eth0/snort.conf) This IP address corresponds to the new vlan we are attempting to make available to our Snort sensor for inspection.

# Setup the network addresses you are protecting
ipvar HOME_NET [192.168.0.0/16,10.0.0.0/8,172.16.0.0/12,66.xxx.xx.0/24]

and changed ipvar value from 'any' to !$HOME_NET

# Set up the external network addresses. Leave as "any" in most situations
###ipvar EXTERNAL_NET any
ipvar EXTERNAL_NET !$HOME_NET

Our intention was to further configure our corporate NIDS to inspect a large segment of our network traffic. After we made these configuration changes we tested and it did not appear we were NOT successful.

Any feed back or insight into our error is much appreciated.

~Thanks in advance...

 

[Noway2]

I assume by tested and not successful you mean that you did not see any alerts? My experience with new snort installations, a small step test process is usually best, as they often times seem to not work.

The first thing I would suggest doing is to use tcpdump and see if you are actually getting the traffic appearing from your span port. I would suggest that you filter based on something you can identify, such as going to an oddball site or a particular IP address or something and then using a machine to try to access that resource. See if you get the anticipated packets.

Once you are sure the data is coming in, then you can use this test rule in your snort configuration (add it to your local.rules file)

alert tcp any any <> any 80 (msg: "Test web activity";sid:1000001;)

This will cause any port 80 traffic to generate an alert called Test web activity.

If you get this alert, then you know that snort is monitoring. As a next step, I would use a tool like Niko and deliberately bang at a machine that snort is supposed to be monitoring and see if you get alerts that way.

 

[cefinla]

Thanks Noway2 for the prompt reply.

Yes, we are not seeing any alerts from 66.xxx.xx.0/24 VLAN and we think these are complication because we a running Security Onion in a virtual environment.

We have now waited and we have not noticed any alerts, or other evidence, that our recent configuration changes have been a success.

Security Onion is running well and we am glade we chose it for this project. Our NIDS does NOT have access to a large amount of our network traffic and we need to fix this. Currently, our NIDS is only analyzing a small amount of our traffic.

I think our team is now agreement, that we need a dedicated physical box to our corporate NIDS, and place it at the edge of our network where we will have more success and the ability to scale up as needed.