Proxmox Web Security Issue

phoneguy

Guru
Joined
Jan 13, 2008
Messages
285
Reaction score
54
Ok so we just had a client in paid support inform us they thought they were hacked. We logged in and saw in the ps aux the following


root 12379 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.241 10000 0
root 12380 3.2 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.242 10000 0
root 12381 3.3 0.2 49960 10980 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.243 10000 0
root 12382 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.244 10000 0


There were about 2000 lines of that and the IP's kept changing as the port scan was running. Load average was up to 150 in TOP.

The file this time was located in /usr/books/.n I tar'd up the files and if anyone wants them please email Ward as I sent them to him also.

The only ports open on the firewall for this customer was 80, 5060 and 10000-20000 for SIP. Everything else was closed down. I see nothing in the access files for apache on someone failing to log in and fail2ban did not detect any failed logins. This was not a FreePBX Distro but a different FreePBX based system. They were running an older apache and php and advised them to upgrade those packages but the customer decided to do a re install instead and than upgrade the packages.

If anyone else runs into this please be very careful killing the processes as each time we have done this they than DoS attack the IP that you killed the process from.
 

malcolmd

Guru
Joined
Aug 12, 2010
Messages
101
Reaction score
7
So after they reinstall, can they trap all traffic going to that machine for a day or two so we can get a better idea of what's being exploited?
 

phoneguy

Guru
Joined
Jan 13, 2008
Messages
285
Reaction score
54
And here is another one from the same customer different box.

root 10209 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10210 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10211 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10212 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10213 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10214 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10215 0.2 0.1 7948 2060 ? Ss 15:20 0:00 sshd: root@pts/
root 10217 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10218 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10219 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10220 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10221 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10222 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10223 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10224 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10225 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10226 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10227 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10228 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10229 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10230 0.5 0.1 4752 1492 pts/2 Ss 15:20 0:00 -bash
root 10254 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10255 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10256 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10257 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10258 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10259 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10260 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10261 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10262 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10263 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10264 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10265 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10266 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10267 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10268 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10269 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
root 10270 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10271 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10272 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10273 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 10274 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
root 11773 0.0 0.0 1748 412 pts/0 S+ 14:20 0:00 ./exim new vuln
root 11814 0.0 0.0 4564 1064 pts/0 S+ 14:20 0:00 sh -c perl x.pl
root 11815 0.0 2.6 38448 34564 pts/0 S+ 14:20 0:00 perl x.pl 202.1
root 12314 0.0 0.0 1748 412 pts/0 S+ 14:21 0:00 ./exim new vuln
root 12377 0.0 0.0 4564 1068 pts/0 S+ 14:21 0:00 sh -c perl x.pl
root 12378 0.0 1.8 28208 24328 pts/0 S+ 14:21 0:00 perl x.pl 202.1
nagios 18092 0.0 0.0 5080 944 ? Ss 12:05 0:00 /usr/local/nagi
root 18899 0.0 0.0 1748 412 pts/0 S+ 14:29 0:00 ./exim new vuln
root 18953 0.0 0.0 4564 1068 pts/0 S+ 14:29 0:00 sh -c perl x.pl
root 18954 0.0 8.1 110128 106244 pts/0 S+ 14:29 0:00 perl x.pl 202.1
root 23242 0.0 0.0 5032 1156 ? Ss 13:28 0:02 SCREEN
root 23243 0.0 0.1 4752 1528 pts/0 Ss 13:28 0:00 /bin/bash
root 23261 0.0 0.0 4756 860 pts/0 S+ 13:28 0:00 /bin/bash

Notice the screen session. If you go into screen you will see the following.
[+] 202.161.45.177
[+] 202.162.217.59
[+] 202.162.217.57
[+] 202.162.217.56
[+] 202.162.217.58
[+] 202.162.217.60
[+] 202.162.217.61
[+] 202.162.77.52
sh: fork: Cannot allocate memory
[+] 202.162.78.9
[+] 202.162.77.36
sh: fork: Cannot allocate memory
[+] 202.162.78.8
sh: fork: Cannot allocate memory
[+] 202.162.77.25
sh: fork: Cannot allocate memory
[+] 202.162.78.59
sh: fork: Cannot allocate memory
[+] 202.164.172.195
[+] 202.164.172.197
sh: fork: Cannot allocate memory
[+] 202.164.189.226
[+] 202.164.189.225
[+] 202.164.189.228
sh: error while loading shared libraries: libtermcap.so.2: failed to map segment from shared object: Cannot allocate memory
sh: fork: Cannot allocate memory
[+] 202.165.247.91
[+] 202.167.231.47
[+] 202.167.231.46
[+] 202.168.64.111
[+] 202.168.64.110
[+] 202.170.120.224
[+] 202.170.120.225
[+] 202.170.122.83
[+] 202.170.126.52
[+] 202.170.127.103
[+] 202.170.127.76
[+] 202.170.127.66
[+] 202.170.127.73
[+] 202.170.127.9
[+] 202.170.127.84
[+] 202.170.38.231
[+] 202.171.133.31
[+] 202.171.136.198
[+] 202.171.136.195
[+] 202.171.136.205
[+] 202.171.136.214
[+] 202.171.136.217
[+] 202.172.169.16
[+] 202.172.169.42
[+] 202.172.169.18
[+] 202.172.32.144
[+] 202.172.32.223
[+] 202.173.130.5
[+] 202.173.138.29
[+] 202.173.141.190
[+] 202.173.142.160
[+] 202.173.140.192
[+] 202.173.151.16
[+] 202.173.180.34

I tar'd up the files this time from /usr/books and sent them again to ward for anyone who wants them.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
And to Apache 2.2.3-43...

The CVE here:
http://www.cvedetails.com/cve/CVE-2011-0419/

Points to:
http://rhn.redhat.com/errata/RHSA-2011-0507.html

on the RedHat site, which indicates that their fix was made to apr-1.2.7-11.

...both of which we're distributing with *now.

So, as ezekielmudd mentioned, maybe Apache / APR isn't the right direction?

Reviewing the notes, it doesn't appear that these would compromise a system other than creating a Denial of Service condition. We still need to figure out how they're getting in. We seem to have a pretty good handle on what's going on once they arrive. :crazy:
 

ezekielmudd

New Member
Joined
Jan 11, 2009
Messages
20
Reaction score
5
That machine is totally compromised. It's a goner. :cryin:

They're exploiting exim - the mail program. Check out this slashdot article: http://it.slashdot.org/story/10/12/10/1529206/Remote-Exim-Exploit-In-the-Wild

Bite the bullet and take that machine off line immediately!

You can just shut down the outward facing eth0 or eth1 if you want to do a forensic examination the system while sitting in front of the machine.

The hacker is using a file called "x.pl" to do nasty things. Find it and delete it. That will clean up the stuff you can see.

If it were me, I'd format the disk and reinstall the most recent version of the OS.
 

phoneguy

Guru
Joined
Jan 13, 2008
Messages
285
Reaction score
54
Just for clarification they did not have exim installed on the system that got hacked just so nobody things the machine got hacked through exim. They were just trying to hack into other machines that had exim running.
 

ezekielmudd

New Member
Joined
Jan 11, 2009
Messages
20
Reaction score
5
Ah, my mistake. Sorry about that Chief.

My money is back on the phpmyadmin exploit then.

I've experienced that one personally.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
Here's Where We Are

Vulnerability: Any stock CentOS 5.x or 6.x system with Apache and PHP exposed to Internet access with no IP address restrictions (WhiteList).

UPDATED: See this post for the latest information. It appears that the security hole may be limited to 64-bit Linux systems as well as Proxmox servers with OpenVZ support.

How Do They Get In: Some (perhaps unknown) vulnerability in stock versions of Apache and PHP on CentOS systems allows the attacker to gain system access. We really don't know any more than that at this juncture. But this does not appear to be a PHPmyAdmin exploit as that utility is locked down by secure htaccess password on some systems that have been compromised. Fail2Ban is not detecting hack attempts so it appears the attacker is walking right in with this exploit.

Privileges: Still unclear whether attacker is gaining root access or merely same access as enjoyed by Apache on the attacked system. To do what they're doing would NOT require root privileges on your system. The attacker brings a customized version of WebMin with their own password.

What Happens Once They're In: In a nutshell, your system is turned into a zombie. Using perl and WebMin (their own version), they can interconnect your server into a worldwide network of machines used to launch denial-of-service and other malicious attacks against other systems on the Internet.

How Do I Know If My Machine Has Been Compromised? Examine some of the previous comments in this thread. Run ps awx on your server and look for long lists of processes running perl scripts. Look in the /usr directory for a directory called game, games, books, etc. Inside those directories, run ls -all which will show hidden files beginning with a period. There will be a directory called .n or .s or something similar. Look in /etc/cron.daily. There will be a new script as outlined in this thread. NOTE: The zombie software is old and signatures already exist in anti-virus programs. The exploit to gain access may be entirely new.

What Should Be in /usr? On a stock PIAF system, you should see the following directories:

bin games kerberos libexec man share tmp etc include lib local sbin src X11R6

The games directory will be empty when you ls -all

What Should Be in /etc/cron.daily? On a stock PIAF system, you should see the following files:

0anacron cups makewhatis.cron prelink tmpwatch 0logwatch logrotate mlocate.cron rpm


How to Fix It: If your system has been compromised, reformat the disk and reinstall. If they haven't gotten in or if you've started over, (1) immediately turn off Internet access to web services on your servers. You can implement a whitelist of safe IP addresses for web access using IPtables or your hardware-based firewall. We will post a HOW-TO shortly. (2) Upgrade Apache to latest version and PHP to latest 5.2 or 5.3 version. For PIAF users, we are working on an upgrade which should be available in the next couple days. It's not easy!

NOTE: Just because your system has not yet been compromised does NOT mean you are safe. Your system still needs to be secured. Turn OFF Web Access Now!

For Standard PIAF servers and Proxmox PIAF installs ONLY:

How to Temporarily Disable Web Access: Log in as root. Issue command: iptables -D INPUT -p tcp -m tcp --dport 80 -j ACCEPT

How to Temporarily Enable Web Access: Log in as root. Issue command: iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

How to Temporarily Enable Web Access for Specific IP Address : iptables -A INPUT -p tcp -m tcp -s 123.45.67.8 --dport 80 -j ACCEPT

NOTE: These settings only work on stock PIAF servers, not hosted systems such as RentPBX. These settings are temporary until a reboot or service iptables restart.

For RentPBX-hosted PIAF servers ONLY:

How to Temporarily Disable Web Access: Log in as root. Issue command: iptables -D RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT

How to Temporarily Enable Web Access: Log in as root. Issue command: iptables -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT

NOTE: These settings only work on RentPBX PIAF servers, not standard PIAF systems. These settings are temporary until a reboot or service iptables restart.
 
Last edited by a moderator:

alesko

Guru
Joined
Jan 22, 2011
Messages
22
Reaction score
0
One question - is it possible to be more secured with HTTPS and if the HTTPS is accesses from different port then 80 or 443 from the outside?
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

The one set of web pages which would be common on most aggregations would be the ARI, which in some versions of PiaF, I understand, is secured by directory access.

I wonder if there is a correlation between hacked machines, and whether the ARI is exposed to the internet without extra apache directory security or not?


Joe
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
ARI is password-protected via htaccess in the default PIAF installs: /etc/pbx/httpdconf/ari.conf
 

newvoiper

Member
Joined
Nov 20, 2010
Messages
94
Reaction score
25
Thanks for the prompt update. For those of us less familiar with all of the entry points for web access to incredible pbx: if sticking with the default configuration + travelling man, will temporarily disabling the firewall port forwarding rules that were needed for travellin man, suffice to turn off Web Access?
 

bucasia

Guru
Joined
Sep 26, 2008
Messages
98
Reaction score
1
This information relates to this post on the FreePBX site - freepbx.org/forum/freepbx-distro/distro-discussion-help/freepbx-distro-rooted


This information only relates the the OPs post. He was kind enough to let me have a copy of the hacked servers drive image. This may or may not relate to any other hacking cases.

I'm 99% sure that the exploit used a vulnerability in phpmyadmin.

The exploit is similar to the the one included here - http://sourceforge.net/tracker/?func=detail&aid=3045132&group_id=23067&a..., although the exact code there does not work (at least I couldn't get it to work). The version of phpmyadmin on the hacked server was 2.11.9.6-1

If you have this file on your server I recommend deleting, or making it inaccessible ASAP - /usr/share/phpmyadmin/scripts/setup.php. The hackers had deleted this file on the hacked box as part of the hack.

I will post more information and the exact exploit when I find it.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
To temporarily disable Travelin' Man, SSH or log in to your server as root and issue this command:

iptables -D INPUT -p tcp -m tcp --dport 83 -j ACCEPT

To enable Travelin' Man,

iptables -A INPUT -p tcp -m tcp --dport 83 -j ACCEPT

All Travelin' Man entries including either of the above are erased by doing:

service iptables restart

So... for the time being, the secure way to use Travelin' Man would be to disable access as the default (as shown above).

When you need to modify a Travelin' Man IP remote address, SSH into your server, enable Travelin' Man (as shown above), run the Travelin' Man web app to set the new IP address, and then disable Travelin' Man again via SSH (as shown above).
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
Interim fix for CentOS web vulnerability is now available here.
 
Last edited by a moderator:

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
ARI is password-protected via htaccess in the default PIAF installs: /etc/pbx/httpdconf/ari.conf

Hi

I realise that this is the case now, as I alluded to in my post. This is a relatively recent innovation, and it is likely that there are number of older PiaF systems that do not have apache directory access enabled with /etc/pbx/httpdconf/ari.conf.

Thus my point was, given this is probably the one set of web pages which are common to all distros, which may not be secured, other than by its own security.

To put it another way, do we know of any recent installs of PiaF, where the recordings directory had an extra layer of apache security on it that have been hacked.


Joe
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,201
Reaction score
5,220
For the record, we have yet to receive first-hand information from a PIAF user that any PIAF system has been hacked. So, no, we have no further information on the ARI issue.
 

eCase

New Member
Joined
Jan 26, 2011
Messages
161
Reaction score
0
For the record, we have yet to receive first-hand information from a PIAF user that any PIAF system has been hacked.
This reminds me of high school debate team...
->Absence of Evidence v. Evidence of Absence<-

Is anyone else breaking out in a cold sweat?
:idea::idea::idea::idea::idea:
Wait a sec...
Ward, how about setting up a $50 Fan Club (or higher amount) where donators receive a:
Gratuitous, zero-warrantied/guaranteed/liability-free
NETWORK/PIAFserver SECURITY CHECK!!!
Donators provide just the ip of their piaf server, and you guys run the latest security vulnerability test to see if they pass/fail???
 

Members online

Forum statistics

Threads
25,810
Messages
167,754
Members
19,240
Latest member
nikko
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top