Firewall Blacklist/Whitelist

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
The sophistication of the attackers is improving so it seems prudent to rethink our security which currently includes (1) a hardware-based firewall, (2) IPtables, and (3) fail2ban. What we think will help improve the overall security of our systems is fourth layer: a blacklist of known attack sites. So the plan is to build up a list of these IP addresses that can be programmatically downloaded and automatically loaded into IPtables regularly. As you discover new attackers, please post the IP addresses here, and we'll add them until we get an automatic way for everyone to do it.

While this is still a work in progress, we wanted to circulate a prototype for comments and suggestions. Currently, there are 3 apps:
  1. a PHP script which checks whether you have the current version of the blacklist
  2. a BASH script which downloads the latest BlackList and then loads the entries in the blacklist into IPtables.
  3. a BASH script which refreshes IPtables with your current copy of the blacklist
If you'd like to try it, log into your server as root and issue the following commands to download the software:

cd /root
wget http://pbxinaflash.com/firewall-blacklist.tgz
tar zxvf firewall-blacklist.tgz


To protect your server, issue the following command periodically:

/root/firewall-check.php


The first time it runs, there will be a couple of errors since you don't have the blacklist or MD5 of the blacklist on your server. Ignore them. It will grab the latest blacklist and load it into IPtables.

When you run it thereafter, it will check whether you have the latest blacklist. If you do, it will merely refresh IPtables with the blacklist you already have. If you don't have the latest blacklist, it will download it and then refresh IPtables.

If you ever reboot your system or restart IPtables, be sure to run firewall-check.php again since the blacklist is not (yet) automatically loaded!

Ultimately, you'll want to add this script as a cron job on your system so that it goes out every few hours and checks for updates. But don't do it yet... until we get the kinks out.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Checking for Creeps

Here's a simple script to check periodically for creeps trying to hack into your system. Just plug in your email address for [email protected]...

#!/bin/bash
found=`awk '/failed / { print 1010; exit }' /var/log/asterisk/full`
if [ $found = "1010" ]; then
grep "failed " /var/log/asterisk/full | mail -s HackAttempts [email protected]
fi


Then run the script as a cron job every few hours:

2 */3 * * * root /root/hackcheck > /dev/null
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
The Rest of the Story

In an effort to be as up to date as we possibly can be, we've added all of the entries from the VoIP Blacklist Project. So, after doing that, it was time for a lunch break. Came back, ran the update once, uploaded the new blacklist, decided to test it again, and the system hung. After 30 minutes of debugging, it turns out one of the entries in the blacklist wiped out two huge ranges of IP addresses (130,000+ addresses!) including the address of pbxinaflash.com. :lol: So... it essentially blocked us from being able to get back to the server.

Should this happen to you. Here's the quick fix.
  1. Reload IPtables: service iptables restart
  2. Remove /etc/firewall.md5
  3. Run /root/firewall-check.php again to grab new file that hopefully doesn't have us in it :crazy:
You probably also need to download the apps again which now include a safeguard to prevent this in the future.
 

lowno

Guru
Joined
Feb 18, 2009
Messages
125
Reaction score
8
Wonder if it would be possible to send fail2ban blocks to a centralized server that collects the ipaddresses. Then if more then one PBX bans the same ip address, it ends up in the blacklist. Or some multiple occurrence behavior.

This would really be a killer feature for PIAF, but would require resources and some donations.
 

bmore

Guru
Joined
Feb 12, 2009
Messages
118
Reaction score
1
Wonder if it would be possible to send fail2ban blocks to a centralized server that collects the ipaddresses. Then if more then one PBX bans the same ip address, it ends up in the blacklist. Or some multiple occurrence behavior.

This would really be a killer feature for PIAF, but would require resources and some donations.

That would be vulnerable to a type of denial of service attack. The database could be easily poisoned to cause it to block access to a voip provider etc.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Wonder if it would be possible to send fail2ban blocks to a centralized server that collects the ipaddresses. Then if more then one PBX bans the same ip address, it ends up in the blacklist. Or some multiple occurrence behavior.

This would really be a killer feature for PIAF, but would require resources and some donations.

The other problem is that the attackers are aware of fail2ban now and typically only send one or two attempts from a given IP. For example, here's ONE that we got about an hour ago. If you get these, we need to record them. This isn't someone dialing a wrong number. :rolleyes: So run the hackcheck script in the second message regularly and pass along the results.

[Nov 20 13:36:18] NOTICE[32322] chan_sip.c: Registration from '"3871802972"<sip:[email protected]>' failed for '64.4.205.88' - No matching peer found
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
I've run through the blacklist just to see where the bad IPs are coming from and the majority seem to be coming from the Pacific rim countries like Viet Nam, Indonesia, Philipines, etc.. I thought that there would be more from the former USSR states but they are pulling short.. Here's the list sorted by countries
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
We're now using a subset of the same list to protect pbxinaflash.com including this forum. Let us know if you have problems gaining access. You may be a lucky list entry, too. :biggrin5:

By the way, you can use a copy of the existing list (name it firewall.deny) to generate Apache .htaccess entries that block the same sites from the web with a command like this:

cp /etc/firewall.blocks /root/firewall.deny
sed -i 's|^|deny from |g' /root/firewall.deny
cd (to your web directory)
cat /root/firewall.deny >> .htaccess
 

vcallaway

Guru
Joined
May 6, 2008
Messages
170
Reaction score
2
Ok, grumpy old man talk now.

Back when we walked uphill in the snow barefoot to school there were some things similar to what you are doing now.

We had things called uumaps, host files and routing tables. Royal pain in the behind when a single typo could shut down your whole system. Kinda like what you just experienced.

I am a real big fan of the real time blacklisting used in mail filtering. A simple DNS query to an RBL server is trivial and does not require each PBX to do updates and hope they get it right. Fail2Ban can be setup to report when a fail comes in. The RBL server can decide what to do at that point. If all fails are reported then a pattern can be detected and locked out.

Not complaining and willing to help. Just been there, done that.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
By the magic of tunnelling via SSH and a USA server belonging to a customer, I can make this post, other than that, my access is blocked to PBX in a Flash.

Joe
 

mag

Guru
Joined
Jul 27, 2008
Messages
74
Reaction score
1
Crowdsecure

You might want to take a look at this project as well. It just started a while ago, but sounds like something similar to what you are looking to do.

http://www.crowdsecure.net/
 
Joined
Nov 14, 2008
Messages
1,398
Reaction score
320
The RBL solution is nice but its barely fast enough for a busy email server, I doubt it would be quick enough for a busy PIAF box...

Brian
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Ok, grumpy old man talk now.

Back when we walked uphill in the snow barefoot to school there were some things similar to what you are doing now.

We had things called uumaps, host files and routing tables. Royal pain in the behind when a single typo could shut down your whole system. Kinda like what you just experienced.

I am a real big fan of the real time blacklisting used in mail filtering. A simple DNS query to an RBL server is trivial and does not require each PBX to do updates and hope they get it right. Fail2Ban can be setup to report when a fail comes in. The RBL server can decide what to do at that point. If all fails are reported then a pattern can be detected and locked out.

Not complaining and willing to help. Just been there, done that.

Yeah. I endured that era, too. The advantage is it gives us another layer of protection with hundreds of thousands of problem IP addresses that already have been vetted by lots of other organizations. It may turn out to be a waste, but communications is so much better now. And a lot of this stuff gets updated in almost real time. That made it worth a try for us judging by the level of hack attempts we're seeing these days.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
The RBL solution is nice but its barely fast enough for a busy email server, I doubt it would be quick enough for a busy PIAF box...

Brian

I agree. The performance is pretty awful, and we'd be totally dependent on an outside lookup source for this one. With the system we're proposing, all of this is in-house on your own server. Worst case: You can't load an update for a bit because pbxinaflash.com is down. That's happened once in about the last 3 years... for a couple of hours.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

I love what is trying to be achieved, but there are two posts that illustrate potential dangers here, and that is denial of service. Ward blocked himself from the forum, and my IP has been blocked as well. I am using a proxy server in the UK, and for the last 18 months, that IP address has never been used for VoIP traffic.

I think the design of this has to be in two parts, the reporting of hacker's IP addresses, and the obviously the blocking of IP addresses.

The blocking is a fairly easy problem to solve, we could rsync to a list, or use a database lookup, download a text file - there are all sorts of ways of doing this. The reporting is a different matter.

What concerns me is the accuracy. Let's assume that someone takes a dislike to some telecom's company, say Vitelity, who support PBX in a Flash, and a range of IP addresses are blocked belonging to Vitelity's servers. This may damage their revenue stream, and cause a massive support headache for them, and with a smaller telco's, may irretrievably damage them.

For customers using Vitelity, they potentially stop getting phone calls, as they are blocking their IP addresses, and cannot make any. That could do damage to the customer of vitelity's business as well.

Imagine if someone on this forum got his box hacked, and it was used to try and hack other peoples PBX systems, his IP address, quite rightly would end up on the list, but when he rebuilt and secured his machine, how would he get his IP off the black list again.

Let's put out minds to a methodology of automatically creating an accurate blacklist, and disseminating that information quickly to everyone else.

Joe

PS. What about resurrecting the GEO-IP look up mentioned a while ago?
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Joe,
All very good points. Like I said, the only ones in the current listing are IPs of folks that have actually attempted to hack into us over the years plus the entries from the VoIP Blacklist. That's all that group does is collect and vet the list which is collected from a number of sites that monitor malicious activity on the net. I really don't want to be in that business, but I'm glad they are! The problem seems to be in a few of their CIDR Address Ranges, not in the individual entries. But I share their belief that we can sort the problems out rather than not having a list at all.

As for blocking entire countries, yes, that's possible. Do we really want to do that?? Who would choose the countries? I guess I still have an ounce of respect left for humanity and believe there are at least a few good souls left in (almost) every country.

As for who could add to the list, my initial thought is to limit it to the VoIP Blacklist + the gurus and senior members on this forum unless one of the gurus personally vouches for an entry submitted by someone else. If every guru would run and report results from the hackcheck script (on page 1) once a day, we'd have a terrific list in almost no time!

Finally, this blacklist is entirely optional. Nobody is going to ram this down anyone's throat. If you don't want to use it, by all means don't. You can also tell instantly if the list causes a problem. If it does, all that is required is service iptables restart to deactivate the list entirely. I would only mention that, after using it for 24 hours, I've gone from dozens of nightly hack attempts to exactly one in the last 24 hours. And now that IP address is in the list, too. ;)

phonedude.gif
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

The GEO IP is interesting, because it reduces the total number of people that can even gain entry to your PBX.

So, if you are in the USA, and all your extensions and carriers are also in the USA, it makes no sense to allow people access to your PBX from any other country other than the USA. If you have a remote extension in Hong Kong, then you can allow IP's from only Hong Kong and USA, thus reducing by a massive proportion, the number of people who can attack.

Of course, if you are publishing a SIP URI for people to contact you on, then Geo-IP may not be appropriate in your case.

You have done the vast majority of the work with Geo-IP and discussed it here - http://nerdvittles.com/?p=639 And you make the last point re-iptables and "What's next", but as we have discussed, you only need to do a look a lookup for the first packet tagged "new"; "established" and "related" traffic can be assumed to be trusted, so I do not expect there to be processor overloads.

To add it on to iptables would be interesting, and as a side effect, could more database entries be added made up of the blacklist above, this using the same methodology for GeoIP as well as Hacker lists.

In respect of gathering IP addresses for blacklisting, a number of us run PBX systems exposed to the public internet by necessity. Could trusted "Gurus" run a script that automatically add IP addresses to the black list from these "honeypot" PBX systems, thus cutting down on the admin, but keeping the list donations to trusted people?

Joe
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
In respect of gathering IP addresses for blacklisting, a number of us run PBX systems exposed to the public internet by necessity. Could trusted "Gurus" run a script that automatically add IP addresses to the black list from these "honeypot" PBX systems, thus cutting down on the admin, but keeping the list donations to trusted people?

Great idea! I've been working on something like that internally, but it would work for gurus with public sites as well. Your A2Billing and Star2Billing sites would be perfect candidates. :wink5: This needs to be coordinated with the VoIP Blacklist Project which will require folks to open an account there. The key to their reporting mechanism is accountability so that, if someone poisons the well, they can identify who did it. We wouldn't want an anonymous reporting system. Then you really do get a mess.
 

Members online

Forum statistics

Threads
25,778
Messages
167,504
Members
19,198
Latest member
serhii
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top