SOLVED after reboot firewall is wide open

[wardmundy]

That version, while it refers to the right paths on CentOS 7 systemd for the iptables rules files, still does not fix the bug from post #29 above, where the `iptables` rule which supposed to be generated by `fail2ban`, `f2b-astrerisk-tcp`, is missing from `iptables -nL`, due to an error in `/etc/fail2ban/jail.conf` section `[asterisk]`.
Also see https://github.com/EspaceNetworks/IncrediblePBXCentOS7/issues/7

Different bug. I'll have a look. Thanks.

 

[wardmundy]

This should get asterisk-tcp working with Fail2Ban. Make this into a script and run it:

TEST=`sed '578q;d' /etc/fail2ban/jail.conf`
if [ "$TEST" = "" ]; then
 echo Applying Fail2Ban asterisk-tcp patch
 THELINE=`grep -n asterisk-tcp] /etc/fail2ban/jail.conf | cut -f 1 -d ":"`
 THELINE=$((THELINE+1))
 sed -i "$THELINE s:^:enabled = yes:" /etc/fail2ban/jail.conf
 THELINE=$((THELINE+1))d
 sed -i "$THELINE" /etc/fail2ban/jail.conf
 service fail2ban restart
 iptables -nL
else
 echo No asterisk-tcp patch required for Fail2Ban
fi

 

[chris_c_]

@wardmundy The output of your above script on an unpatched /etc/fail2ban/jail.conf is a false negative.

No asterisk-tcp patch required for Fail2Ban

.
The problem is there's 3 existing sections in jail.conf, the new enabled [asterisk] and the old disabled [asterisk-tcp] and [asterisk-udp]
The enabled [asterisk] needs to set enabled = false.
The disabled [asterisk-tcp] and [asterisk-udp] need to be
enabled = true
maxretry = 5
bantime = 1800
sendmail-whois = dest= and sender= need to be the admin email or root as default, and a valid sender email for the pbx if the dest= is not root.
sendmail-whois = dest = needs to get the valid admin email addresses entered during the install script, or root, for all asterisk related entries in jail.conf including ssh and http/https.
logfile = may have to change from /var/log/asterisk/messages to /var/log/asterisk/full

 

[wardmundy]

You can obviously tweak it in any way desired? If you're getting the false report, it's because you already have modified jail.conf. We have to build systems assuming folks haven't improved things.

What happens if you perform a base install with CentOS 7, followed by an Incredible PBX install, and then run the script?

 

[wardmundy]

@chris_c_: Let me ask you about your build methodology if you don't mind.

I just downloaded a fresh copy of CentOS 7-64-minimal and loaded it in VirtualBox. After booting, I ran the following commands:

setenforce 0
yum -y upgrade --skip-broken
yum -y install net-tools nano wget tar
reboot

After reboot, I logged back in as root and ran the Incredible PBX installer choosing Google Voice OAuth support:

cd /root
wget http://incrediblepbx.com/incrediblepbx13-12.2-centos.tar.gz
tar zxvf incrediblepbx*
./IncrediblePBX*
reboot

When it finished, I logged back in and let the Automatic Update Utility load the latest updates. Then I reviewed status and ran iptables -nL. IPtables worked as expected and Fail2Ban shows these rules including asterisk-tcp:

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain f2b-ASTERISK (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain f2b-BadBots (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain f2b-SSH (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain f2b-asterisk-tcp (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain f2b-asterisk-udp (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

No Fail2Ban patch was necessary which may explain the message you received assuming you've made no changes to jail.conf.

So tell me what you're doing differently??

This is starting to look more and more like a bug in your CentOS 7/32-bit platform that neither we nor CentOS supports.

 

[wardmundy]

Just tried a 32-bit, CentOS 7 install on VMware ESXi using the same method as above. Everything worked except for a minor reporting error in status that showed Fail2Ban down when, in fact, is was UP including asterisk-tcp jail; however, asterisk-udp jail was not activated:

2016-09-04 17:07:33,104 fail2ban.jail   : INFO   Jail 'asterisk-tcp' uses Gamin
2016-09-04 17:07:33,104 fail2ban.jail   : INFO   Initiated 'gamin' backend
2016-09-04 17:07:33,178 fail2ban.jail   : INFO   Jail 'asterisk-tcp' started

This mod to /usr/local/sbin/status fixed status:

test=`service fail2ban status | grep running | cut -f 5 -d " "`
if [[ "$test" == "running..." ]]; then
fail2ban="$UP"
else
fail2ban="$DN"
fi

The following mod in /etc/fail2ban/jail.conf fixed the asterisk-udp jail:

[asterisk-udp]

enabled  = true

So... what are you doing differently??

 

[chris_c_]

You can obviously tweak it in any way desired? If you're getting the false report, it's because you already have modified jail.conf. We have to build systems assuming folks haven't improved things.

This fail2ban bug occurred with the stock unmodified /etc/fail2ban/jail.conf
EDIT: In fact the install script overwrites jail.conf with your customized copy from http://incrediblepbx.com/jail-R.tar.gz see around line 874 of the install script.
By unmodified, this means, it isn't further modified.

 

[wardmundy]

As I've documented later in thread, the problem appears to be the Fail2Ban implementation on the CentOS 7/32-bit platform, not jail.conf. It works fine as is on 64-bit versions of CentOS 6 and 7. You're using a platform we don't support. Code bugs at our end we're more than willing to fix, but we're not planning to build out revisions to our stock code to accommodate a platform that is not supported by RedHat or CentOS. I've spent the better part of yesterday documenting how to fix it for the time being, but the rest is up to you. As new versions of Fail2Ban are released, you'll need to carefully monitor whether additional fixes become necessary.

I appreciate your help finding the iptables-restart glitches on the CentOS 7 platform. These were not specific to the 32-bit implementation, and we are addressing have fixed them for the benefit of everyone deploying Incredible PBX on the CentOS 7 platform.

 

[chris_c_]

Even when you manually set the legacy asterisk-tcp section to enabled in jail.conf, the notification parameters are wrong, all the email addresses for notifications are invalid so nobody will ever receive notification of hack attempts. The install script must substitute the admin email address provided during the install, for all the dummy default email addresses that come with the stock jail.conf.

~ $ service fail2ban restart
Redirecting to /bin/systemctl restart  fail2ban.service
WARNING: Always run Incredible PBX behind a secure hardware-based firewall.
~ $ service fail2ban status
Redirecting to /bin/systemctl status  fail2ban.service
● fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-09-05 12:45:22 EDT; 9s ago
     Docs: man:fail2ban(1)
  Process: 23150 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=0/SUCCESS)
  Process: 22760 ExecReload=/usr/bin/fail2ban-client reload (code=exited, status=255)
  Process: 23206 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
Main PID: 23209 (fail2ban-server)
   CGroup: /system.slice/fail2ban.service
           ├─23209 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
           └─23211 /usr/libexec/gam_server

Sep 05 12:45:21 (hostname) fail2ban-client[23206]: ERROR  NOK: ('Action iptables-multiport already exists',)
Sep 05 12:45:22 (hostname) systemd[1]: Started Fail2Ban Service.
Sep 05 12:45:22 (hostname) sendmail[23225]: u85GjMHn023225: [email protected], size=228, class=0, nrcpts=1, msgid=<20160905...calhost
Sep 05 12:45:22 (hostname) sendmail[23225]: u85GjMHn023225: [email protected], [email protected] (0/0), delay=00:00:00,...livery)
Sep 05 12:45:22 (hostname) sendmail[23247]: u85GjMqP023247: [email protected], size=238, class=0, nrcpts=1, msgid=<20160905...calhost
Sep 05 12:45:22 (hostname) sendmail[23247]: u85GjMqP023247: [email protected], [email protected] (0/0), delay=00:00:00,...livery)
Sep 05 12:45:22 (hostname) sendmail[23257]: u85GjM2W023257: from=fail2ban, size=240, class=0, nrcpts=1, msgid=<201609051645.u85GjM2...calhost
Sep 05 12:45:22 (hostname) sendmail[23257]: u85GjM2W023257: [email protected], delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=...livery)
Sep 05 12:45:22 (hostname) sendmail[23265]: u85GjMQX023265: [email protected], size=230, class=0, nrcpts=1, msgid=<20160...calhost
Sep 05 12:45:22 (hostname) sendmail[23265]: u85GjMQX023265: to=root, [email protected] (0/0), delay=00:00:00, xdelay=...livery)
Hint: Some lines were ellipsized, use -l to show in full.
WARNING: Always run Incredible PBX behind a secure hardware-based firewall.
~ $

 

[chris_c_]

@wardmundy
All the Fail2ban rpm packages, which the script installs, come from EPEL, and are "noarch" packages ie same code, works exactly the same on 32 and 64 bit OS. Have a look:
https://dl.fedoraproject.org/pub/epel/7/x86_64/f/
This should make sense because fail2ban is python interpreted not compiled binary.
Fail2ban still failing to start the asterisk tcp jail with error "Action iptables-multiport already exists".
This bug is possibly a regression bug in fail2ban itself, already reported as an issue on github.

The serious bit: The install script really must apply the admin email address to the jail.conf because it could be tragic for the admin to fail to receive those notifications of robo-hack servers carrying out attacks on the pbx system, simply to to neglecting to apply that email address with a sed statement.

 

[wardmundy]

We don't currently obtain email addresses from those using our software. It's a good idea, and we'll put it on the Wish List. However, we don't configure notification email addresses for any software included in Incredible PBX. That currently is the administrator's responsibility.

Keep in mind that Fail2Ban is a second layer of protection on Incredible PBX servers. The IPtables Firewall WhiteList will protect users against all brute force attacks, and Fail2Ban as configured would secondarily block the attacks unless they originated from a really powerful platform against an underperforming platform. As for Fail2Ban being the same code regardless of architecture, all I can tell you is that the results of running service fail2ban status give different results on the two platforms, and that's what we use to determine whether Fail2Ban is running. In addition, the iptables-multiport error does not occur on the 64-bit platform. So... there appear to be differences.

 

[chris_c_]

We don't currently obtain email addresses from those using our software. It's a good idea, and we'll put it on the Wish List. However, we don't configure notification email addresses for any software included in Incredible PBX. That currently is the administrator's responsibility.

Keep in mind that Fail2Ban is a second layer of protection on Incredible PBX servers. The IPtables Firewall WhiteList will protect users against all brute force attacks, and Fail2Ban as configured would secondarily block the attacks unless they originated from a really powerful platform against an underperforming platform. As for Fail2Ban being the same code regardless of architecture, all I can tell you is that the results of running service fail2ban status give different results on the two platforms, and that's what we use to determine whether Fail2Ban is running. In addition, the iptables-multiport error does not occur on the 64-bit platform. So... there appear to be differences.

OK it's the incrediblefax installer which asks for an email address for forwarding the incoming faxes.

Incrediblepbx as well, really ought to ask for an admin email for security notifications. If none provided, mail destination should default to "root" which sends mail to the local server, way better than sending mail to the poor owners of example.com.
Hack notifications can be extremely serious, so setting the forwarding emails really shouldn't be assigned to a human as yet another easily skipped chore to remember to (forget to) replace the [email protected] in jail.conf. A simple sed statement or two solves that and eliminates the very real risk of the admin skipping that, and from then on, never hearing about all the hack attempts, and falsely concluding, no news is good news! AND most importantly that deafening silence when a bug causes a critical service such as iptables to be down therefore zero notifications is a HUGE sign that the intrusion alert systems are malfunctioning whether due to bug or any reason, .and you must look at them immediately or else run an idiotic risk losing your shirt to toll fraud.

 

[chris_c_]

Message 15:
From MAILER-DAEMON@(incredible pbx) Wed Sep 7 01:30:17 2016
Return-Path: <MAILER-DAEMON@(incredible pbx)>
Date: Wed, 7 Sep 2016 01:30:17 -0400
From: Mail Delivery Subsystem <MAILER-DAEMON@(incredible pbx)>
To: postmaster@(incredible pbx)
Content-Type: multipart/report; report-type=delivery-status;
boundary="u875THor001814.1473226217/(incredible pbx)"
Subject: Postmaster notify: see transcript for details
Auto-Submitted: auto-generated (postmaster-notification)
Status: RO

Part 1:

The original message was received at Fri, 2 Sep 2016 00:47:23 -0400
from localhost
with id u824kNMM016341

----- The following addresses had permanent fatal errors -----
<[email protected]>

----- Transcript of session follows -----
<[email protected]>... Deferred: Connection timed out with example.com.
Message could not be delivered for 5 days
Message will be deleted from queue

Part 2:
Content-Type: message/delivery-status


Part 3:
Content-Type: message/rfc822

From MAILER-DAEMON Fri Sep 2 00:47:23 2016
Return-Path: <MAILER-DAEMON>
Date: Fri, 2 Sep 2016 00:47:23 -0400
From: Mail Delivery Subsystem <MAILER-DAEMON>
To: <[email protected]>
Content-Type: multipart/report; report-type=delivery-status;
boundary="u824kNMM016341.1472791643/(incredible pbx)"
Subject: Warning: could not send message for past 4 hours
Auto-Submitted: auto-generated (warning-timeout)


Part 3.1:

**********************************************
** THIS IS A WARNING MESSAGE ONLY **
** YOU DO NOT NEED TO RESEND YOUR MESSAGE **
**********************************************

The original message was received at Thu, 1 Sep 2016 20:25:28 -0400
from pbx.local [127.0.0.1]

----- Transcript of session follows -----
<[email protected]>... Deferred: Connection timed out with example.com.
Warning: message still undelivered after 4 hours
Will keep trying until message is 5 days old

Part 3.2:
Content-Type: message/delivery-status


Part 3.3:
Content-Type: message/rfc822

From [email protected] Thu Sep 1 20:25:28 2016
Return-Path: <[email protected]>
Subject: [Fail2Ban] SSH: stopped on (incredible pbx)
Date: Thu, 01 Sep 2016 20:25:28 -0400
From: Fail2Ban <[email protected]>
To: [email protected]

Hi,

The jail SSH has been stopped.

Regards,

Fail2Ban
&

 

[chris_c_]

Just tried a 32-bit, CentOS 7 install on VMware ESXi using the same method as above. Everything worked except for a minor reporting error in status that showed Fail2Ban down when, in fact, is was UP including asterisk-tcp jail; however, asterisk-udp jail was not activated:

2016-09-04 17:07:33,104 fail2ban.jail   : INFO   Jail 'asterisk-tcp' uses Gamin
2016-09-04 17:07:33,104 fail2ban.jail   : INFO   Initiated 'gamin' backend
2016-09-04 17:07:33,178 fail2ban.jail   : INFO   Jail 'asterisk-tcp' started

This mod to /usr/local/sbin/status fixed status:

test=`service fail2ban status | grep running | cut -f 5 -d " "`
if [[ "$test" == "running..." ]]; then
fail2ban="$UP"
else
fail2ban="$DN"
fi

The following mod in /etc/fail2ban/jail.conf fixed the asterisk-udp jail:

[asterisk-udp]

enabled  = true

So... what are you doing differently??

Script used to install on 32 bit CentOS 7:
https://github.com/EspaceNetworks/IncrediblePBXCentOS7/blob/master/IncrediblePBX13-12R-OAUTH.sh