QUESTION IPBX13-13 Fail2Ban stops and starts a lot

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
Hi,
Just installed 13-13 and set bans to notify my email. Fail2ban seems to stop and start a lot.
I only wanted notifications of actual bans but the start/stop emails are very noisy.

Just wondering if this is normal?
Thanks.
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
So just to follow up here - this seems to happen pretty randomly (no pattern I can see) but when it happens it happens in spurts. So no activity for a while (many hours) and then suddenly a whole bunch of asterisk-jail, badbot-jail, ssh-jail start and stop emails. All typically within the same minute, it can repeat 2, 3 or even 4 times in a row and it always ends up with them started and running so...

Again, just wondering if this is normal (do any of you see this behavior?) or if this is a symptom of a problem?
I THINK f2b is doing it's job but I want to be sure.

For ref this is IPBX 13-13, whole enchilada (minus fax install) and using the stock TM3 firewall.
 

restamp

Member
Joined
Apr 24, 2016
Messages
97
Reaction score
53
I see lots of email from Fail2Ban about SSH, BadBots, and Asterisk stopping and starting.

If you want to simply quarantine all fail2ban mail into a separate account, create a "fail2ban" user, go to /etc/fail2ban, edit jail.conf, and change all instances of "dest=root" to "dest=fail2ban".

If you don't want to ever see the stop/start emails, find the active members in jail.conf -- in my case they were "sendmail-whois" and "sendmail-buffered" -- then go to the action.d directory and edit the corresponding conf file. Change the <dest> at the end of the "actionstart" and "actionstop" operatives to "nobody" and, if it isn't already set, add "nobody: /dev/null" to to your /etc/mail/aliases file (and run newaliases).
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
Hi Restamp,
Thanks for your answer.
My overarching question is really just - is this activity normal?
Is it normal to get these random stop/starts or is there maybe some issue on my server? (like maybe running out of memory or smth)


I see lots of email from Fail2Ban about SSH, BadBots, and Asterisk stopping and starting.

So you get the same behavior on your install? Is that going on for a long time? No ill effects or other issues?

If you want to simply quarantine all fail2ban mail into a separate account, create a "fail2ban" user, go to /etc/fail2ban, edit jail.conf, and change all instances of "dest=root" to "dest=fail2ban".

I'm receiving the emails to one of my email personal addresses (that iirc I provided to the setup script at install time.) I'm not collecting root@... though, now that you mention it, I guess root@ on the server is eventually going to get cluttered/filled by these messages.

If you don't want to ever see the stop/start emails, find the active members in jail.conf -- in my case they were "sendmail-whois" and "sendmail-buffered" -- then go to the action.d directory and edit the corresponding conf file. Change the <dest> at the end of the "actionstart" and "actionstop" operatives to "nobody" and, if it isn't already set, add "nobody: /dev/null" to to your /etc/mail/aliases file (and run newaliases).

I confess, I have to play with this to understand exactly what you said. It's a wee bit over my head. But if the point is simply to deep-six emails about stop/start, we go back to my first point / question. Is it normal for f2b to be stopping/restarting like this?
Because it's not so much the emails themselves that are the "problem" (they're a minor annoyance but I could write a filter for example to deal with them,) just trying to understand if this is a symptom of a real/bigger problem.

Thanks,
 

restamp

Member
Joined
Apr 24, 2016
Messages
97
Reaction score
53
I believe a lot of these stop/starts are generated from PAIF programs which are invoked out of Cron. Some of mine are semi-random. Others, like the pair of start/stops that regularly take place at 00:15 daily, are not. To be honest, I can't recall fail2ban ever actually banning anyone although it seems to me that that would be unlikely unless you have ports open to the world.

I'm not worrying about these, but let's see if Ward or some other power user cares to offer advice.
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
I believe a lot of these stop/starts are generated from PAIF programs which are invoked out of Cron. Some of mine are semi-random. Others, like the pair of start/stops that regularly take place at 00:15 daily, are not. To be honest, I can't recall fail2ban ever actually banning anyone although it seems to me that that would be unlikely unless you have ports open to the world.

I'm not worrying about these, but let's see if Ward or some other power user cares to offer advice.

Thanks for your insight. I had not yet noticed a patterned stop/start (like 00:15) but will keep an eye out for it now.
I have seen some bans. I'm running this on Vultr as suggested in the how to so mine is definitely open to the outside.

In the 12 or so hour period from when I'd acquired the VPS and followed just a first few steps till actually installing Freepbx/whole enchilada I was greeted on login with a message that there 1600+ failed login attempts. So certainly the front door gets banged on and why I want to ensure that this isn't a symptom of a problem. (yes, my root password and all passwords are secure ;) )

The IPs I have seen banned are predominantly from China. No real surprise there I think and at some point (when I have a bit more time) I will look into adding some geo-blocking to iptables to just flat out block China and maybe a few others. I have no relations or need to call to or from there so I think there's no reason not to.

Hopefully others will chime in and share. Thanks.
 

phonebuff

Guru
Joined
Feb 7, 2008
Messages
1,115
Reaction score
129
Bumping an older thread - Same basic issues.

I am seeing this as well on a new RentPBX install done this past week ..
-- RentPBX
-- CentOS 6.10
-- Asterisk 13.21.1.
-- Incredible 130.120.10 Lean & Enchilada
-- rpm -q fail2ban
fail2ban-0.8.7.1-1.el6.rf.noarch


One thing I did notice in the /var/log/fail2ban.log is some ERROR lines,.. Need to decode the action and I am not sure what the return codes mean..

Code:
2018-07-15 11:00:03,506 fail2ban.server : INFO   Stopping all jails
2018-07-15 11:00:03,634 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports http,https -j fail2ban-BadBots
iptables -F fail2ban-BadBots
iptables -X fail2ban-BadBots returned 100
2018-07-15 11:00:03,981 fail2ban.jail   : INFO   Jail 'apache-badbots' stopped
2018-07-15 11:00:03,996 fail2ban.actions.action: ERROR  iptables -D INPUT -p udp -m multiport --dports 5060,5061 -j fail2ban-asterisk-udp
iptables -F fail2ban-asterisk-udp
iptables -X fail2ban-asterisk-udp returned 100
2018-07-15 11:00:04,009 fail2ban.actions.action: ERROR  iptables -D INPUT -p all -j fail2ban-ASTERISK
iptables -F fail2ban-ASTERISK
iptables -X fail2ban-ASTERISK returned 100
2018-07-15 11:00:04,669 fail2ban.jail   : INFO   Jail 'asterisk' stopped
2018-07-15 11:00:04,994 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports 5060,5061 -j fail2ban-asterisk-tcp
iptables -F fail2ban-asterisk-tcp
iptables -X fail2ban-asterisk-tcp returned 100
2018-07-15 11:00:05,670 fail2ban.jail   : INFO   Jail 'asterisk-tcp' stopped
2018-07-15 11:00:05,992 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp --dport ssh -j fail2ban-SSH
iptables -F fail2ban-SSH
iptables -X fail2ban-SSH returned 100
2018-07-15 11:00:06,671 fail2ban.jail   : INFO   Jail 'ssh-iptables' stopped
2018-07-15 11:00:06,672 fail2ban.server : INFO   Exiting Fail2ban
2018-07-15 11:00:08,941 fail2ban.server : INFO   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.7
2018-07-15 11:00:08,942 fail2ban.jail   : INFO   Creating new jail 'ssh-iptables'
2018-07-15 11:00:08,943 fail2ban.jail   : INFO   Jail 'ssh-iptables' uses Gamin
2018-07-15 11:00:08,959 fail2ban.jail   : INFO   Initiated 'gamin' backend
2018-07-15 11:00:08,960 fail2ban.filter : INFO   Added logfile = /var/log/secure
2018-07-15 11:00:08,961 fail2ban.filter : INFO   Set maxRetry = 5
2018-07-15 11:00:08,962 fail2ban.filter : INFO   Set findtime = 600
2018-07-15 11:00:08,962 fail2ban.actions: INFO   Set banTime = 600
2018-07-15 11:00:09,086 fail2ban.jail   : INFO   Creating new jail 'apache-badbots'
2018-07-15 11:00:09,086 fail2ban.jail   : INFO   Jail 'apache-badbots' uses Gamin
2018-07-15 11:00:09,087 fail2ban.jail   : INFO   Initiated 'gamin' backend
2018-07-15 11:00:09,088 fail2ban.filter : INFO   Added logfile = /var/log/httpd/access_log
2018-07-15 11:00:09,089 fail2ban.filter : INFO   Set maxRetry = 1
2018-07-15 11:00:09,090 fail2ban.filter : INFO   Set findtime = 600
2018-07-15 11:00:09,090 fail2ban.actions: INFO   Set banTime = 172800
2018-07-15 11:00:09,120 fail2ban.jail   : INFO   Creating new jail 'asterisk'
2018-07-15 11:00:09,120 fail2ban.jail   : INFO   Jail 'asterisk' uses Gamin
2018-07-15 11:00:09,122 fail2ban.filter : INFO   Added logfile = /var/log/asterisk/full
2018-07-15 11:00:09,123 fail2ban.filter : INFO   Set maxRetry = 5
2018-07-15 11:00:09,124 fail2ban.filter : INFO   Set findtime = 600
2018-07-15 11:00:09,125 fail2ban.actions: INFO   Set banTime = 1800
2018-07-15 11:00:09,167 fail2ban.jail   : INFO   Creating new jail 'asterisk-tcp'
2018-07-15 11:00:09,168 fail2ban.jail   : INFO   Jail 'asterisk-tcp' uses Gamin
2018-07-15 11:00:09,168 fail2ban.jail   : INFO   Initiated 'gamin' backend
2018-07-15 11:00:09,169 fail2ban.filter : INFO   Added logfile = /var/log/asterisk/messages
2018-07-15 11:00:09,170 fail2ban.filter : INFO   Set maxRetry = 10
2018-07-15 11:00:09,171 fail2ban.filter : INFO   Set findtime = 600
2018-07-15 11:00:09,172 fail2ban.actions: INFO   Set banTime = 600
2018-07-15 11:00:09,204 fail2ban.jail   : INFO   Jail 'ssh-iptables' started
2018-07-15 11:00:09,208 fail2ban.jail   : INFO   Jail 'apache-badbots' started
2018-07-15 11:00:09,210 fail2ban.jail   : INFO   Jail 'asterisk' started
2018-07-15 11:00:09,211 fail2ban.jail   : INFO   Jail 'asterisk-tcp' started
2018-07-15 11:00:09,241 fail2ban.actions.action: ERROR  iptables -N fail2ban-asterisk-udp
iptables -A fail2ban-asterisk-udp -j RETURN
iptables -I INPUT -p udp -m multiport --dports 5060,5061 -j fail2ban-asterisk-udp returned 400
2018-07-15 11:00:11,062 fail2ban.server : INFO   Stopping all jails
2018-07-15 11:00:11,730 fail2ban.jail   : INFO   Jail 'apache-badbots' stopped
2018-07-15 11:00:12,683 fail2ban.jail   : INFO   Jail 'asterisk' stopped
2018-07-15 11:00:13,243 fail2ban.jail   : INFO   Jail 'asterisk-tcp' stopped
2018-07-15 11:00:14,212 fail2ban.jail   : INFO   Jail 'ssh-iptables' stopped
2018-07-15 11:00:14,213 fail2ban.server : INFO   Exiting Fail2ban

Also, a little earlier I see this in the same log. I am not sure how the 140 address is getting into the box right now as my iptables rules should be blocking it but that's a separate headache.

But it's the "fail2ban.actions.action" errors that I think are a bigger issue in my mind and possibly related to the BadBots emails cycles..

Code:
2018-07-15 00:15:15,944 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: sheujp.ee.ncu.edu.tw = ['140.115.70.160']
2018-07-15 00:15:16,725 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: sheujp.ee.ncu.edu.tw = ['140.115.70.160']
2018-07-15 03:28:02,721 fail2ban.filter : INFO   Log rotation detected for /var/log/secure
2018-07-15 03:28:02,996 fail2ban.filter : INFO   Log rotation detected for /var/log/asterisk/full
2018-07-15 05:00:03,228 fail2ban.server : INFO   Stopping all jails
2018-07-15 05:00:03,680 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports http,https -j fail2ban-BadBots
iptables -F fail2ban-BadBots
iptables -X fail2ban-BadBots returned 100
2018-07-15 05:00:04,067 fail2ban.jail   : INFO   Jail 'apache-badbots' stopped
2018-07-15 05:00:04,683 fail2ban.actions.action: ERROR  iptables -D INPUT -p udp -m multiport --dports 5060,5061 -j fail2ban-asterisk-udp
iptables -F fail2ban-asterisk-udp
iptables -X fail2ban-asterisk-udp returned 100
2018-07-15 05:00:04,694 fail2ban.actions.action: ERROR  iptables -D INPUT -p all -j fail2ban-ASTERISK
iptables -F fail2ban-ASTERISK
iptables -X fail2ban-ASTERISK returned 100
2018-07-15 05:00:04,695 fail2ban.jail   : INFO   Jail 'asterisk' stopped
2018-07-15 05:00:05,685 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports 5060,5061 -j fail2ban-asterisk-tcp
iptables -F fail2ban-asterisk-tcp
iptables -X fail2ban-asterisk-tcp returned 100
2018-07-15 05:00:05,931 fail2ban.jail   : INFO   Jail 'asterisk-tcp' stopped
2018-07-15 05:00:06,684 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp --dport ssh -j fail2ban-SSH
iptables -F fail2ban-SSH
iptables -X fail2ban-SSH returned 100
2018-07-15 05:00:06,939 fail2ban.jail   : INFO   Jail 'ssh-iptables' stopped
2018-07-15 05:00:06,940 fail2ban.server : INFO   Exiting Fail2ban
2018-07-15 05:00:09,247 fail2ban.server : INFO   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.7

Lastly, my Asterisk cron looks like this.. Something here might be related to the starts & stop, but while I expected to see the " 15 0 * * * root /usr/local/sbin/iptables-restart >/dev/null 2>&1 " and be able to possibly map that, I do not. Could it be related to fwconsole running every 15 minutes.

[cpde]
*/15 * * * * /usr/sbin/fwconsole userman --syncall -q
* * * * * [ -x /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php > /dev/null 2>&1
0 * * * * /var/lib/asterisk/bin/freepbx-cron-scheduler.php
50 * * * * /usr/local/sbin/fwconsole util cleanplaybackcache -q
* * * * * [ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php
[/code]
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
I can't help with the errors but in my case the 'random' stop/starts came from my mobile phone - every time ddns registered a change in my IP I got emails corresponding to all 3 (?) jails stopping and starting.
hth.
 

phonebuff

Guru
Joined
Feb 7, 2008
Messages
1,115
Reaction score
129
You may be on to something -

ipchecker and iptables-restart could be the root reason, with detected changes causing the messaging.

[code[
15 0 * * * root /usr/local/sbin/iptables-restart >/dev/null 2>&1
*/10 5-22 * * * root /root/ipchecker > /dev/null 2>&1
[/code]

More research on my part needed..
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
You may be on to something -

ipchecker and iptables-restart could be the root reason, with detected changes causing the messaging.

[code[
15 0 * * * root /usr/local/sbin/iptables-restart >/dev/null 2>&1
*/10 5-22 * * * root /root/ipchecker > /dev/null 2>&1
[/code]

More research on my part needed..

Hope you get to the root. Debugging this is over my head but I'd be grateful to be rid of those messages and happy to help by testing, etc. where I could.

Best of Luck.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Whenever you restart IPtables, you must also restart Fail2Ban. iptables-restart does this automatically. For security, Incredible PBX servers do an iptables-restart every night which means you also will get Fail2Ban log entries as it is subsequently restarted.
 

phonebuff

Guru
Joined
Feb 7, 2008
Messages
1,115
Reaction score
129
@ward --

Back to learning and trying to understand the logic behind some things.,

IPChecker does an IPTables Restart but does not redirect the out to dev null

Current Code looks like this --

Code:
if [ "$restartflag" == "Y" ]; then
echo "iptables-restart"
echo "iptables-restart" >> /var/log/ipchecker.log
/usr/local/sbin/iptables-restart
fi

So in a Bash shell does this command get a separate stdout stderr or inherit from the calling shell ?
Wondering if I need to change this to suppress all the emails regarding restarts.

/usr/local/sbin/iptables-restart > /dev/null 2>&1

Also the current /usr/local/sbin/iptables-restart seems to have a duplicate statement --
Where line 4 does an init.d/fail2ban restart and line 13 does a service fail2ban restart.
Lastly, are these commands really in the right order in that the call to iptables-custom is after the three restarts ?

Code:
#!/bin/bash
service iptables restart
service ip6tables restart
/etc/init.d/fail2ban restart
/usr/local/sbin/iptables-custom
/usr/sbin/fwconsole chown
if [ -d "/var/www/html/avantfax" ]; then
 chmod -R 777 /var/www/html/avantfax
 chown -R asterisk:asterisk /var/www/html/avantfax
 chmod -R 0770 /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
 chown -R asterisk:uucp /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
fi
service fail2ban restart

As always Thank you for your help and the time you give to the community...
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
@ward --

Back to learning and trying to understand the logic behind some things.,

IPChecker does an IPTables Restart but does not redirect the out to dev null

Current Code looks like this --

Code:
if [ "$restartflag" == "Y" ]; then
echo "iptables-restart"
echo "iptables-restart" >> /var/log/ipchecker.log
/usr/local/sbin/iptables-restart
fi

So in a Bash shell does this command get a separate stdout stderr or inherit from the calling shell ?
Wondering if I need to change this to suppress all the emails regarding restarts.

/usr/local/sbin/iptables-restart > /dev/null 2>&1

Also the current /usr/local/sbin/iptables-restart seems to have a duplicate statement --
Where line 4 does an init.d/fail2ban restart and line 13 does a service fail2ban restart.
Lastly, are these commands really in the right order in that the call to iptables-custom is after the three restarts ?

Code:
#!/bin/bash
service iptables restart
service ip6tables restart
/etc/init.d/fail2ban restart
/usr/local/sbin/iptables-custom
/usr/sbin/fwconsole chown
if [ -d "/var/www/html/avantfax" ]; then
 chmod -R 777 /var/www/html/avantfax
 chown -R asterisk:asterisk /var/www/html/avantfax
 chmod -R 0770 /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
 chown -R asterisk:uucp /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
fi
service fail2ban restart

As always Thank you for your help and the time you give to the community...

Wish I could help with code more but I think you're asking the right questions.
If I'm following along correctly then imho the restarts should probably not be sunk to /dev/null as we would want them logged, we just don't want them triggering email notifications. Agree?
 

HermanMiller

Member
Joined
Apr 4, 2018
Messages
32
Reaction score
3
Whenever you restart IPtables, you must also restart Fail2Ban. iptables-restart does this automatically. For security, Incredible PBX servers do an iptables-restart every night which means you also will get Fail2Ban log entries as it is subsequently restarted.
Ward,
I think the issue here is the constant flow of notification emails this generates, not so much the log entries themselves. (I'd expect the log should have these entries to track what goes on)

Also, part 2 of this is not just log entries when the regular restarts but the notifications that are triggered everytime a ddns user's IP changes. And not just 1 email but 6. With every IP change.
As above, entries to the log files are fine, even desired. Just not the constant noise of emails. Emails should only come when something genuinely bad happens. Like blacklisting an actual attacker.
That way a barrage of emails means something actually is happening not just that you have users :auto: around passing in and out of public wifi hotspots.. :eek:
 

phonebuff

Guru
Joined
Feb 7, 2008
Messages
1,115
Reaction score
129
So I am digging around some more and the quantity of the emails is directly related to changes in the IPTables route when a dynamic address changes..

/root/ipchecker runs every 10 mins between 0500 & 2200 and every time a change is detected it will stop and start the world generating a log entry to /var/log/ipchecker.log as well as the six emails * 2.

It looks to me that this script calls /usr/local/sbin/iptables-restart to do the actual restart when it has detected a change but simply adding the stdout / stderr redirect here does not suppress the extra emails all day long.

For me the right answer is get the emails when iptables-restart runs at 0015 each day, but not all day long.

This has to do with the way the bash scripts are nested but I am not sure of the correct fix.
============

Also, still have not resolved the difference between the service and init.d restarts noted above in #12.
 

Trimline2

Guru
Joined
May 23, 2013
Messages
524
Reaction score
96
I'm running 13-13 as well and see the same thing; Asterisk, Fail2ban and SSH all stop and restart. I did notice a strange message that I haven't researched yet - run-parts /etc/cron.hourly appears after all the mail is received. I'm on a fixed IP address and am not running port knocker, etc..
 

Attachments

  • fail2ban.jpg
    fail2ban.jpg
    202.1 KB · Views: 4

domiflichi

Member
Joined
Jan 18, 2012
Messages
39
Reaction score
4
Glad I came across this thread. I was wondering why I'm getting a bunch of Fail2Ban emails every morning at 00:15. I feel better now knowing that this is intentional.
 
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top