FOOD FOR THOUGHT fail2ban stuck processing (old?) data, maxing CPU and repeatedly reloading iptables

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
I went out of town and came back to complaints that phone quality was bad. Turns out fail2ban-server was using 100%+ of the CPU. Further investigation shows that I'm getting SSH login attempts from Pakistan, etc., and tests from remote hosts indicates my ports are open. (Fortunately, I have very strong passwords, and there's no indication they've gotten in so far.)

I am looking through iptables and cannot figure out why I'm open to the public. I'm continuing to scramble and work through the issue now but figured I'd post here for some quick help in case anyone else is a pro at this and can immediately spot what I'm missing:

Code:
[email protected]:/var/log $ iptables -nvL
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 fail2ban-BadBots  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 80,443
    2   144 fail2ban-SSH  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22
    2   100 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    7   678 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x10/0x10
  261 35222 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state ESTABLISHED
    1    71 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp spt:53 dpts:9999:65535
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 0
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 3
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 4
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 11
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 12
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:113
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:1723
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:4569
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpts:10000:20000
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:32976
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:4445
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:123
    1    70 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:69
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:9022
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:5353
    0     0 ACCEPT     udp  --  *      *       64.2.142.215         0.0.0.0/0           multiport dports 5060,5061,5062,5063,5064,5065,5066,5067,5068,5069,4569
    0     0 ACCEPT     udp  --  *      *       64.2.142.216         0.0.0.0/0           multiport dports 5060,5061,5062,5063,5064,5065,5066,5067,5068,5069,4569
    0     0 ACCEPT     udp  --  *      *       64.2.142.9           0.0.0.0/0           multiport dports 5060,5061,5062,5063,5064,5065,5066,5067,5068,5069,4569

[contents omitted due to 10000-character limit on PIAF forums, but it's mostly the stock secure-iptables stuff for VoIP providers and a few of the IPs that I've manually added via ./add-fqdn or ./add-ip]

    0     0 ACCEPT     udp  --  *      *       184.72.223.118       0.0.0.0/0           udp dpts:5060:5069
    0     0 ACCEPT     tcp  --  *      *       184.72.223.118       0.0.0.0/0           tcp dpts:5060:5069
    0     0 ACCEPT     udp  --  *      *       162.243.57.52        0.0.0.0/0           udp dpts:5060:5069
    0     0 ACCEPT     tcp  --  *      *       162.243.57.52        0.0.0.0/0           tcp dpts:5060:5069
    0     0 ACCEPT     all  --  *      *       107.77.209.52        0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       107.77.217.65        0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       70.195.86.221        0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       206.174.28.123       0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       47.214.21.226        0.0.0.0/0
    0     0 ACCEPT     udp  --  *      *       85.17.87.148         0.0.0.0/0           udp dpts:5060:5069
    0     0 ACCEPT     tcp  --  *      *       85.17.87.148         0.0.0.0/0           tcp dpts:5060:5069
    0     0 ACCEPT     all  --  *      *       10.0.0.0/8           0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       127.0.0.0/8          0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       192.168.0.0/16       0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       67.202.98.137        0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       47.214.21.226        0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       67.202.98.137        0.0.0.0/0
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:1194
    0     0 ACCEPT     all  --  tun+   *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  tun+   *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  tun+   eth0    0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  eth0   tun+    0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT 88 packets, 5406 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain fail2ban-BadBots (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain fail2ban-SSH (1 references)
 pkts bytes target     prot opt in     out     source               destination
    2   144 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
Any help is appreciated. The tun+ interface is because I'm running OpenVPN.

The IP trying to log in is 116.31.116.15.
 

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
Slight update--I think IPtables is now working fine (maybe), though there is still occasional activity in /var/log/secure--but I think it's because fail2ban keeps unloading and loading iptables, which I'm guessing temporarily opens the machine to the public while it does that?

The drive had gotten full (small drive provided by RentPBX, wish I could pay for more storage space) and that may have gunked stuff up, and so now that I've freed up the drive, I think fail2ban is working its way through the logs and catching up on several days worth of logging activity, and that's what's causing fail2ban to use so much CPU and keep unloading and loading iptables.

I suppose it will just keep processing for awhile and eventually catch up, but I guess my question is morphing: is there any way to sort of "reset" fail2ban and have it start from scratch without processing old stuff? Maybe that'll fix this.

Edit: I was able to disable fail2ban (commenting out ipchecker in the crontab and service iptables stop and killing the process eventually stopped it) and everything returned to normal: no login attempts in /var/log/secure and processor usage 99% idle. As soon as I turned fail2ban back on, though, processor usage jumped with fail2ban-server and/or fail2ban-client and /var/log/fail2ban started churning through entries like this:

Code:
2017-07-14 15:48:30,239 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 15:48:30,537 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 15:48:30,793 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 15:48:37,153 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 15:48:37,443 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 15:48:37,562 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: dsl-216-221-32-6.mtl.aei.ca = ['216.221.32.6']
2017-07-14 15:48:37,584 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: dsl-216-221-32-6.mtl.aei.ca = ['216.221.32.6']
2017-07-14 15:48:43,527 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 15:48:43,796 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 15:48:43,821 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 15:48:44,084 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 15:48:44,943 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 15:48:45,254 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
I think those are all old, though, from perhaps while iptables was stuck offline when the disk was full, because none of those IPs have shown any recent activity in /var/log/secure.
 
Last edited:

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
OK, one more update:

If I comment out the root /root/ipchecker # > /dev/null 2>&1 line from crontab, after a short while, fail2ban's CPU usage dies off and all the activity in /var/log/fail2ban.log stops. So it seems that it does eventually stop processing whatever it's processing or catches up or something.

However, as soon as I re-enable that line in crontab and it launches again, the fail2ban activity starts back up.

Obviously, for Travelin' Man to work, I need to keep that line enabled, so I need to figure out what's going on and why ipchecker and/or fail2ban are doing what they're doing. Any help along that line would be greatly appreciated.

Here's a typical (longer) excerpt from the fail2ban log:

Code:
2017-07-14 18:57:29,021 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,045 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,067 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,098 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,176 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,203 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,277 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,305 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,397 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,746 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,771 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,801 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:29,826 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 18:57:29,850 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 18:57:33,763 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: host-95-152-40-110.dsl.sura.ru = ['95.152.40.110']
2017-07-14 18:57:34,528 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: host-95-152-40-110.dsl.sura.ru = ['95.152.40.110']
2017-07-14 18:57:36,568 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 59-120-6-100.hinet-ip.hinet.net = ['59.120.6.100']
2017-07-14 18:57:36,605 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 74-194-6-5.rsvlcmta01.com.dyn.suddenlink.net = ['74.194.6.5']
2017-07-14 18:57:36,928 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 60-248-105-227.hinet-ip.hinet.net = ['60.248.105.227']
2017-07-14 18:57:39,077 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: p2231-ipngn4001hodogaya.kanagawa.ocn.ne.jp = ['153.215.129.231']
2017-07-14 18:58:12,320 fail2ban.server : INFO   Stopping all jails
2017-07-14 18:58:13,412 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports http,https -j fail2ban-BadBots
iptables -F fail2ban-BadBots
iptables -X fail2ban-BadBots returned 100
2017-07-14 18:58:14,519 fail2ban.jail   : INFO   Jail 'apache-badbots' stopped
2017-07-14 18:58:15,521 fail2ban.actions.action: ERROR  iptables -D INPUT -p udp -m multiport --dports 5060,5061 -j fail2ban-asterisk-udp
iptables -F fail2ban-asterisk-udp
iptables -X fail2ban-asterisk-udp returned 100
2017-07-14 18:58:15,682 fail2ban.actions.action: ERROR  iptables -D INPUT -p all -j fail2ban-ASTERISK
iptables -F fail2ban-ASTERISK
iptables -X fail2ban-ASTERISK returned 100
2017-07-14 18:58:15,683 fail2ban.jail   : INFO   Jail 'asterisk' stopped
2017-07-14 18:58:16,522 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp --dport ssh -j fail2ban-SSH
iptables -F fail2ban-SSH
iptables -X fail2ban-SSH returned 100
2017-07-14 18:58:16,527 fail2ban.jail   : INFO   Jail 'ssh-iptables' stopped
2017-07-14 18:58:16,528 fail2ban.server : INFO   Exiting Fail2ban
2017-07-14 18:58:28,345 fail2ban.server : INFO   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.7
2017-07-14 18:58:28,345 fail2ban.jail   : INFO   Creating new jail 'ssh-iptables'
2017-07-14 18:58:28,347 fail2ban.jail   : INFO   Jail 'ssh-iptables' uses Gamin
2017-07-14 18:58:28,364 fail2ban.jail   : INFO   Initiated 'gamin' backend
2017-07-14 18:58:28,366 fail2ban.filter : INFO   Added logfile = /var/log/secure
2017-07-14 18:58:28,366 fail2ban.filter : INFO   Set maxRetry = 5
2017-07-14 18:58:28,368 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 18:58:28,369 fail2ban.actions: INFO   Set banTime = 600
2017-07-14 18:58:30,237 fail2ban.jail   : INFO   Creating new jail 'apache-badbots'
2017-07-14 18:58:30,237 fail2ban.jail   : INFO   Jail 'apache-badbots' uses Gamin
2017-07-14 18:58:30,238 fail2ban.jail   : INFO   Initiated 'gamin' backend
2017-07-14 18:58:30,239 fail2ban.filter : INFO   Added logfile = /var/log/httpd/access_log
2017-07-14 18:58:30,240 fail2ban.filter : INFO   Set maxRetry = 1
2017-07-14 18:58:30,241 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 18:58:30,242 fail2ban.actions: INFO   Set banTime = 172800
2017-07-14 18:58:31,117 fail2ban.jail   : INFO   Creating new jail 'asterisk'
2017-07-14 18:58:31,117 fail2ban.jail   : INFO   Jail 'asterisk' uses Gamin
2017-07-14 18:58:31,118 fail2ban.jail   : INFO   Initiated 'gamin' backend
2017-07-14 18:58:31,119 fail2ban.filter : INFO   Added logfile = /var/log/asterisk/full
2017-07-14 18:58:31,120 fail2ban.filter : INFO   Set maxRetry = 5
2017-07-14 18:58:31,121 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 18:58:31,121 fail2ban.actions: INFO   Set banTime = 1800
2017-07-14 18:58:32,019 fail2ban.jail   : INFO   Jail 'ssh-iptables' started
2017-07-14 18:58:32,314 fail2ban.jail   : INFO   Jail 'apache-badbots' started
2017-07-14 18:58:32,329 fail2ban.jail   : INFO   Jail 'asterisk' started
2017-07-14 18:58:32,615 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:33,022 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:33,322 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:33,621 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:33,646 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:34,022 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 18:58:35,313 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:35,602 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:35,624 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:35,913 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:36,201 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:36,224 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:36,512 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:36,801 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:36,824 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:37,233 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:37,522 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:37,544 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:38,012 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:38,601 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 18:58:38,624 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
668
Reaction score
248
It is possaible that fail2ban is not using the fastest "backend" if you are not, I suggest you install and use 'pyinotify'
 

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
It is possaible that fail2ban is not using the fastest "backend" if you are not, I suggest you install and use 'pyinotify'
It does look like it's using gamin. Seems odd that it would have been working fine for months and then quit working now, but at this point, I'll try anything. :)

I will take a stab at switching over to pyinotify over the weekend. Thanks.
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
668
Reaction score
248
If you install it, it will be used automatically on a restart
 

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
If you install it, it will be used automatically on a restart
Well, that was easy enough -- pyinotify is now installed.

However, the problem didn't get solved. fail2ban.log excerpt below:

Code:
2017-07-14 23:44:53,332 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 23:44:53,359 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 114-40-190-35.dynamic-ip.hinet.net = ['114.40.190.35']
2017-07-14 23:44:53,385 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 122-117-114-87.hinet-ip.hinet.net = ['122.117.114.87']
2017-07-14 23:44:56,691 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: host-95-152-40-110.dsl.sura.ru = ['95.152.40.110']
2017-07-14 23:44:56,881 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: host-95-152-40-110.dsl.sura.ru = ['95.152.40.110']
2017-07-14 23:44:58,264 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 59-120-6-100.hinet-ip.hinet.net = ['59.120.6.100']
2017-07-14 23:44:58,343 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 74-194-6-5.rsvlcmta01.com.dyn.suddenlink.net = ['74.194.6.5']
2017-07-14 23:44:58,433 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: 60-248-105-227.hinet-ip.hinet.net = ['60.248.105.227']
2017-07-14 23:45:00,953 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: p2231-ipngn4001hodogaya.kanagawa.ocn.ne.jp = ['153.215.129.231']
2017-07-14 23:46:39,537 fail2ban.server : INFO   Stopping all jails
2017-07-14 23:46:40,703 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp -m multiport --dports http,https -j fail2ban-BadBots
iptables -F fail2ban-BadBots
iptables -X fail2ban-BadBots returned 100
2017-07-14 23:46:40,966 fail2ban.jail   : INFO   Jail 'apache-badbots' stopped
2017-07-14 23:46:41,854 fail2ban.actions.action: ERROR  iptables -D INPUT -p udp -m multiport --dports 5060,5061 -j fail2ban-asterisk-udp
iptables -F fail2ban-asterisk-udp
iptables -X fail2ban-asterisk-udp returned 100
2017-07-14 23:46:41,864 fail2ban.actions.action: ERROR  iptables -D INPUT -p all -j fail2ban-ASTERISK
iptables -F fail2ban-ASTERISK
iptables -X fail2ban-ASTERISK returned 100
2017-07-14 23:46:41,865 fail2ban.jail   : INFO   Jail 'asterisk' stopped
2017-07-14 23:46:42,705 fail2ban.actions.action: ERROR  iptables -D INPUT -p tcp --dport ssh -j fail2ban-SSH
iptables -F fail2ban-SSH
iptables -X fail2ban-SSH returned 100
2017-07-14 23:46:42,709 fail2ban.jail   : INFO   Jail 'ssh-iptables' stopped
2017-07-14 23:46:42,710 fail2ban.server : INFO   Exiting Fail2ban
2017-07-14 23:46:52,116 fail2ban.server : INFO   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.7
2017-07-14 23:46:52,117 fail2ban.jail   : INFO   Creating new jail 'ssh-iptables'
2017-07-14 23:46:52,127 fail2ban.jail   : INFO   Jail 'ssh-iptables' uses pyinotify
2017-07-14 23:46:52,776 fail2ban.jail   : INFO   Initiated 'pyinotify' backend
2017-07-14 23:46:52,778 fail2ban.filter : INFO   Added logfile = /var/log/secure
2017-07-14 23:46:52,780 fail2ban.filter : INFO   Set maxRetry = 5
2017-07-14 23:46:52,781 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 23:46:52,782 fail2ban.actions: INFO   Set banTime = 600
2017-07-14 23:46:53,339 fail2ban.jail   : INFO   Creating new jail 'apache-badbots'
2017-07-14 23:46:53,339 fail2ban.jail   : INFO   Jail 'apache-badbots' uses pyinotify
2017-07-14 23:46:53,643 fail2ban.jail   : INFO   Initiated 'pyinotify' backend
2017-07-14 23:46:54,037 fail2ban.filter : INFO   Added logfile = /var/log/httpd/access_log
2017-07-14 23:46:54,038 fail2ban.filter : INFO   Set maxRetry = 1
2017-07-14 23:46:54,039 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 23:46:54,040 fail2ban.actions: INFO   Set banTime = 172800
2017-07-14 23:46:54,340 fail2ban.jail   : INFO   Creating new jail 'asterisk'
2017-07-14 23:46:54,340 fail2ban.jail   : INFO   Jail 'asterisk' uses pyinotify
2017-07-14 23:46:54,825 fail2ban.jail   : INFO   Initiated 'pyinotify' backend
2017-07-14 23:46:54,827 fail2ban.filter : INFO   Added logfile = /var/log/asterisk/full
2017-07-14 23:46:54,828 fail2ban.filter : INFO   Set maxRetry = 5
2017-07-14 23:46:54,829 fail2ban.filter : INFO   Set findtime = 600
2017-07-14 23:46:54,830 fail2ban.actions: INFO   Set banTime = 1800
2017-07-14 23:46:55,142 fail2ban.jail   : INFO   Jail 'ssh-iptables' started
2017-07-14 23:46:55,295 fail2ban.jail   : INFO   Jail 'apache-badbots' started
2017-07-14 23:46:55,505 fail2ban.jail   : INFO   Jail 'asterisk' started
2017-07-14 23:46:59,260 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:46:59,285 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:46:59,322 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:46:59,348 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:46:59,372 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:46:59,411 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: ec2-54-153-108-243.us-west-1.compute.amazonaws.com = ['54.153.108.243']
2017-07-14 23:47:00,333 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,436 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,459 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,482 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,504 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,528 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,551 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,636 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,660 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,736 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,760 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,783 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,867 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,892 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,916 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,941 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,964 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:00,987 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,010 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,034 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,058 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,082 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,276 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,300 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,385 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
2017-07-14 23:47:01,411 fail2ban.filter : WARNING Determined IP using DNS Reverse Lookup: mbl-65-136-109.dsl.net.pk = ['58.65.136.109']
(Notice in the section where fail2ban got restarted that it is indeed using pyinotify -- "Jail 'asterisk' uses pyinotify", etc.)

Any additional thoughts? I'm kind of at a loss for where to even begin, here...
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
15,247
Reaction score
2,670
Check the size of your log files. Perhaps one of them didn't get rotated and is enormous. That would bring fail2ban to its knees.
 

jackal

New Member
Joined
Sep 17, 2015
Messages
25
Reaction score
2
Check the size of your log files. Perhaps one of them didn't get rotated and is enormous. That would bring fail2ban to its knees.
Ward, you're a genius! Can't believe I didn't think of it--it was so obvious.

There was another component to the issue, though, before it was fully resolved. One of the xyz.iptables files had gotten stuck with a hundred lines of the same IP address repeating over and over. So even though I finally got fail2ban working fine (by manually rotating the 30MB logfile it was stuck on), every time ipchecker ran, it would restart fail2ban because it kept thinking the IP had changed. I've had that happen before and forget why it happened and what I did to fix it, but recreating the xyz.ipchecker file fixed it, so I'm good to go now--I think.

Thanks!
 
  • Like
Reactions: wardmundy

Members online

PIAF 5 - Powered by 3CX

Forum statistics

Threads
22,456
Messages
138,073
Members
14,620
Latest member
Brads#Bell