Bug 1379110

Summary: IP's show as banned, but aren't
Product: [Fedora] Fedora Reporter: Gerald Cox <gbcox>
Component: fail2banAssignee: Orion Poplawski <orion>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 24CC: athmanem, axel.thimm, jonathan.underwood, orion, vedran, vonsch
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: fail2ban-0.9.5-3.fc24 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-09 06:20:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gerald Cox 2016-09-25 03:31:42 UTC
Description of problem:
I'm not sure exactly how long this has been going on, but just noticed it within the last week.  In my log, I show that addresses are being banned but I keep getting retries - so they aren't.  I tested this myself, and sure enough fail2ban shows the traffic as being banned, but apparently is not.  

Below is the fail2ban log... notice where I get valid hits, followed by an "already banned" message.  

The "ipset list" command also shows the ip address listed.  In the sample below I just included a sample, but all the ban commands show up in the ipset list.

This was working, but now apparently, not so much.  If I manually enter a firewall-cmd to ban, it works... but if I enter the command manually through fail2ban-client it shows that it is banned (as per the log) and shows up in ipset... but doesn't block any traffic.

2016-09-24 20:17:14,393 fail2ban.actions        [17595]: NOTICE  [sshd] Ban 91.224.160.131
2016-09-24 20:17:14,530 fail2ban.actions        [17595]: NOTICE  [sshd] Ban 91.224.160.184
2016-09-24 20:17:14,675 fail2ban.actions        [17595]: NOTICE  [sshd] Ban 91.224.161.103
2016-09-24 20:17:14,811 fail2ban.actions        [17595]: NOTICE  [sshd] Ban 91.224.161.69
2016-09-24 20:17:14,947 fail2ban.actions        [17595]: NOTICE  [sshd] 116.31.116.16 already banned
2016-09-24 20:17:20,250 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:17:22,254 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:17:24,999 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:17:27,499 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:18:19,251 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:18:20,028 fail2ban.actions        [17595]: NOTICE  [sshd] 116.31.116.16 already banned
2016-09-24 20:18:20,500 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16
2016-09-24 20:18:22,753 fail2ban.filter         [17595]: INFO    [sshd] Found 116.31.116.16


ipset list
Name: fail2ban-sshd
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536 timeout 2592000
Size in memory: 12704
References: 0
Members:
14.175.62.142 timeout 4294433
14.175.101.234 timeout 4294433
59.148.215.44 timeout 4294445
169.44.131.51 timeout 4294436
91.224.160.131 timeout 4294447
118.233.14.92 timeout 4294431



Version-Release number of selected component (if applicable):
rpm -qa | grep fail2ban
fail2ban-firewalld-0.9.4-5.fc24.noarch
fail2ban-server-0.9.4-5.fc24.noarch
fail2ban-sendmail-0.9.4-5.fc24.noarch
fail2ban-0.9.4-5.fc24.noarch

Comment 1 Gerald Cox 2016-09-25 03:35:11 UTC
Here is the workaround command I've been using as the entries show up in the log, but it kind of defeats the purpose of fail2ban if I have to manually check the log and issue block commands:

firewall-cmd --zone=home --add-rich-rule='rule family="ipv4" source address="116.31.116.16" drop

Comment 2 Gerald Cox 2016-09-25 15:30:06 UTC
I believe I have found the problem...

firewall-cmd --direct --get-all-rules
ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable

This is the default rule being created... it isn't blocking traffic.

To get it to work I needed to change "REJECT --reject-with icmp-port-unreachable" to "DROP"

However, to do this, I had to manually change the firewallcmd-ipset.conf file 
<blocktype> variable to the fixed value DROP as follows:

actionstart = ipset create fail2ban-<name> hash:ip timeout <bantime>
              firewall-cmd --direct --add-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j <blocktype>

actionstop = firewall-cmd --direct --remove-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j <blocktype>
             ipset flush fail2ban-<name>
             ipset destroy fail2ban-<name>

to

actionstart = ipset create fail2ban-<name> hash:ip timeout <bantime>
              firewall-cmd --direct --add-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j DROP

actionstop = firewall-cmd --direct --remove-rule ipv4 filter <chain> 0 -p <protocol> -m multiport --dports <port> -m set --match-set fail2ban-<name> src -j DROP
             ipset flush fail2ban-<name>
             ipset destroy fail2ban-<name>


If I read the documentation correction, I should be able to simple create a file
iptables-blocktype.local which will override the <blocktype> defaults...

[DEFAULT]

blocktype = DROP

It doesn't work...

Comment 3 Gerald Cox 2016-09-25 15:57:28 UTC
I found where apparently the file should be:

iptables-common.local now instead of iptables-blocktype.local

I created it as follows:

[Init]

blocktype = DROP


So that works around the issue, but we probably shouldn't have a command that doesn't work as the default - unless I'm missing something...

Comment 4 Gerald Cox 2016-09-25 18:15:02 UTC
Well, I still have it working but the whole DROP vs REJECT reason for failure was a red herring - although the exercise allowed me to figure out how to change the default via iptables-common.local

What I found was at times when fail2ban is shutdown, for whatever reason the ipfilter rule:  firewall-cmd --direct --get-all-rules isn't deleted - and apparently this somehow causes the ban functionality to not work when fail2ban is restarted.  

I found that if I manually delete the rule when this happens before restarting fail2ban that circumvents the issue.  I did notice there is a new release of fail2ban while going through all of this... maybe that fixes the issue.  I haven't been able to figure out causes the issue.

Comment 5 Gerald Cox 2016-09-26 17:45:02 UTC
Sorry for all the comments, thought I'd add one last one for people who may encounter this issue so they don't have to drudge through all my blather above:

If you notice that things aren't being banned correctly, first shutdown fail2ban

systemctl stop fail2ban.service

Then, check to see if there are any "--match-set fail2ban-sshd" rules:

firewall-cmd --direct --get-all-rules

There shouldn't be... if you notice that there are, get rid of them by:
copying the rule:  starts with:  ipv4 filter ... 

issue the command:  
firewall-cmd --permanent --direct --remove-rule <paste line copied here>

Now you should be able to start fail2ban:  
systemctl start fail2ban.service

if you reissue the:
filewall-cmd --direct --get-all-rules

you'll notice that the "--match-set fail2ban-sshd" rule has been recreated.

Whenever you shutdown fail2ban, this rule should be deleted.  At times, and I haven't figured out why... it isn't - and it appears this causes an issue with things being banned.  At least that is what I have observed.

The other thing I learned (as mentioned above) is that if you want to change the default behavior of the ban from REJECT to DROP you need to create the

iptables-common.local

file as mentioned in comment #3.

You'll find references on the web and in some of the actual files in fail2ban  to "iptables-blocktype.local" - this doesn't work.  Not sure if it used to be called that and they changed their mind or it has some other function now.  In any event... it doesn't work to change the default behavior... for that, use iptables-common.local

Comment 6 Orion Poplawski 2016-10-03 22:07:24 UTC
FWIW - I can't reproduce this on rawhide with 0.9.5, so I'm hoping it will be fixed with the update.

Comment 7 Fedora Update System 2016-10-05 02:26:00 UTC
fail2ban-0.9.5-3.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7db83f651b

Comment 8 Gerald Cox 2016-10-07 20:22:46 UTC
Just left feedback on bodhi... appears to have fixed the issue.  Thanks!

Comment 9 Fedora Update System 2016-10-09 06:20:36 UTC
fail2ban-0.9.5-3.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.