Description of problem: fail2ban is installed and running, but is not banning any IP addresses that are attempting to log in via ssh and fail to do so. Version-Release number of selected component (if applicable): 0.3.git1f1a561.fc20 How reproducible: Install fail2ban via yum, keep the standard fail2ban.conf and jail.conf and make sure it is running via: $ ps ax | grep fail2ban 1220 ? S 0:00 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x $ sudo systemctl status fail2ban fail2ban.service - Fail2ban Service Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled) Active: active (running) since Thu 2013-12-26 13:33:45 CST; 4h 36min ago Process: 1185 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS) Main PID: 1220 (fail2ban-server) CGroup: /system.slice/fail2ban.service └─1220 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x Bruceforce ssh into the system exceeding maxretry in jail.conf, and fail2ban will NOT ban the offending host IP, even though the failed attempts can be seen via: $ sudo less /var/log/secure | grep Failed Steps to Reproduce: 1. 2. 3. Actual results: fail2ban is not banning host IP addresses exceeding maxretry ssh attempts in jail.conf Expected results: fail2ban should ban IP addresses that exceed maxretry ssh attempts in jail.conf Additional info:
Cau you please attach lines from /var/log/secure which indicates login failures? (Or simply attach your /var/log/secure if there are no private information there) Also please attach messages from fail2ban daemon, from /var/log/messages (or attach your entire /var/log/messages). Thank you in advance.
I can only give excerpts from /var/log/secure and /var/log/messages because of private information now, sorry if this does not help big, I plan to install F20 on another machine these days and then I can provide the whole files. The rejected fail2ban messages from /var/log/messages are my own doing trying to make sure fail2ban is running. $ sudo less /var/log/secure | grep Failed Dec 27 15:32:10 bhost sshd[29668]: Failed password for invalid user root from 114.80.246.146 port 7569 ssh2 Dec 27 15:32:14 bhost sshd[29671]: Failed password for invalid user root from 114.80.246.146 port 8250 ssh2 Dec 27 15:32:17 bhost sshd[29674]: Failed password for invalid user root from 114.80.246.146 port 8986 ssh2 Dec 27 15:32:22 bhost sshd[29677]: Failed password for invalid user root from 114.80.246.146 port 9751 ssh2 Dec 27 15:32:26 bhost sshd[29680]: Failed password for invalid user root from 114.80.246.146 port 10645 ssh2 Dec 27 15:32:30 bhost sshd[29683]: Failed password for invalid user root from 114.80.246.146 port 11312 ssh2 Dec 27 15:32:33 bhost sshd[29691]: Failed password for invalid user root from 114.80.246.146 port 11934 ssh2 Dec 27 15:32:37 bhost sshd[29694]: Failed password for invalid user root from 114.80.246.146 port 12466 ssh2 Dec 27 15:32:41 bhost sshd[29697]: Failed password for invalid user root from 114.80.246.146 port 13088 ssh2 Dec 27 15:32:45 bhost sshd[29700]: Failed password for invalid user root from 114.80.246.146 port 13751 ssh2 Dec 27 15:32:49 bhost sshd[29703]: Failed password for invalid user root from 114.80.246.146 port 14448 ssh2 $ sudo less /var/log/secure | grep Invalid Dec 27 09:42:46 bhost sshd[23421]: Invalid user postgres from 212.146.83.246 Dec 27 09:42:50 bhost sshd[23423]: Invalid user postgres from 212.146.83.246 Dec 27 09:42:53 bhost sshd[23425]: Invalid user postgres from 212.146.83.246 Dec 27 09:42:56 bhost sshd[23427]: Invalid user postgres from 212.146.83.246 Dec 27 09:42:59 bhost sshd[23429]: Invalid user postgres from 212.146.83.246 Dec 27 09:43:04 bhost sshd[23431]: Invalid user postgres from 212.146.83.246 Dec 27 09:43:08 bhost sshd[23433]: Invalid user postgres from 212.146.83.246 $ sudo less /var/log/messages | grep fail2ban Dec 26 13:24:42 bhost fail2ban-client: Shutdown successful Dec 26 13:29:27 bhost fail2ban-client: 2013-12-26 13:29:27,257 fail2ban.server.server: INFO Starting Fail2ban v0.9.0a1 Dec 26 13:29:27 bhost fail2ban-client: 2013-12-26 13:29:27,258 fail2ban.server.server: INFO Starting in daemon mode Dec 26 13:32:42 bhost fail2ban-client: Shutdown successful Dec 26 13:33:46 bhost fail2ban-client: 2013-12-26 13:33:45,287 fail2ban.server.server: INFO Starting Fail2ban v0.9.0a1 Dec 26 13:33:46 bhost fail2ban-client: 2013-12-26 13:33:45,287 fail2ban.server.server: INFO Starting in daemon mode Dec 26 17:39:03 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.188" (uid=1000 pid=13789 comm="systemctl enable fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="EnableUnitFiles" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ") Dec 26 17:39:03 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.188" (uid=1000 pid=13789 comm="systemctl enable fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="EnableUnitFiles" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ") Dec 26 17:39:14 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.189" (uid=1000 pid=13798 comm="systemctl start fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ") Dec 26 17:39:14 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.189" (uid=1000 pid=13798 comm="systemctl start fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ") Dec 26 17:42:03 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.190" (uid=1000 pid=13815 comm="/bin/systemctl start fail2ban.service ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ") Dec 26 17:42:03 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.190" (uid=1000 pid=13815 comm="/bin/systemctl start fail2ban.service ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
It seems you haven't enabled sshd jail in /etc/fail2ban/jail.conf. Please ensure that you have "enabled = true" in sshd statement in your jail.conf (sshd jail isn't enabled by default). After you enable sshd jail, you might hit another problem - current F20 fail2ban is not compatible with firewalld so if you use firewalld instead of iptables, please wait some time for fixed package. I will release update which adds firewalld support for fail2ban during this week.
Adam, you are right, "enabled = true" was commented out for sshd in in /etc/fail2ban/jail.conf. I will wait for your update as recommended, if there is any other testing info that you need from me, please let me know. Thanks for looking into it.
Adam, it looks like fail2ban-0.9-0.3.git1f1a561.fc20.noarch regresses bug #979622 in that there is no firewall-cmd action anymore. Did the firewall-cmd get integrated in a different way between 0.8.1 and 0.9.0?
Ok, first for enabling jails please create a /etc/fail2ban/jail.local that contains: [ssh-iptables] enabled = true action = firewall-cmd-direct-new The 0.9 builds of fail2ban are significantly older than the 0.8.11 ones (http://pkgs.fedoraproject.org/cgit/fail2ban.git/ ) hence it got made before firewall-cmd was introduced. As you can see from https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d there still is a firewallcmd-new (the original 0.8 name too long) and a new firewallcmd-ipset. In a future 0.9 you'll be able to set banaction = firewallcmd-ipset in jail.local and make all jails use this by default. Derek, I hope this answers your questions. There's still a lot of work going on with 0.9 but we will do a release eventually. Daniel upstream fail2ban dev
Daniel, I'm rather new to both firewalld and fail2ban (I've been using swatch for the past ~5+ years to protect my systems), so I'm still on the beginning of the learning curve. But it looks like the current fail2ban package(s) in Fedora are just plain broken w.r.t. firewalld. I don't know why they upgraded to 0.9.0 instead of releasing 0.8.11. Maybe Orion Poplawski can answer that. So I guess my question is: what do I have to do with my current FC20 system to enable fail2ban to work with firewalld? I don't see anything in the installed package that references firewall-cmd. Thanks for your help.
firewalld was a quick last minute addition before the 0.8.11 release and I did get a few things wrong. Both 0.8.11 and 0.9.0 where being developed at the same time. I'm assuming Orion was getting a fail2ban package ready for a 0.9.0 release upstream (which hasn't happened yet). To get a firewalld working version of fail2ban you can download firewallcmd-ipset.conf or firewallcmd-new from https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d and put it into your /etc/fail2ban/action.d directory. Then set "action = firewallcmd-ipset" (or firewallcmd-new) in your /etc/fail2ban/jail.local file for the appropriate jail. Recommend using ipset version and have the ipset package installed.
Thanks, Daniel. I'll try it out. I did take a quick look and it looks like both sets are limited to ipv4. What happens if an attack comes in via ipv6? What would it take to get v6 support? Thanks!
> What happens if an attack comes in via ipv6? It doesn't get detected. > What would it take to get v6 support? About 2-3 full days of uninterrupted development time.
I've been busy and distracted with other things, and yes waiting for 0.9.0. However, it may be time to update to a newer 0.9.0 snapshot. Daniel - any suggestion? Current branch head okay?
I'd suggest waiting until I write a quick implementation of https://github.com/fail2ban/fail2ban/issues/551 which is waiting on Steven to finish https://github.com/fail2ban/fail2ban/pull/549 (its almost there). I'll see how the other folks feel about calling that, and a few other PRs, as a release. Then we've got to do a bit of QA to ensure that all filters have meaningful jail definitions (help gratefully accepted).
(In reply to Daniel Black from comment #8) > To get a firewalld working version of fail2ban you can download > firewallcmd-ipset.conf or firewallcmd-new from > https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d and put it > into your /etc/fail2ban/action.d directory. > > Then set "action = firewallcmd-ipset" (or firewallcmd-new) in your > /etc/fail2ban/jail.local file for the appropriate jail. Recommend using > ipset version and have the ipset package installed. I did this (using the ipset version) but it doesn't appear to be working properly: Jan 22 18:46:16 mail2 fail2ban.server.actions: WARNING [sshd] Ban 218.242.101.164 Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- stdout: '' Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- stderr: '' Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- returned 1 Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR Invariant check failed. Trying to restore a sane environment [snipped a bunch of "already banned" messages] [snipped the failure to unban] [root@mail2 ~]# firewall-cmd --direct --get-chains ipv4 filter [root@mail2 ~]# ipset -V ipset v6.20.1, protocol version: 6 [root@mail2 ~]# fail2ban-client status Status |- Number of jail: 6 `- Jail list: apache-auth, apache-badbots, apache-noscript, apache-overflows, sshd, sshd-ddos
(In reply to Derek Atkins from comment #13) I made a mistake. firewallcmd-ipset shouldn't have that actioncheck. You can leave it blank until I can think of a better one. Thanks for telling me.
(In reply to Daniel Black from comment #14) > (In reply to Derek Atkins from comment #13) > > I made a mistake. firewallcmd-ipset shouldn't have that actioncheck. You can > leave it blank until I can think of a better one. Thanks for telling me. Okay. I just cleared it, but I noticed that before I cleared it the banning wasn't doing anything (was that due to the actioncheck?) In particular in my logs I see: Jan 23 11:18:25 mail2 fail2ban.server.actions: INFO [sshd] 185.26.144.140 already banned Jan 23 11:18:47 mail2 fail2ban.server.actions: INFO [sshd] 185.26.144.140 already banned Jan 23 11:19:07 mail2 fail2ban.server.actions: INFO [sshd] 185.26.144.140 already banned And corresponding entries in /var/log/secure of failed attempts. Of course this happens after: Jan 23 11:18:08 mail2 fail2ban.server.action: ERROR Invariant check failed. Trying to restore a sane environment and Jan 23 11:18:10 mail2 fail2ban.server.action: CRITICAL Unable to restore environment ... so maybe the ban didn't take place? Clearly fail2ban *thought* it was banned, but iptables didn't because ssh was still getting through: Jan 23 11:18:22 mail2 sshd[20356]: reverse mapping checking getaddrinfo for 140144.rdns.hemenhosting.org [185.26.144.140] failed - POSSIBLE BREAK-IN ATTEMPT! Jan 23 11:18:22 mail2 sshd[20356]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.26.144.140 user=root Jan 23 11:18:24 mail2 sshd[20356]: Failed password for root from 185.26.144.140 port 52364 ssh2 Jan 23 11:18:24 mail2 sshd[20356]: Received disconnect from 185.26.144.140: 11: Bye Bye [preauth] I've restarted fail2ban after removing the actioncheck; now I'm waiting for the next script kiddie to hit this server. I'll let you know what happens.
Derek, Yes the original "Invariant check failed." was because the actioncheck was wrong. It couldn't ban it because it didn't think the firewalld bits where right. There is a fail2ban users group to seek help on http://sourceforge.net/p/fail2ban/mailman/?source=navbar
Ahh, just had a hit. Looks like after removing the actioncheck the ban actually takes effect properly.
Daniel - status update on the 0.9 branch please? I think we are long overdue for an update in Fedora to fix a number of issues.
0.9 Status update - aka thinks hoping to fix before a release. I was hoping to fix https://github.com/fail2ban/fail2ban/issues/509 / https://github.com/fail2ban/fail2ban/issues/597 which seem fairly major and likely to affect 0.9 too (though I haven't tested this assumption). That and merging https://github.com/fail2ban/fail2ban/issues/315 (Thanks for your feedback. Waiting on Yaroslav ) Was also thinking of removing jail.conf entries that just show off alternative actions/paths but I'm open for feedback on this one.
(In reply to Daniel Black from comment #19) > Was also thinking of removing jail.conf entries that just show off > alternative actions/paths but I'm open for feedback on this one. IMHO I think it would be reasonable to have a documentation example showing the alternate actions/paths, but it would be nice to just have a jail.conf that only contains the main actions/paths. Ideally I'd like to have a jail.local that has: [DEFAULT] enabled=true banaction = firewallcmd-ipset And have it work for all the pre-defined jails. Alas, right now I have to go through and pick-and-choose which ones to enable (and hope I didn't choose wrong).
0.9.0 released > nice to just have a jail.conf that only contains the main actions/paths. Done. Paths have been abstracted out to paths-{distro}.conf. The guff of variants have been removed from jail.conf. > Ideally I'd like to have a jail.local that has [DEFAULT] banaction = firewallcmd-ipset without enabled=true this works > And have it work for all the pre-defined jails. Alas, right now I have to go through and pick-and-choose which ones to enable Haven't quite got to autoenable for jails. With multiple things from sendmail,postfix,dovecot etc looking at a maillog is seems a little over the top to enable all of those if a mail.log exists. Better ideas on autoenableing welcome on the fail2ban-users email list.
Daniel - is firewallcmd-ipset preferred over firewallcmd-new? How about a fail2ban-firewalld sub-package with: /etc/fail2ban/jail.d/fedora-firewalld.conf: [DEFAULT] banaction = firewallcmd-ipset and requires firewalld and ipset?
OK, with this minimal /etc/fail2ban/jail.d/jail.local [DEFAULT] bantime = 3600 banaction = firewallcmd-ipset backend = systemd [sshd] enabled = true I get the output below - seems firewalld is struggling to initialize the ipset for fail2ban. 2014-03-19 19:36:14,812 fail2ban.server.server[5711]: INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.0 2014-03-19 19:36:14,813 fail2ban.server.database[5711]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3' 2014-03-19 19:36:14,891 fail2ban.server.jail[5711]: INFO Creating new jail 'sshd' 2014-03-19 19:36:14,891 fail2ban.server.jail[5711]: INFO Jail 'sshd' uses systemd 2014-03-19 19:36:14,927 fail2ban.server.jail[5711]: INFO Initiated 'systemd' backend 2014-03-19 19:36:15,014 fail2ban.server.filter[5711]: INFO Set maxRetry = 5 2014-03-19 19:36:15,015 fail2ban.server.actions[5711]: INFO Set banTime = 3600 2014-03-19 19:36:15,015 fail2ban.server.filter[5711]: INFO Set findtime = 600 2014-03-19 19:36:15,015 fail2ban.server.filter[5711]: INFO Set maxlines = 10 2014-03-19 19:36:15,043 fail2ban.filter [5711]: INFO Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd' 2014-03-19 19:36:15,046 fail2ban.server.jail[5711]: INFO Jail 'sshd' started 2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR ipset create fail2ban-sshd hash:ip timeout 600 firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stdout: "\x1b[91mError: COMMAND_FAILED: '/sbin/iptables -t filter -I INPUT_direct 1 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable' failed: iptables v1.4.19.1: Set fail2ban-sshd doesn't exist.\n\nTry `iptables -h' or 'iptables --help' for more information.\x1b[00m\n" 2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR ipset create fail2ban-sshd hash:ip timeout 600 firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stderr: '/bin/sh: ipset: command not found\n' 2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR ipset create fail2ban-sshd hash:ip timeout 600 firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- returned 13 2014-03-19 19:36:15,148 fail2ban.server.actions[5711]: ERROR Failed to start jail 'sshd' action 'firewallcmd-ipset': Error starting action
you need the ipset package installed
Ah, yes, duh. Installing ipset then causes me to hit BZ 1069640 (which is fixed in an updated SELinux policy). The jail starts fine with firewallcmd-new
> Daniel - is firewallcmd-ipset preferred over firewallcmd-new? I generally prefer ipset as its a cleaner approach to big lists in the iptables list especially with the networking locks that involves when pushing a new netfilter ruleset into production. Thomas seems in favour of it https://bugzilla.redhat.com/show_bug.cgi?id=979622#c8
Any update on this functionality? I am currently trying to use the firewallcmd-ipset.conf in action.d on my Fedora 20 system with a similar configuration to Jonathan but I am running into a SELinux denial. I can attach it here or open a new bug (not sure which you would prefer as it's not official functionality). Summary of denial: SELinux is preventing /usr/bin/python2.7 from using the net_admin capability.
For SELinux issues, file a new bug against selinux-policy.
Reopen if there are still problems with fail2ban-0.9-2.fc20.noarch