Bug 1046816

Cau you please attach lines from /var/log/secure which indicates login failures? (Or simply attach your /var/log/secure if there are no private information there)

Also please attach messages from fail2ban daemon, from /var/log/messages (or attach your entire /var/log/messages).

Thank you in advance.

I can only give excerpts from /var/log/secure and /var/log/messages because of private information now, sorry if this does not help big, I plan to install F20 on another machine these days and then I can provide the whole files. 

The rejected fail2ban messages from /var/log/messages are my own doing trying to make sure fail2ban is running.

$ sudo less /var/log/secure | grep Failed

Dec 27 15:32:10 bhost sshd[29668]: Failed password for invalid user root from 114.80.246.146 port 7569 ssh2
Dec 27 15:32:14 bhost sshd[29671]: Failed password for invalid user root from 114.80.246.146 port 8250 ssh2
Dec 27 15:32:17 bhost sshd[29674]: Failed password for invalid user root from 114.80.246.146 port 8986 ssh2
Dec 27 15:32:22 bhost sshd[29677]: Failed password for invalid user root from 114.80.246.146 port 9751 ssh2
Dec 27 15:32:26 bhost sshd[29680]: Failed password for invalid user root from 114.80.246.146 port 10645 ssh2
Dec 27 15:32:30 bhost sshd[29683]: Failed password for invalid user root from 114.80.246.146 port 11312 ssh2
Dec 27 15:32:33 bhost sshd[29691]: Failed password for invalid user root from 114.80.246.146 port 11934 ssh2
Dec 27 15:32:37 bhost sshd[29694]: Failed password for invalid user root from 114.80.246.146 port 12466 ssh2
Dec 27 15:32:41 bhost sshd[29697]: Failed password for invalid user root from 114.80.246.146 port 13088 ssh2
Dec 27 15:32:45 bhost sshd[29700]: Failed password for invalid user root from 114.80.246.146 port 13751 ssh2
Dec 27 15:32:49 bhost sshd[29703]: Failed password for invalid user root from 114.80.246.146 port 14448 ssh2

$ sudo less /var/log/secure | grep Invalid

Dec 27 09:42:46 bhost sshd[23421]: Invalid user postgres from 212.146.83.246
Dec 27 09:42:50 bhost sshd[23423]: Invalid user postgres from 212.146.83.246
Dec 27 09:42:53 bhost sshd[23425]: Invalid user postgres from 212.146.83.246
Dec 27 09:42:56 bhost sshd[23427]: Invalid user postgres from 212.146.83.246
Dec 27 09:42:59 bhost sshd[23429]: Invalid user postgres from 212.146.83.246
Dec 27 09:43:04 bhost sshd[23431]: Invalid user postgres from 212.146.83.246
Dec 27 09:43:08 bhost sshd[23433]: Invalid user postgres from 212.146.83.246

$ sudo less /var/log/messages | grep fail2ban

Dec 26 13:24:42 bhost fail2ban-client: Shutdown successful
Dec 26 13:29:27 bhost fail2ban-client: 2013-12-26 13:29:27,257 fail2ban.server.server: INFO   Starting Fail2ban v0.9.0a1
Dec 26 13:29:27 bhost fail2ban-client: 2013-12-26 13:29:27,258 fail2ban.server.server: INFO   Starting in daemon mode
Dec 26 13:32:42 bhost fail2ban-client: Shutdown successful
Dec 26 13:33:46 bhost fail2ban-client: 2013-12-26 13:33:45,287 fail2ban.server.server: INFO   Starting Fail2ban v0.9.0a1
Dec 26 13:33:46 bhost fail2ban-client: 2013-12-26 13:33:45,287 fail2ban.server.server: INFO   Starting in daemon mode
Dec 26 17:39:03 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.188" (uid=1000 pid=13789 comm="systemctl enable fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="EnableUnitFiles" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
Dec 26 17:39:03 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.188" (uid=1000 pid=13789 comm="systemctl enable fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="EnableUnitFiles" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
Dec 26 17:39:14 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.189" (uid=1000 pid=13798 comm="systemctl start fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
Dec 26 17:39:14 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.189" (uid=1000 pid=13798 comm="systemctl start fail2ban.system ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
Dec 26 17:42:03 bhost dbus-daemon: dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.190" (uid=1000 pid=13815 comm="/bin/systemctl start fail2ban.service ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")
Dec 26 17:42:03 bhost dbus[659]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.190" (uid=1000 pid=13815 comm="/bin/systemctl start fail2ban.service ") interface="org.freedesktop.systemd1.Manager" member="StartUnit" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/usr/lib/systemd/systemd --switched-root --system ")

It seems you haven't enabled sshd jail in /etc/fail2ban/jail.conf. Please ensure that you have "enabled = true" in sshd statement in your jail.conf (sshd jail isn't enabled by default).

After you enable sshd jail, you might hit another problem - current F20 fail2ban is not compatible with firewalld so if you use firewalld instead of iptables, please wait some time for fixed package. I will release update which adds firewalld support for fail2ban during this week.

Adam, you are right, "enabled = true" was commented out for sshd in in /etc/fail2ban/jail.conf. I will wait for your update as recommended, if there is any other testing info that you need from me, please let me know. Thanks for looking into it.

Adam, it looks like fail2ban-0.9-0.3.git1f1a561.fc20.noarch regresses bug #979622 in that there is no firewall-cmd action anymore.  Did the firewall-cmd get integrated in a different way between 0.8.1 and 0.9.0?

Ok, first for enabling jails please create a /etc/fail2ban/jail.local that contains:

[ssh-iptables]
enabled = true
action = firewall-cmd-direct-new


The 0.9 builds of fail2ban are significantly older than the 0.8.11 ones (http://pkgs.fedoraproject.org/cgit/fail2ban.git/ ) hence it got made before firewall-cmd was introduced.

As you can see from https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d there still is a firewallcmd-new (the original 0.8 name too long) and a new firewallcmd-ipset. In a future 0.9 you'll be able to set banaction = firewallcmd-ipset in jail.local and make all jails use this by default.

Derek, I hope this answers your questions.

There's still a lot of work going on with 0.9 but we will do a release eventually.

Daniel
upstream fail2ban dev

Daniel,

I'm rather new to both firewalld and fail2ban (I've been using swatch for the past ~5+ years to protect my systems), so I'm still on the beginning of the learning curve.  But it looks like the current fail2ban package(s) in Fedora are just plain broken w.r.t. firewalld.  I don't know why they upgraded to 0.9.0 instead of releasing 0.8.11.  Maybe Orion Poplawski can answer that.

So I guess my question is:  what do I have to do with my current FC20 system to enable fail2ban to work with firewalld?  I don't see anything in the installed package that references firewall-cmd.

Thanks for your help.

firewalld was a quick last minute addition before the 0.8.11 release and I did get a few things wrong. Both 0.8.11 and 0.9.0 where being developed at the same time. I'm assuming Orion was getting a fail2ban package ready for a 0.9.0 release upstream (which hasn't happened yet).

To get a firewalld working version of fail2ban you can download firewallcmd-ipset.conf or firewallcmd-new from https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d and put it into your /etc/fail2ban/action.d directory.

Then set "action = firewallcmd-ipset"  (or firewallcmd-new) in your /etc/fail2ban/jail.local file for the appropriate jail. Recommend using ipset version and have the ipset package installed.

Thanks, Daniel.  I'll try it out.
I did take a quick look and it looks like both sets are limited to ipv4.
What happens if an attack comes in via ipv6?
What would it take to get v6 support?
Thanks!

> What happens if an attack comes in via ipv6?

It doesn't get detected.

> What would it take to get v6 support?

About 2-3 full days of uninterrupted development time.

I've been busy and distracted with other things, and yes waiting for 0.9.0.  However, it may be time to update to a newer 0.9.0 snapshot.  Daniel - any suggestion?  Current branch head okay?

I'd suggest waiting until I write a quick implementation of https://github.com/fail2ban/fail2ban/issues/551 which is waiting on Steven to finish https://github.com/fail2ban/fail2ban/pull/549 (its almost there). I'll see how the other folks feel about calling that, and a few other PRs, as a release.

Then we've got to do a bit of QA to ensure that all filters have meaningful jail definitions (help gratefully accepted).

(In reply to Daniel Black from comment #8)

> To get a firewalld working version of fail2ban you can download
> firewallcmd-ipset.conf or firewallcmd-new from
> https://github.com/fail2ban/fail2ban/tree/0.9/config/action.d and put it
> into your /etc/fail2ban/action.d directory.
> 
> Then set "action = firewallcmd-ipset"  (or firewallcmd-new) in your
> /etc/fail2ban/jail.local file for the appropriate jail. Recommend using
> ipset version and have the ipset package installed.

I did this (using the ipset version) but it doesn't appear to be working properly:

Jan 22 18:46:16 mail2 fail2ban.server.actions: WARNING [sshd] Ban 218.242.101.164
Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR  firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- stdout: ''
Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR  firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- stderr: ''
Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR  firewall-cmd --direct --get-chains ipv4 filter | grep -q '^fail2ban-sshd$' -- returned 1
Jan 22 18:46:16 mail2 fail2ban.server.action: ERROR  Invariant check failed. Trying to restore a sane environment
[snipped a bunch of "already banned" messages]
[snipped the failure to unban]

[root@mail2 ~]# firewall-cmd --direct --get-chains ipv4 filter
[root@mail2 ~]# ipset -V
ipset v6.20.1, protocol version: 6
[root@mail2 ~]# fail2ban-client status
Status
|- Number of jail:	6
`- Jail list:		apache-auth, apache-badbots, apache-noscript, apache-overflows, sshd, sshd-ddos

(In reply to Derek Atkins from comment #13)

I made a mistake. firewallcmd-ipset shouldn't have that actioncheck. You can leave it blank until I can think of a better one. Thanks for telling me.

(In reply to Daniel Black from comment #14)
> (In reply to Derek Atkins from comment #13)
> 
> I made a mistake. firewallcmd-ipset shouldn't have that actioncheck. You can
> leave it blank until I can think of a better one. Thanks for telling me.

Okay.  I just cleared it, but I noticed that before I cleared it the banning wasn't doing anything (was that due to the actioncheck?)  In particular in my logs I see:

Jan 23 11:18:25 mail2 fail2ban.server.actions: INFO   [sshd] 185.26.144.140 already banned
Jan 23 11:18:47 mail2 fail2ban.server.actions: INFO   [sshd] 185.26.144.140 already banned
Jan 23 11:19:07 mail2 fail2ban.server.actions: INFO   [sshd] 185.26.144.140 already banned

And corresponding entries in /var/log/secure of failed attempts.  Of course this happens after:

Jan 23 11:18:08 mail2 fail2ban.server.action: ERROR  Invariant check failed. Trying to restore a sane environment

and 

Jan 23 11:18:10 mail2 fail2ban.server.action: CRITICAL Unable to restore environment

... so maybe the ban didn't take place?  Clearly fail2ban *thought* it was banned, but iptables didn't because ssh was still getting through:

Jan 23 11:18:22 mail2 sshd[20356]: reverse mapping checking getaddrinfo for 140144.rdns.hemenhosting.org [185.26.144.140] failed - POSSIBLE BREAK-IN ATTEMPT!
Jan 23 11:18:22 mail2 sshd[20356]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=185.26.144.140  user=root
Jan 23 11:18:24 mail2 sshd[20356]: Failed password for root from 185.26.144.140 port 52364 ssh2
Jan 23 11:18:24 mail2 sshd[20356]: Received disconnect from 185.26.144.140: 11: Bye Bye [preauth]

I've restarted fail2ban after removing the actioncheck; now I'm waiting for the next script kiddie to hit this server.  I'll let you know what happens.

Derek,

Yes the original "Invariant check failed." was because the actioncheck was wrong. It couldn't ban it because it didn't think the firewalld bits where right.

There is a fail2ban users group to seek help on http://sourceforge.net/p/fail2ban/mailman/?source=navbar

Ahh, just had a hit.  Looks like after removing the actioncheck the ban actually takes effect properly.

Daniel - status update on the 0.9 branch please?  I think we are long overdue for an update in Fedora to fix a number of issues.

0.9 Status update - aka thinks hoping to fix before a release.

I was hoping to fix https://github.com/fail2ban/fail2ban/issues/509 / https://github.com/fail2ban/fail2ban/issues/597 which seem fairly major and likely to affect 0.9 too (though I haven't tested this assumption).

That and merging https://github.com/fail2ban/fail2ban/issues/315 (Thanks for your feedback. Waiting on Yaroslav )

Was also thinking of removing jail.conf entries that just show off alternative actions/paths but I'm open for feedback on this one.

(In reply to Daniel Black from comment #19)

> Was also thinking of removing jail.conf entries that just show off
> alternative actions/paths but I'm open for feedback on this one.

IMHO I think it would be reasonable to have a documentation example showing the alternate actions/paths, but it would be nice to just have a jail.conf that only contains the main actions/paths.  Ideally I'd like to have a jail.local that has:

[DEFAULT]
enabled=true
banaction = firewallcmd-ipset

And have it work for all the pre-defined jails.  Alas, right now I have to go through and pick-and-choose which ones to enable (and hope I didn't choose wrong).

0.9.0 released

> nice to just have a jail.conf that only contains the main actions/paths. 

Done. Paths have been abstracted out to paths-{distro}.conf. The guff of variants have been removed from jail.conf.

> Ideally I'd like to have a jail.local that has

[DEFAULT]
banaction = firewallcmd-ipset


without enabled=true this works

> And have it work for all the pre-defined jails.  Alas, right now I have to go through and pick-and-choose which ones to enable

Haven't quite got to autoenable for jails. With multiple things from sendmail,postfix,dovecot etc looking at a maillog is seems a little over the top to enable all of those if a mail.log exists.

Better ideas on autoenableing welcome on the fail2ban-users email list.

Daniel - is firewallcmd-ipset preferred over firewallcmd-new?

How about a fail2ban-firewalld sub-package with:

/etc/fail2ban/jail.d/fedora-firewalld.conf:
[DEFAULT]
banaction = firewallcmd-ipset

and requires firewalld and ipset?

OK, with this minimal /etc/fail2ban/jail.d/jail.local

[DEFAULT]
bantime = 3600
banaction = firewallcmd-ipset
backend = systemd

[sshd]
enabled = true


I get the output below - seems firewalld is struggling to initialize the ipset for fail2ban.

2014-03-19 19:36:14,812 fail2ban.server.server[5711]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.0
2014-03-19 19:36:14,813 fail2ban.server.database[5711]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2014-03-19 19:36:14,891 fail2ban.server.jail[5711]: INFO    Creating new jail 'sshd'
2014-03-19 19:36:14,891 fail2ban.server.jail[5711]: INFO    Jail 'sshd' uses systemd
2014-03-19 19:36:14,927 fail2ban.server.jail[5711]: INFO    Initiated 'systemd' backend
2014-03-19 19:36:15,014 fail2ban.server.filter[5711]: INFO    Set maxRetry = 5
2014-03-19 19:36:15,015 fail2ban.server.actions[5711]: INFO    Set banTime = 3600
2014-03-19 19:36:15,015 fail2ban.server.filter[5711]: INFO    Set findtime = 600
2014-03-19 19:36:15,015 fail2ban.server.filter[5711]: INFO    Set maxlines = 10
2014-03-19 19:36:15,043 fail2ban.filter [5711]: INFO    Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2014-03-19 19:36:15,046 fail2ban.server.jail[5711]: INFO    Jail 'sshd' started
2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR   ipset create fail2ban-sshd hash:ip timeout 600
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stdout: "\x1b[91mError: COMMAND_FAILED: '/sbin/iptables -t filter -I INPUT_direct 1 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable' failed: iptables v1.4.19.1: Set fail2ban-sshd doesn't exist.\n\nTry `iptables -h' or 'iptables --help' for more information.\x1b[00m\n"
2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR   ipset create fail2ban-sshd hash:ip timeout 600
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stderr: '/bin/sh: ipset: command not found\n'
2014-03-19 19:36:15,148 fail2ban.server.action[5711]: ERROR   ipset create fail2ban-sshd hash:ip timeout 600
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- returned 13
2014-03-19 19:36:15,148 fail2ban.server.actions[5711]: ERROR   Failed to start jail 'sshd' action 'firewallcmd-ipset': Error starting action

you need the ipset package installed

Ah, yes, duh. Installing ipset then causes me to hit BZ 1069640 (which is fixed in an updated SELinux policy). The jail starts fine with firewallcmd-new

> Daniel - is firewallcmd-ipset preferred over firewallcmd-new?

I generally prefer ipset as its a cleaner approach to big lists in the iptables list especially with the networking locks that involves when pushing a new netfilter ruleset into production.

Thomas seems in favour of it https://bugzilla.redhat.com/show_bug.cgi?id=979622#c8

Any update on this functionality? I am currently trying to use the firewallcmd-ipset.conf in action.d on my Fedora 20 system with a similar configuration to Jonathan but I am running into a SELinux denial. I can attach it here or open a new bug (not sure which you would prefer as it's not official functionality).

Summary of denial: SELinux is preventing /usr/bin/python2.7 from using the net_admin capability.

For SELinux issues, file a new bug against selinux-policy.

Reopen if there are still problems with fail2ban-0.9-2.fc20.noarch