Bug 1048639 - Logwatch doesn't process fail2ban logs
Summary: Logwatch doesn't process fail2ban logs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: logwatch
Version: 22
Hardware: noarch
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Jan Synacek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-06 01:32 UTC by Edward Kuns
Modified: 2016-02-16 13:03 UTC (History)
11 users (show)

Fixed In Version: logwatch-7.4.1-5.20150731svn293.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-03 21:18:36 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Proposed patch to fix logwatch processing for fail2ban (2.44 KB, patch)
2014-01-06 01:41 UTC, Edward Kuns
no flags Details | Diff
Proposed patch (3.52 KB, patch)
2014-10-18 19:15 UTC, Alan Hamilton
no flags Details | Diff

Description Edward Kuns 2014-01-06 01:32:41 UTC
Description of problem:

Logwatch doesn't process the fail2ban logs.  In Fedora 19, fail2ban is set up to use syslog, and its messages go to /var/log/messages.  Logwatch assumes that the fail2ban logs will be found in /var/log/fail2ban.  After fixing this, I found many unknown messages, so I fixed those as well.

Version-Release number of selected component (if applicable):

logwatch-7.4.0-25.20130522svn140.fc19.noarch
fail2ban-0.8.11-2.fc19.noarch

How reproducible:

100%

Steps to Reproduce:
1. Configure fail2ban in the default Fedora 19 manner
2. Wait for fail2ban to ban something, or force it to do so
3. Wait for the next logwatch execution, or manually run logwatch

Actual results:

No fail2ban analysis in the logwatch output.

Expected results:

I expect to see fail2ban analysis in the logwatch output.

Additional info:

I will attach the patch I used to fix this.  I don't know how different Fedora's fail2ban is from the default, so I don't know whether or not these patches apply to the generic logwatch distribution or not.  But they definitely apply to Fedora, as it configures these applications.

Comment 1 Edward Kuns 2014-01-06 01:41:51 UTC
Created attachment 845873 [details]
Proposed patch to fix logwatch processing for fail2ban

The patch should apply just fine from /usr/share/logwatch -- I fixed these issues:

1) LogWatch expects fail2ban to have its own log file, when it logs (by default) to /var/log/messages

2) fail2ban logs to syslog using a variety of service names, and we want to process all of them.

3) There are some new lines output by fail2ban that we want to ignore, such as when it says a jail uses pyinotify.  I added all of these lines that I have personally observed.

4) The "already banned" complaint is now output by fail2ban at INFO, not WARNING.  There may be more instances of such changes, of course.  I only changed the instance I have personally observed.

Comment 2 Gerald Cox 2014-05-04 18:00:23 UTC
Thanks.  Testing for F20.

Comment 3 Gerald Cox 2014-05-06 15:53:52 UTC
Works great for F20.  Hope this patch is incorporated into the rpm.
Thanks again!

Comment 4 Gerald Cox 2014-05-06 15:54:54 UTC
Updated to F20 as this also applies there...

Comment 5 Jan Synacek 2014-05-12 10:25:55 UTC
Edward, since you already wrote a patch, would you mind proposing it upstream (http://sourceforge.net/p/logwatch/mailman/logwatch-devel) ?

Comment 6 Gerald Cox 2014-05-25 18:38:55 UTC
Jan, Edward:  Regarding the patch... I just installed the update fail2ban-0.9-2.fc20.noarch and now the messages are no longer going to /var/log/messages... now they go to /var/log/fail2ban.log which is where they probably should have been going in the first place.

Comment 7 Gerald Cox 2014-05-26 18:01:16 UTC
Looks like I missed the fact that logwatch is looking for "fail2ban" and not the new location which is "fail2ban.log".  I made the change and now am testing.

Comment 8 Gerald Cox 2014-05-28 15:47:35 UTC
With the latest update the patch no longer works.

Comment 9 Kevin Masaryk 2014-06-02 22:50:29 UTC
This is also affecting me...

Did a "yum downgrade fail2ban" to go back to fail2ban-0.9-0.3.git1f1a561.fc20.noarch from fail2ban-0.9-2.fc20.noarch and logwatcher is working again.

I did a "yum changelog 3 fail2ban" before downgrading and didn't see much:

fail2ban-0.9-2.fc20.noarch               installed
* Tue Mar 18 05:00:00 2014 Orion Poplawski <orion.com> - 0.9-2
- Use Fedora paths
- Start after firewalld (bug #1067147)

* Mon Mar 17 05:00:00 2014 Orion Poplawski <orion.com> - 0.9-1
- Update to 0.9

* Tue Sep 24 05:00:00 2013 Orion Poplawski <orion.com> - 0.9-0.3.git1f1a561
- Update to current 0.9 git branch
- Rebase init patch, drop jail.d and notmp patch applied upstream

Comment 10 Edward Kuns 2014-06-15 04:37:37 UTC
Unfortunately, I am still on Fedora 19.  I haven't had the free time to upgrade this system.  This means I'm not able to look at what is going on with Fedora 20.  However, I did supply my patch upstream as

https://sourceforge.net/p/logwatch/patches/35/

That will handle the other improvements I made to the logwatch script for handling fail2ban logs.

Fedora 19 is still on older versions of these packages:

fail2ban-0.8.11-2.fc19.noarch
logwatch-7.4.0-25.20130522svn140.fc19.noarch

Comment 11 Alan Hamilton 2014-06-22 22:04:21 UTC
I just set up fail2ban, and noticed the lack of reporting in Logwatch.

From looking at /usr/share/logwatch/scripts/services/fail2ban, the issue appears that it's expecting messages preceded by WARNING rather than NOTICE.

Example fail2ban line:
2014-06-22 14:40:29,084 fail2ban.server.actions[6938]: NOTICE  [postfix-sasl] Ban xx.xx.xx.xx

Logwatch is looking for a line staring with WARNING:
    } elsif ( my ($Service,$Action,$Host) = ($ThisLine =~ m/WARNING:?\s\[?(.*?)[]:]?\s(Ban|Unban)[^\.]* (\S+)/)) {

So it doesn't match and doesn't get reported. If I change this to NOTICE, it works.

Comment 12 Gerald Cox 2014-06-23 23:34:29 UTC
@Alan, thanks very much for the circumvention.  Worked perfectly!  

@Edward, did you put that into your patch that you submitted upstream?

Comment 13 Gerald Cox 2014-08-17 16:10:10 UTC
There was an update to logwatch:
logwatch-7.4.0-33.20140704svn198.fc20.noarch

Which meant I needed to reapply the change from @Alex described above.

Comment 14 Gerald Cox 2014-08-17 16:28:09 UTC
Created upstream ticket:
https://sourceforge.net/p/logwatch/bugs/37/

Comment 15 Gerald Cox 2014-09-09 22:26:00 UTC
Edward, the patch you suggested was accepted and incorporated upstream and will be available in v7.4.1 - refer to the upstream ticket:
http://sourceforge.net/p/logwatch/patches/35/
for more information.

Alan's patch is also required in addition to what Edward reported.  The ticket I created reporting Alan's change was accepted and incorporated upstream and will be available in v7.4.1 - refer to the upstream ticket:
https://sourceforge.net/p/logwatch/bugs/37/
for more information.

Comment 16 Jan Synacek 2014-09-24 07:01:04 UTC
Please, try https://admin.fedoraproject.org/updates/logwatch-7.4.1-1.20140924svn242.fc20 if it resolves the issue.

Comment 17 Gerald Cox 2014-10-17 14:08:51 UTC
No, it isn't working.  I'll go back upstream and ask them what happened.

Comment 18 Alan Hamilton 2014-10-18 19:15:38 UTC
Created attachment 948123 [details]
Proposed patch

I went through the issues with the current release (logwatch-7.4.1-2.20140924svn242) and came up with a patch.

In the config file, OnlySerice only works with syslogd-style loglines, but fail2ban's log has a different date format so nothing matches. Also, the empty regex in (|\..+) (before the pipe) will match anything.

I changed it to use OnlyContains, and match "fail2ban" not followed by a '-'.

In the script file, I added some additional matches to keep informational lines from showing up as unmatched. These mismatches don't print at --detail low, but will at higher --detail levels.

Service names were showing up with a leading square bracket, like "[dovecot". This is due to a missing + in line 86.

"Already banned" messages can show up with a NOTICE tag. so I added that.

Comment 19 Gerald Cox 2014-10-19 14:09:24 UTC
Thanks for the patch Alan, works great.  Are you going to propose upstream or do you want me to?

Comment 20 Alan Hamilton 2014-10-19 19:08:10 UTC
Sure, go ahead and submit it.

Comment 21 Gerald Cox 2014-10-19 22:17:43 UTC
Created upstream ticket:
https://sourceforge.net/p/logwatch/patches/38/

Comment 22 Derek Atkins 2015-03-12 13:31:41 UTC
Just adding a "me too".  I tried this patch and it works for me.  (although I'd kind of like to see a list of the banned IPs, but a summary is okay -- and I can see the list from the sshd errors).

Running FC20 (I'll try FC21 shortly).

Comment 23 Alan Hamilton 2015-03-12 14:54:16 UTC
(In reply to Derek Atkins from comment #22)

You can display the IPs by setting "Detail = 5" in conf/services/fail2ban.conf.

This bug is also present in FC21.

Comment 24 Fedora End Of Life 2015-05-29 10:21:13 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 25 Gerald Cox 2015-05-29 20:49:41 UTC
Carry forward to F22.

Comment 26 Jan Synacek 2015-07-31 12:08:00 UTC
Could anyone please try again with the latest package? It's built for F23, but it should work in F22 as well.

http://koji.fedoraproject.org/koji/buildinfo?buildID=674481

Comment 27 Gerald Cox 2015-08-01 19:30:02 UTC
(In reply to Jan Synacek from comment #26)
> Could anyone please try again with the latest package? It's built for F23,
> but it should work in F22 as well.
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=674481

Thanks, I'll give it a whirl.

Comment 28 Gerald Cox 2015-08-16 16:34:53 UTC
Hey Jan,

It works!  This can now be closed.  Thanks much!

Comment 29 Fedora Update System 2015-08-25 09:29:50 UTC
logwatch-7.4.1-5.20150731svn293.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-14203

Comment 30 Fedora Update System 2015-08-26 17:27:37 UTC
logwatch-7.4.1-5.20150731svn293.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.\nIf you want to test the update, you can install it with \n su -c 'yum --enablerepo=updates-testing update logwatch'. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-14203

Comment 31 Fedora Update System 2015-10-03 21:18:33 UTC
logwatch-7.4.1-5.20150731svn293.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 32 Osipov Oleg 2016-02-16 09:44:56 UTC
The version logwatch-7.4.1-5.20150731svn293.fc22 is imperfect. It still outputs service names with extra "[":

--------------------- fail2ban-messages Begin ------------------------ 
 Banned services with Fail2Ban:                          Bans:Unbans
     [sshd:                                                 [ 17:1  ]
---------------------- fail2ban-messages End ------------------------- 

Its happens because of wrong regex in /usr/share/logwatch/scripts/services/fail2ban on line 87:

m/(?:WARNING|NOTICE):?\s\[?(.*?)[]:]?\s(Ban|Unban)[^\.]* (\S+)/)

it should be

m/(?:WARNING|NOTICE):?\s+\[?(.*?)[]:]?\s(Ban|Unban)[^\.]* (\S+)/)

The patch by Alan Hamilton does have that extra plus signg but for some reason it ended missing from logwatch-7.4.1-5.20150731svn293.fc22.


Note You need to log in before you can comment on or make changes to this bug.