RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1113215 - rsyslogd logs kernel messages under the facility 'user' instead of 'kern'
Summary: rsyslogd logs kernel messages under the facility 'user' instead of 'kern'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.0
Hardware: x86_64
OS: Linux
low
low
Target Milestone: rc
: 7.1
Assignee: systemd-maint
QA Contact: Karel Volný
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-06-25 16:20 UTC by Chris Siebenmann
Modified: 2019-04-16 14:12 UTC (History)
7 users (show)

Fixed In Version: systemd-208-12.el7
Doc Type: Bug Fix
Doc Text:
Setting only SYSLOG_IDENTIFIER="kernel" was not sufficient and tools reading journal were confused by missing SYSLOG_FACILITY field for kernel log messages. Now we set SYSLOG_FACILITY field for kernel log messages too.
Clone Of:
Environment:
Last Closed: 2015-03-05 11:11:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Fix syslog facility for kernel logs (1.47 KB, patch)
2014-07-25 12:34 UTC, Michal Sekletar 		no flags Details | Diff
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
CentOS 7197 0 None None None Never
Red Hat Bugzilla 1204217 0 unspecified CLOSED rsyslog (or something) spams all consoles and shells with SELinux audit messages 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2015:0509 0 normal SHIPPED_LIVE systemd bug fix and enhancement update 2015-03-05 16:01:17 UTC

Internal Links: 1204217

Description Chris Siebenmann 2014-06-25 16:20:19 UTC 
Description of problem:
When logging kernel messages rsyslogd appears to consider them to be
in facility user. It logs them not under whatever was configured for
'kern.*' but under whatever was configured for 'user.*', and when
forwarding them to another machine via at least UDP it identifies them
as facility 'user' messages (verified both on the other machine and by
tcpdump of UDP traffic from a CentOS 7 preliminary test machine).

I don't think that the kernel is at fault here, as 'dmesg -f kern'
shows all the kernel messages while 'dmesg -f user' shows nothing.
In addition, using 'logger' to talk directly to rsyslogd also has
the same results.

Version-Release number of selected component (if applicable):

rsyslog-7.4.7-6.el7.x86_64
kernel-3.10.0-123.4.2.el7.x86_64

How reproducible:
Always.

Steps to Reproduce:
Create an /etc/rsyslog.d/test.conf that contains:

  kern.*      /var/log/test-kern
  user.*      /var/log/test-user

Reboot your test machine. Observe that kernel boot messages show up
in /var/log/test-user and don't show up in /var/log/test-kern.

Also:
  logger -p kern.info "This is a test"

Observe that this message again appeared in /var/log/test-user.

Additional info:

According to reports from rigeld2 on the #rhel IRC channel, this bug
is not present in an earlier rsyslog package from RHEL 7 beta; they
did not see the bug in rsyslog-7.4.2-4.el7.x86_64 but did after they
updated to the release rsyslog RPM.

Also observed and reported in/against CentOS, as
https://bugs.centos.org/view.php?id=7197

I have reproduced this on an official RHEL 7 install with the latest
updates.
Comment 2 Tomas Heinrich 2014-06-26 10:16:13 UTC 
(In reply to Chris Siebenmann from comment #0)
> Description of problem:
> When logging kernel messages rsyslogd appears to consider them to be
> in facility user. It logs them not under whatever was configured for
> 'kern.*' but under whatever was configured for 'user.*',

Hello Chris,

I think rsyslog is not at fault here, but rather systed/journald. rsyslog merely takes the logs from the journal, not /dev/log.

You can list the journald logs with more details using:
journalct -o json -f

If you now repeat your test by logging messages with different facilities, in the output of the command above, you can see a property named "SYSLOG_FACILITY". If journald labels the messages with the 'user' facility, rsyslog simply takes them as such.

If you can confirm that the issue is not on rsyslog's side, switch the bug to the systemd component, please.

> Additional info:
> 
> According to reports from rigeld2 on the #rhel IRC channel, this bug
> is not present in an earlier rsyslog package from RHEL 7 beta; they
> did not see the bug in rsyslog-7.4.2-4.el7.x86_64 but did after they
> updated to the release rsyslog RPM.

It might be of interest to compare the versions of systemd.
Comment 3 Chris Siebenmann 2014-06-26 15:09:19 UTC 
Okay, I don't know to interpret what I'm seeing in journalctl here so
I'm going to put it here. However, based on other evidence I don't think
that systemd is at fault here.

What I see for 'journalctl -o json --no-tail' and then looking for
both kernel and 'logger -p kern.info message' results, I get:

* real kernel messages have "_TRANSPORT" : "kernel",
  "SYSLOG_IDENTIFIER" : "kernel", "PRIORITY" : "6"; this is for a boot
  time message of 'XFS (dm-0): Ending clean mount'. They do not have
  a 'SYSLOG_FACILITY' setting.

* logger messages have: "_TRANSPORT" : "syslog", "SYSLOG_FACILITY" : "1",
  "SYSLOG_IDENTIFIER" : "root".

On my Fedora 20 machine, 'journalctl -o json' reports that real kernel
messages are (to pick one boot time message) "_TRANSPORT" : "kernel",
"SYSLOG_IDENTIFIER" : "kernel", "PRIORITY" : "6" (and no SYSLOG_FACILITY
setting). They do get logged correctly by the Fedora 20 rsyslogd, which
is rsyslog-7.4.8-1.fc20.1.x86_64. These fields are identical to what the
RHEL 7 systemd journal is showing.

The systemd on my RHEL 7 test machine is systemd-208-11.el7.x86_64.
Fedora 20 has systemd-208-19.fc20.x86_64.
Comment 4 Tomas Heinrich 2014-06-27 09:58:59 UTC 
(In reply to Chris Siebenmann from comment #3)
> Okay, I don't know to interpret what I'm seeing in journalctl here so
> I'm going to put it here. However, based on other evidence I don't think
> that systemd is at fault here.

Thanks for the data. I still think this is an issue in systemd, let me try to explain why.

> What I see for 'journalctl -o json --no-tail' and then looking for
> both kernel and 'logger -p kern.info message' results, I get:
> 
> * real kernel messages have "_TRANSPORT" : "kernel",
>   "SYSLOG_IDENTIFIER" : "kernel", "PRIORITY" : "6"; this is for a boot
>   time message of 'XFS (dm-0): Ending clean mount'. They do not have
>   a 'SYSLOG_FACILITY' setting.
> 
> * logger messages have: "_TRANSPORT" : "syslog", "SYSLOG_FACILITY" : "1",
>   "SYSLOG_IDENTIFIER" : "root".

The missing 'SYSLOG_FACILITY' field is the crux of the issue.

As per systemd.journal-fields(7), these fields should be supplied to be able to filter messages according to syslog priority. The imjournal plugin, that parses the messages from systemd, relies on these fields to be present. If they are not, default facility value is used. There is no guesswork involved.

It's not possible for all messages to have SYSLOG_FACILITY provided by the log source; not all logs come in through /dev/log. But the journald should be able to emulate it at least in the case of kernel messages.
It actually does so already - for the syslog socket it provides: the messages journald forwards to /run/systemd/journal/syslog have a correct kernel facility.
Therefore I think the missing SYSLOG_FACILITY field is a bug.

> On my Fedora 20 machine, 'journalctl -o json' reports that real kernel
> messages are (to pick one boot time message) "_TRANSPORT" : "kernel",
> "SYSLOG_IDENTIFIER" : "kernel", "PRIORITY" : "6" (and no SYSLOG_FACILITY
> setting). They do get logged correctly by the Fedora 20 rsyslogd, which
> is rsyslog-7.4.8-1.fc20.1.x86_64. 

Actually they don't.
There is a bug in the configuration on F20. The message excerpt above gets logged to /var/log/messsages, you can check. The messages that you see as having 'kern' facility are coming through the imklog plugin which is turned on by default by mistake. Thus there are duplicate kernel messages; one copy from imjournal and the other from imklog. The ones from imklog go nowhere by default.

> These fields are identical to what the
> RHEL 7 systemd journal is showing.

Yes, they are. The difference is the bug in configuration.
Comment 5 Chris Siebenmann 2014-06-27 21:44:18 UTC 
You're right about the situation on Fedora 20. I have a syslog destination
configured with (among other things) 'kern.!*' and it gets kernel log
messages. I was fooled by the extra copy showing up in my /var/log/kern
file that I have just for kern.* messages.

So I guess that this is a systemd bug instead of an rsyslog bug. I'll
switch the component to systemd as you requested.
Comment 6 Tomas Heinrich 2014-06-28 17:26:54 UTC 
(In reply to Chris Siebenmann from comment #5)
> So I guess that this is a systemd bug instead of an rsyslog bug. I'll
> switch the component to systemd as you requested.

Thanks. I'll reset the contact for the bug.
Comment 7 Michal Sekletar 2014-07-25 12:34:49 UTC 
Created attachment 920973 [details]
Fix syslog facility for kernel logs
Comment 8 Michal Sekletar 2014-07-28 08:36:54 UTC 
http://10.3.11.34/msekleta/systemd-rhel/commit/74f419223c04026d03495a20a49f30a9cb443b09 -> POST
Comment 9 Michal Sekletar 2014-07-28 08:53:52 UTC 
Amended commit message.

http://10.3.11.34/msekleta/systemd-rhel/commit/7f85c4184fc45c3dd2c76b4867ab252c3530f84d
Comment 15 Karel Volný 2015-01-22 14:30:58 UTC 
Note that the original reproducer is wrong, as `logger` cannot use the 'kern' facility. It gets converted to 'user' silently, without any error/warning and without mention in the docs (manpage) that it works this was - I've filed a new bug 1184558 for util-linux about this.
Comment 16 Karel Volný 2015-01-22 14:35:31 UTC 
(In reply to Karel Volný from comment #15)
s/this was/this way/

sorry for the typo

and I forgot to mention that I've tested the fix on real kernel messages which are processed correctly with the new systemd-journald version
Comment 19 errata-xmlrpc 2015-03-05 11:11:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0509.html
Note You need to log in before you can comment on or make changes to this bug.