Bug 1177336

Problem here is that rsyslog does not use the syslog.socket but reads directly from journal. I am not sure if there is something we can do on systemd side.

Tomas, any idea from rsyslog point of view?

I agree with Lukáš on this. Looks like a not-a-bug.

As rsyslog doesn't use the "traditional syslog socket" anymore by default, no messages are received in your configuration.

Maybe the systemd documentation can be more precise, e.g.:
"Forwarding to ... a syslog {,daemon via a} socket...".

I have some doubts about this.

# rpm -qf /etc/rsyslog.d/listen.conf
systemd-208-11.el7_0.5.x86_64

# cat listen.conf 
$SystemLogSocketName /run/systemd/journal/syslog

# ls -l /run/systemd/journal/syslog
ls: cannot access /run/systemd/journal/syslog: No such file or directory

# lsof | grep /syslog
#

OK. Fine...

# echo 'input(type="imuxsock" Socket="/dev/log" CreatePath="off" Unlink="off")' > /etc/rsyslog.d/dummy.conf

# sed -ie 's/volatile/none/' /etc/systemd/journald.conf

# service systemd-journald restart
Redirecting to /bin/systemctl restart  systemd-journald.service

# service rsyslog restart
Redirecting to /bin/systemctl restart  rsyslog.service

# lsof | grep /dev/log
systemd       1            root   25u     unix 0xffff8801397ce1c0       0t0       7217 /dev/log
systemd-j 14921            root    5u     unix 0xffff8801397ce1c0       0t0       7217 /dev/log
rsyslogd  14940            root    3u     unix 0xffff880102763480       0t0     995945 /dev/log
in:imuxso 14940 14941      root    3u     unix 0xffff880102763480       0t0     995945 /dev/log
in:imjour 14940 14942      root    3u     unix 0xffff880102763480       0t0     995945 /dev/log
rs:main   14940 14943      root    3u     unix 0xffff880102763480       0t0     995945 /dev/log
rs:action 14940 14944      root    3u     unix 0xffff880102763480       0t0     995945 /dev/log

# systemctl daemon-reload
Nothing logged to /var/log/messages!

# service sshd restart
Nothing logged to /var/log/messages!

# rm -f /etc/rsyslog.d/dummy.conf 

# service rsyslog restart
Redirecting to /bin/systemctl restart  rsyslog.service

# sed -ie 's/=none/=volatile/' /etc/systemd/journald.conf

# service systemd-journald restart
Redirecting to /bin/systemctl restart  systemd-journald.service

# systemctl daemon-reload
>>
Jan  5 19:25:34 XXXX systemd: Reloading.
<<

# service sshd restart
Redirecting to /bin/systemctl restart  sshd.service
>>
Jan  5 19:26:03 XXXX systemd: Stopping OpenSSH server daemon...
Jan  5 19:26:03 XXXX systemd: Starting OpenSSH server daemon...
Jan  5 19:26:03 XXXX systemd: Started OpenSSH server daemon.
<<

(In reply to Gabriel VLASIU from comment #4)
> I have some doubts about this.

Let's dispel them.

> # rpm -qf /etc/rsyslog.d/listen.conf
> systemd-208-11.el7_0.5.x86_64
> 
> # cat listen.conf 
> $SystemLogSocketName /run/systemd/journal/syslog
> 
> # ls -l /run/systemd/journal/syslog
> ls: cannot access /run/systemd/journal/syslog: No such file or directory
> 
> # lsof | grep /syslog
> #
> 
> OK. Fine...

Try "systemctl status syslog.socket". That's where the socket is actually defined.
The manual page will refer you to this document:

http://www.freedesktop.org/wiki/Software/systemd/syslog

Since your rsyslog.service file has "Requires=syslog.socket" commented out, it is not started and the file doesn't exist.


> # echo 'input(type="imuxsock" Socket="/dev/log" CreatePath="off" Unlink="off")' > /etc/rsyslog.d/dummy.conf

This is most likely not what you'd want to do. This way you steal the socket from under journald and Bad Things may happen.
And you mix the new rsyslog syntax with the old one which is probably still used in /etc/rsyslog.conf. That doesn't work well.

You probably want to do:
# cat > /etc/rsyslog.d/dummy.conf
module(load="imuxsock" syssock.use="off")
input(type="imuxsock" Socket="/run/systemd/journal/syslog" CreatePath="off" Unlink="off")
# ln -s /lib/systemd/system/rsyslog.service /etc/systemd/system/syslog.service
# systemctl start syslog.socket

There are more nuances to it but this should be the bare minimum.


> # sed -ie 's/volatile/none/' /etc/systemd/journald.conf
> 
> # service systemd-journald restart
> Redirecting to /bin/systemctl restart  systemd-journald.service
> 
> # service rsyslog restart
> Redirecting to /bin/systemctl restart  rsyslog.service
> 
> # lsof | grep /dev/log
> systemd       1            root   25u     unix 0xffff8801397ce1c0       0t0 
> 7217 /dev/log
> systemd-j 14921            root    5u     unix 0xffff8801397ce1c0       0t0 
> 7217 /dev/log
> rsyslogd  14940            root    3u     unix 0xffff880102763480       0t0 
> 995945 /dev/log
> in:imuxso 14940 14941      root    3u     unix 0xffff880102763480       0t0 
> 995945 /dev/log
> in:imjour 14940 14942      root    3u     unix 0xffff880102763480       0t0 
> 995945 /dev/log
> rs:main   14940 14943      root    3u     unix 0xffff880102763480       0t0 
> 995945 /dev/log
> rs:action 14940 14944      root    3u     unix 0xffff880102763480       0t0 
> 995945 /dev/log
> 
> # systemctl daemon-reload
> Nothing logged to /var/log/messages!

I'm not sure systemd writes to /dev/log.


> # service sshd restart
> Nothing logged to /var/log/messages!

Who knows what has happened to /dev/log at this point. The bits in /etc/rsyslog.conf in old syntax probably don't have any effect in your setup.

The default setup works, you need to understand how the pieces fall together if you want to change it.

> The manual page will refer you to this document:
> 
> http://www.freedesktop.org/wiki/Software/systemd/syslog
No. The manual for journald.conf does not say anything about http://www.freedesktop.org/wiki/Software/systemd/syslog (nor systemd.special).


> The default setup works, you need to understand how the pieces fall together if
> you want to change it.
The default is not suitable for me. "The default setup works" - that's not an argument.
If journald had proper documentation we have not had this discussion.

(In reply to Gabriel VLASIU from comment #6)
> > The manual page will refer you to this document:
> > 
> > http://www.freedesktop.org/wiki/Software/systemd/syslog
>
> No. The manual for journald.conf does not say anything about
> http://www.freedesktop.org/wiki/Software/systemd/syslog (nor
> systemd.special).

systemd.special(7) and systemctl status syslog.socket both mention it here on my el7.

> > The default setup works, you need to understand how the pieces fall together if you want to change it.
> The default is not suitable for me.

Sorry to hear that.

> "The default setup works" - that's not an argument.

It is; the software is not broken, your configuration is.

> If journald had proper documentation we have not had this discussion.

We got to this point in comment 3.
The bit about syslog forwarding should be expanded to, at least, mention the syslog.socket unit and where to get documentation. Otherwise it's just about following links and reading up.

> systemd.special(7) and systemctl status syslog.socket both mention it here on my el7
Maybe. But "man journald.conf" does not. It just say:
"none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer or a syslog daemon will still work however.

And setting storage to none for journald in F19 has been working fine until EOL. No syslog, no rsyslog, no nothing.

Also rsyslog.service has some line commented and no explanation why those line are commented. Yes, I already opened that file prior to my bug report. man syslog.service (present in rsyslog.service) does not exist. And I missed to open/read syslog.service file.

We should probably mention the case of imjournal and storage=none in the manpage. But that is definitely 7.2 material.

(In reply to Gabriel VLASIU from comment #8)
> > systemd.special(7) and systemctl status syslog.socket both mention it here on my el7
> Maybe. But "man journald.conf" does not. It just say:
> "none" turns off all storage, all log data received will be dropped.
> Forwarding to other targets, such as the console, the kernel log buffer or a
> syslog daemon will still work however.

Yeah, this piece of documentation is what's needs to be fixed.

> And setting storage to none for journald in F19 has been working fine until
> EOL. No syslog, no rsyslog, no nothing.

I'm not totally sure I understand here - I assume you mean:
"messages reached /var/log/messages even with Storage=none"

There was a change from using imuxsock to using imjournal. Since systemd took ownership of /dev/log, staying with imuxsock would bring a different set of issues.

> Also rsyslog.service has some line commented and no explanation why those
> line are commented. Yes, I already opened that file prior to my bug report.
> man syslog.service (present in rsyslog.service) does not exist. And I missed
> to open/read syslog.service file.

The rsyslog.service file is provided by upstream and those lines are commented out by a script in the el7 rpm.
It might make sense to duplicate the information provided in the syslog.socket file into rsyslog.service file. I'll think about sending a patch upstream.

> I'm not totally sure I understand here - I assume you mean:
> "messages reached /var/log/messages even with Storage=none"
Yes. Exactly that.

# cat /etc/systemd/journald.conf:
...
# See journald.conf(5) for details

[Journal]
#Storage=auto
# Gabriel
#Storage=volatile
Storage=none
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=30s
....

I did not touch /etc/rsyslog.conf.
There where only 3 extra config files for rsyslog: /etc/rsyslog.d/listen.conf (systemd), one file that enable xconsole (there is a .service file that create /dev/xconsole with apropriate permissions before rsyslog start) and a file that enable syslog forwarding to a central syslog server. My laptop had this configuration and also a test server which is now migrated to RHEL7.

https://github.com/lnykryn/systemd-rhel/commit/24d007a0a8a77a6b75c6c7a403fc8d107875ebdc
https://github.com/lnykryn/systemd-rhel/commit/363e3817cc3462c42e837677768f10fa549966f7
->post

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2092.html