Hello,

Rsyslog is not responsible for managing content of the journal. By default, rsyslog reads logs from journal and outputs them to diverse destinations, such as /var/log/messages.

Could you compare the output of /var/log/messages with the output of 'journalctl' on both rhel7 and rhel8?
If there is a difference between the two outputs then it's definitely a bug. Otherwise, it needs to be solved somewhere else.

Thank you,
Attila

Hello Attila,
  Thanks for your help on this BUG.

  Right now, it looks like RHEL8 rsyslog haven’t record logs from journal and outputs them to /var/log/messages during reboot/shutdown.  RHEL7 is normal.

>Could you compare the output of /var/log/messages with the output of 'journalctl' on both rhel8?
>>In RHEL8 /var/log/messages cannot record graceful shutdown/reboot

Aug 14 05:30:01 testserver systemd[1]: sysstat-collect.service: Succeeded.
Aug 14 05:30:01 testserver systemd[1]: Started system activity accounting tool.
Aug 14 05:40:01 testserver systemd[1]: Starting system activity accounting tool...
Aug 14 05:40:01 testserver systemd[1]: sysstat-collect.service: Succeeded.
Aug 14 05:40:01 testserver systemd[1]: Started system activity accounting tool.
Aug 14 05:50:01 testserver systemd[1]: Starting system activity accounting tool...
Aug 14 05:50:01 testserver systemd[1]: sysstat-collect.service: Succeeded.
Aug 14 05:50:01 testserver systemd[1]: Started system activity accounting tool.    <———— initial reboot
Aug 14 05:58:54 testserver kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.13.2.el8_2.x86_64

>> journal log also cannot show graceful shutdown/reboot log since journal normally only record log in /run/log/journal and will dispeared once reboot
# journalctl 
-- Logs begin at Mon 2023-08-14 05:58:54 EDT, end at Mon 2023-08-14 07:29:01 EDT. --
8月 14 05:58:54 testserver kernel: Linux version 4.18.0-193.13.2.el8_2.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (>
8月 14 05:58:54 testserver kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.13.2.el8_2.x86_64 root=/dev/mapper/rhel_vm250--84-root ro cras>


In RHEL8:
 -if We let journal log save to disk,  then /var/log/messages can record the log which from journal log(save to local such as /var/log/journal
~~~
# mkdir -p /var/log/journal
# systemctl restart systemd-journald.service
# systemctl reboot
~~~
Then from /var/log/message, we know it is a graceful shutdown/reboot
~~~
Jul 31 01:30:23 testserver systemd-logind[974]: System is rebooting.
…
Jul 31 01:30:23 testserver sssd[nss][963]: Shutting down
…
Jul 31 01:30:23 testserver NetworkManager[987]: <info>  [1690781423.5682] caught SIGTERM, shutting down normally.
…
Jul 31 01:30:24 testserver systemd[1]: Shutting down.
~~~
 -if we let journal log keep default setting（save in ring-buffer in /run/log/journal），then /var/log/messages can record the log during the system reboot/shutdown, then rsyslog can not

Hi,

On a clean VM, I can not see a difference in the /var/log/messages file. I've tried RHEL7, RHEL8, RHEL9 as well with the default configurations. I could take a deeper look on the problem if we had a reproducer, however:
If you want to ensure that journal data should be stored persistently, and don't want to create the appropriate folders manually every time after boot, then I suggest to use the Storage="persistent" journald configuration option in /etc/systemd/journald.conf.

From the journald.conf manual page:
Storage=Controls where to store journal data.If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable. The "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes. Defaults to "auto".

Hi Attila,
    Sorry late to reply to you.
    Let me make more clarify the issue in this BUG.

    When we reboot/shutdown the server normally， the system will receives 'signal 15'.

    - In RHEL7 system (default configuration) will record the information of 'signal 15' or 'shutdown' or 'critical service stop' similar information in the messages log, then we can easy judge the system was reboot/shutdown gracefully through the messages.  see below messages from reboot command, we then know the server reboot/shutdown caused by gracefully shutdown.
~~~
Aug  9 16:16:48 rhel7 systemd: Stopped Dump dmesg to /var/log/dmesg.
Aug  9 16:16:48 rhel7 systemd: Stopped target RPC Port Mapper.
Aug  9 16:16:48 rhel7 systemd: Stopped target Timers.
Aug  9 16:16:48 rhel7 systemd: Stopping RPC bind service...
Aug  9 16:16:48 rhel7 systemd: Stopped target Multi-User System.        <--------
Aug  9 16:16:48 rhel7 systemd: Stopping OpenSSH server daemon...
Aug  9 16:16:48 rhel7 systemd: Stopped Resets System Activity Logs.
Aug  9 16:16:48 rhel7 systemd: Stopping Command Scheduler...
Aug  9 16:16:48 rhel7 systemd: Unmounting RPC Pipe File System...
Aug  9 16:16:48 rhel7 systemd: Stopping Authorization Manager...
Aug  9 16:16:48 rhel7 systemd: Stopped Daily Cleanup of Temporary Directories.
Aug  9 16:16:48 rhel7 systemd: Stopping LSB: Starts the Spacewalk Daemon...        <--------
Aug  9 16:16:48 rhel7 systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
Aug  9 16:16:48 rhel7 systemd: Stopping Enable periodic update of entitlement certificates....
Aug  9 16:16:48 rhel7 systemd: Stopping irqbalance daemon...         <--------
...
Aug  9 16:17:19 rhel7 kernel: Linux version 3.10.0-1160.95.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Fri Jun 23 08:44:55 EDT 2023
~~~

    - But in current RHEL8 system (default configuration) does not record 'signal 15' or 'shutdown' or 'critical service stop' similar information in the messages log, so in RHEL8, we cannot judge the system belong to graceful shutdown through the messages. See below messages from reboot command, we haven't found out any 'signal 15' or 'shutdown' or 'critical service stop' info in our messages log, then we cannot judge the system was reboot/shutdown gracefully through the messages.
~~~
Aug 14 05:50:01 testserver systemd[1]: Starting system activity accounting tool...
Aug 14 05:50:01 testserver systemd[1]: sysstat-collect.service: Succeeded.
Aug 14 05:50:01 testserver systemd[1]: Started system activity accounting tool.    <———— initial reboot command
Aug 14 05:58:54 testserver kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.13.2.el8_2.x86_64
~~~

     As a workaround in RHEL8, we can configure journal to store the log in disk, so that the messages log will record specific 'signal 15' or 'shutdown' or 'critical service stop'. But this workaround is not the default configuration of the system journald.conf.  the current requirement is that by default, RHEL8 can record 'signal 15' or 'shutdown' or 'critical service stop' in messages same with RHEL7 when rebooting/shutdown, so that the system can be judged directly it as gracefully shutdown in RHEL8 through the messages. 
######
~~~
# mkdir -p /var/log/journal
# systemctl restart systemd-journald.service
# systemctl reboot
~~~
Then from /var/log/message, we know it is a graceful shutdown/reboot
~~~
Jul 31 01:30:23 testserver systemd-logind[974]: System is rebooting.
…
Jul 31 01:30:23 testserver sssd[nss][963]: Shutting down
…
Jul 31 01:30:23 testserver NetworkManager[987]: <info>  [1690781423.5682] caught SIGTERM, shutting down normally.
…
Jul 31 01:30:24 testserver systemd[1]: Shutting down.
~~~
######

Hello,

When migrating between major versions of RHEL, there can quite big changes in terms of default values. Let me clarify the difference between these two. In general, Journal events can be transferred to a different logging daemon(e.g. rsyslog) in two different ways:

1. On RHEL7, messages are immediately forwarded to a socket (/run/systemd/journal/syslog), where the traditional syslog daemon can read them. This method is controlled by the 'ForwardToSyslog=' journald option. These messages are read almost immediately [1].

2. However on RHEL8+, the syslog daemon behaves like a normal journal client, and reads messages from the journal files, similarly to journalctl(1). With this, messages do not have to be read immediately, which allows a logging daemon which is only started late in boot to access all messages since the start of the system. In addition, full structured meta-data is available to it. This method of course is available only if the messages are stored in a journal file at all. So it will not work if no Storage is set. It should be noted that usually the second method is used by syslog daemons, so the Storage= option, and not the ForwardToSyslog= option, is relevant for them [1].

Because the default value of ForwardToSyslog= has changed between the two major versions of RHEL, in rsyslog we need to use the journal API (similarly to journalctl) to read the journal. That's why you might experience 'message loss' in rsyslog when shutting down a system. However, this is not message loss, rather it's expected to work like this. If you want to store journal logs during shutdown/reboot, please set the Storage="persistent" option.

Please note that even on RHEL7 with the legacy behavior, not all the "Stopping target XXX" messages are written to /var/log/messages. Try to create the /var/log/journal folder and you will see that there are many others. This is because rsyslog needs to process those messages immediately but that's not always possible, because eventually rsyslog will be shut down as well. So there is a race.

Feel free to ask other questions if needed.
[1] man journald.conf

Attila

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.