I wonder if this is a mileage may vary scenario? For example, I provisioned a clean RHEL 8.6 system and installed 6.11 and I have a /var/log/messages present with the logs pre and post reboot.

(In reply to Eric Helms from comment #2)
> I wonder if this is a mileage may vary scenario? For example, I provisioned
> a clean RHEL 8.6 system and installed 6.11 and I have a /var/log/messages
> present with the logs pre and post reboot.

Nice finding.

I am double-checking how I installed my RHEL8.6 . It apparently lacks rsyslog package - while default installation of that package does configure into /etc/rsyslog.conf:

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

If we want to enforce persistent logs from a Satellite perspective versus how the OS is configured, we could make rsyslog a dependency of the satellite and satellite-capsule meta packages.

Hmm... I installed 8.6 GA from corporate PXE, once tried "Minimal Install" and once the default "Server" - both times, rsyslog package is missing (and BZ reproduced). Anyway I would appreciate if somebody else can confirm my observation (to prove I am not insane :) ).

Our installer doc mentions just @base package group, not sure if that does contain rsyslog or not - but till the very default installation ends up with no rsyslog, it is easy to hit this problem.

Having rsyslog as dependency of satellite and satellite-capsule sounds a great idea to me (isnt there yet better *one* similar package for both Sat and Caps?).

On RHEL7, it seems that even Minimal installation does contain rsyslog package - at least I do see it present on RHEL7.9 minimal install.

Anyway somebody from Anaconda should ideally confirm those findings (rsyslog in RHEL7 minimal but not in RHEL8 Server or Minimal) - or somebody else shall independently on me double-check my findings.

FWIW, the RHEL 8.6 KVM Guest Image distributed in the Customer Portal has rsyslog installed.

The KCS https://access.redhat.com/solutions/696893 would indicate that the proper way to enable persistent logging is:


# sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
# systemctl restart systemd-journald.service


This would appear to be the more direct, proper solution but it has the downside of requiring system configuration versus just installing a package. That being said, if this is something we desire to have as the base configuration for a system running Satellite or a Capsule we could handle this through the installer.

@ekohlvan -- mind sharing your thoughts on solution path?

(In reply to Eric Helms from comment #9)
> The KCS https://access.redhat.com/solutions/696893 would indicate that the
> proper way to enable persistent logging is:
> 
> # sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
> # systemctl restart systemd-journald.service

Installer route: add this to custom-hiera.yaml

systemd::journald_settings:
  Storage: persistent

See https://forge.puppet.com/modules/puppet/systemd#journald-configuration

> This would appear to be the more direct, proper solution but it has the
> downside of requiring system configuration versus just installing a package.
> That being said, if this is something we desire to have as the base
> configuration for a system running Satellite or a Capsule we could handle
> this through the installer.
> 
> @ekohlvan -- mind sharing your thoughts on solution path?

Until now we've always avoided managing the system in this way. systemd-journald and (r)syslog were always up to the admin. You could make the argument that the application works fine without persistent logging and it gets into the area of monitoring.

A small side step: we have https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html/administering_red_hat_satellite/logging_and_reporting_problems_admin#Configuring_Logging_to_Journal_admin today and I recall there was also a KCS about shipping those logs. Back then we also had some debate over whether we should or shouldn't manage the logging stack.

One thing I have found with systemd-journald is that by default it doesn't have any maximum retention and it does get large. On my system with spinning disks this is noticeably slow and I have set up retention. The only question is, what should that retention be?

If a simple dnf install rsyslog is sufficient for persistent logging, I'd say adding rsyslog as a dependency is the safest route and it's what a non-minimal install sets up anyway so the impact is limited for most users.

VERIFIED.

@Satellite 6.12.0 Snap13
satellite-common-6.12.0-3.el8sat.noarch

by this very simple step:

# rpm -qR satellite-common | grep log
rsyslog

>>> rsyslog requirement is correctly expressed as a rpm dependency (for both Satellite and Capsule)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506