Description of problem: pulp(*) logs to rsyslog, which usually means to /var/log/messages and journalctl. BUT on default installation of Sat6.11 on RHEL8: - there is no /var/log/messages - journal logs are *transient*, which means a reboot purges all its logs (*) pulp among others, but this is the most important component, I would say) Therefore, rebooting such Satellite means loosing *all* pulp logs. That would limit support engineers from investigating customer issues in scenarios like: - customer hitting a problem (say swapping) - they reboot the OS as an immediate intervention - then they provide sosreport or any other data TTL;DR: we need default installations of Sat6.11+ on RHEL8 to permanently keep (pulp) logs - either in /var/log/messages or changing journal logs to permanent Version-Release number of selected component (if applicable): 6.11 on RHEL8 How reproducible: 100% Steps to Reproduce: 1. Do a bad thing to pulp 2. Check the bad thing in logs (journalctl --since=today) 3. Reboot the OS 4. Check the bad thing in the logs again Actual results: 4. Logs are gone. Expected results: 4. Logs are kept (optionally with some logrotation / housekeeping) Additional info:
I wonder if this is a mileage may vary scenario? For example, I provisioned a clean RHEL 8.6 system and installed 6.11 and I have a /var/log/messages present with the logs pre and post reboot.
(In reply to Eric Helms from comment #2) > I wonder if this is a mileage may vary scenario? For example, I provisioned > a clean RHEL 8.6 system and installed 6.11 and I have a /var/log/messages > present with the logs pre and post reboot. Nice finding. I am double-checking how I installed my RHEL8.6 . It apparently lacks rsyslog package - while default installation of that package does configure into /etc/rsyslog.conf: *.info;mail.none;authpriv.none;cron.none /var/log/messages
If we want to enforce persistent logs from a Satellite perspective versus how the OS is configured, we could make rsyslog a dependency of the satellite and satellite-capsule meta packages.
Hmm... I installed 8.6 GA from corporate PXE, once tried "Minimal Install" and once the default "Server" - both times, rsyslog package is missing (and BZ reproduced). Anyway I would appreciate if somebody else can confirm my observation (to prove I am not insane :) ). Our installer doc mentions just @base package group, not sure if that does contain rsyslog or not - but till the very default installation ends up with no rsyslog, it is easy to hit this problem. Having rsyslog as dependency of satellite and satellite-capsule sounds a great idea to me (isnt there yet better *one* similar package for both Sat and Caps?).
On RHEL7, it seems that even Minimal installation does contain rsyslog package - at least I do see it present on RHEL7.9 minimal install. Anyway somebody from Anaconda should ideally confirm those findings (rsyslog in RHEL7 minimal but not in RHEL8 Server or Minimal) - or somebody else shall independently on me double-check my findings.
FWIW, the RHEL 8.6 KVM Guest Image distributed in the Customer Portal has rsyslog installed.
The KCS https://access.redhat.com/solutions/696893 would indicate that the proper way to enable persistent logging is: # sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf # systemctl restart systemd-journald.service This would appear to be the more direct, proper solution but it has the downside of requiring system configuration versus just installing a package. That being said, if this is something we desire to have as the base configuration for a system running Satellite or a Capsule we could handle this through the installer. @ekohlvan -- mind sharing your thoughts on solution path?
(In reply to Eric Helms from comment #9) > The KCS https://access.redhat.com/solutions/696893 would indicate that the > proper way to enable persistent logging is: > > # sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf > # systemctl restart systemd-journald.service Installer route: add this to custom-hiera.yaml systemd::journald_settings: Storage: persistent See https://forge.puppet.com/modules/puppet/systemd#journald-configuration > This would appear to be the more direct, proper solution but it has the > downside of requiring system configuration versus just installing a package. > That being said, if this is something we desire to have as the base > configuration for a system running Satellite or a Capsule we could handle > this through the installer. > > @ekohlvan -- mind sharing your thoughts on solution path? Until now we've always avoided managing the system in this way. systemd-journald and (r)syslog were always up to the admin. You could make the argument that the application works fine without persistent logging and it gets into the area of monitoring. A small side step: we have https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html/administering_red_hat_satellite/logging_and_reporting_problems_admin#Configuring_Logging_to_Journal_admin today and I recall there was also a KCS about shipping those logs. Back then we also had some debate over whether we should or shouldn't manage the logging stack. One thing I have found with systemd-journald is that by default it doesn't have any maximum retention and it does get large. On my system with spinning disks this is noticeably slow and I have set up retention. The only question is, what should that retention be? If a simple dnf install rsyslog is sufficient for persistent logging, I'd say adding rsyslog as a dependency is the safest route and it's what a non-minimal install sets up anyway so the impact is limited for most users.
VERIFIED. @Satellite 6.12.0 Snap13 satellite-common-6.12.0-3.el8sat.noarch by this very simple step: # rpm -qR satellite-common | grep log rsyslog >>> rsyslog requirement is correctly expressed as a rpm dependency (for both Satellite and Capsule)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:8506