Bug 2172912
Summary: | Broken /dev/log socket created during boot in recovery, causing grub2-mkconfig to hang forever | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Renaud Métrich <rmetrich> |
Component: | rear | Assignee: | Pavel Cahyna <pcahyna> |
Status: | CLOSED ERRATA | QA Contact: | Jakub Haruda <jharuda> |
Severity: | high | Docs Contact: | Šárka Jana <sjanderk> |
Priority: | high | ||
Version: | 9.1 | CC: | jharuda, pcahyna |
Target Milestone: | rc | Keywords: | Triaged |
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | rear-2.6-18.el9 | Doc Type: | Bug Fix |
Doc Text: |
.The `rsyslog` logging service now starts at boot of the rescue system
Previously, the `rsyslog` service for message logging did not automatically start in the rescue system. The `/dev/log` socket kept receiving messages during the recovery process with no service listening at this socket. Consequently, the `/dev/log` socket was filled with messages and caused the recovery process to be stuck. For example, the `grub2-mkconfig` command to regenerate the GRUB configuration produces a high amount of log messages depending on the number of mounted file systems. If you used ReaR to recover systems with many mounted file systems, numerous log messages would fill the `/dev/log` socket, and the recovery process froze.
With this fix, the `systemd` units in the rescue system now include the sockets target in the boot procedure to start the logging socket at boot. As a result, the `rsyslog` service starts in the rescue environment when required, and the processes that need to log messages during recovery are no longer stuck. The recovery process completes successfully and you can find the log messages in the `/var/log/messages` file in the rescue RAM disk.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2023-11-07 08:37:21 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Renaud Métrich
2023-02-23 13:48:43 UTC
(In reply to Renaud Métrich from comment #0) > Description of problem: > > With RHEL9, the /dev/log inode is supposed to be a symlink to > /run/systemd/journal/dev-log. Thank you for the analysis. Is it a new problem in RHEL 9, or has it existed in RHEL 8 as well? I see a similar situation in RHEL 8: # ls -l /dev/log lrwxrwxrwx. 1 root root 28 Feb 22 04:16 /dev/log -> /run/systemd/journal/dev-log I don't know if this affects RHEL8. For sure the good inode is: # ls -l /dev/log lrwxrwxrwx. 1 root root 28 Feb 22 04:16 /dev/log -> /run/systemd/journal/dev-log I am curious though how does having correct systemd unit outside the chroot help the program running in the chroot? Is it because /run is shared so that connecting to /run/systemd/journal/dev-log in the chroot actually connects to the daemon that runs outside? It's because /dev/log outside the chroot is broken, causing /dev/log inside the chroot to be broken as well since it's a bind mount Hi Renaud, thank you for the analysis again, I have looked into the details of systemd units startup in the rescue system. IMO, your proposed workaround (to copy all the systemd logging-related units) is not very well suitable for inclusion in upstream, as ReaR needs to support many distros and these details will vary among them. At least, it would require lots of difficult testing in all the supported distros. Therefore, I propose a less invasive solution. I found that there are multiple problems with the current systemd units: nothing wants basic.target and therefore the services/sockets that it contains get never started (this affect the /dev/log socket and the rsyslogd service that is listening on it). Moreover, if I fix this, the socket starts very early and for some reason this does not work. If I order it after basic system initialization, everything starts working. The socket gets started, when one attempts to log to it rsyslogd is spawned and sends the messages to /var/log/messages. (/dev/log is not a symlink to /run/systemd/journal/dev-log, but I don't think it is a big problem). By the way, I can reproduce the problem as well using a simple for loop: for i in `seq 1 1000`; do echo foo$i; done this hangs when the problem occur, because the socket gets filled. Wit my fixes to the systemd units, it is fine, the output goies to /var/log/messages. I can also see the output from grub2-mkconfig (actually, from os-prober) there. So the problem you are seeing should be fixed. The changes are on my branch: https://github.com/pcahyna/rear/tree/rsyslog . What do you think? Regarding RHEL 8, I see that the logs go into the systemd journal by default, so it seems that the problem does not occur there and so I won't touch it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (rear bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6571 |