Bug 2050556

Summary: RFE: Harden the shutdown phase to avoid dropping into the emergency prompt
Product: Red Hat Enterprise Linux 9 Reporter: David Tardon <dtardon>
Component: dracutAssignee: Lukáš Nykrýn <lnykryn>
Status: CLOSED ERRATA QA Contact: Vojtech Eichler <veichler>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: dracut-maint-list, dtardon, jikortus, pvalena, veichler
Target Milestone: rcKeywords: FeatureBackport, FutureFeature, Reproducer, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: dracut-055-30.git20220216.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 16:04:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Tardon 2022-02-04 09:14:26 UTC
This bug was initially created as a copy of Bug #1924587

I am copying this bug because: 
It still applies.


Description of problem:

Currently the shutdown phase executes dracut-shutdown.service which unpacks the initramfs into /run/initramfs. Then plymouth-switch-root-initramfs.service executes and switches root to /run/initramfs and final shutdown operations execute.

Sometimes, the initramfs couldn't be unpacked properly, which leads to dropping into the following prompt (executed by "shutdown" binary once switch-root happened):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
/shutdown: line 115: reboot: command not found
[  145.951127] dracut Warning: reboot failed!
dracut Warning: reboot failed!


[  145.953007] dracut Warning: 
dracut Warning: 


Generating "/run/initramfs/rdsosreport.txt"
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

To get more debug information in the report,
reboot with "rd.debug" added to the kernel command line.

Dropping to debug shell.

shutdown:/# 
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

So far we got some customers hitting this or something similar (e.g. /usr/bin/multipath: command no found) in different scenarios:

1. Upon installation with a virtual DVD mounted on the iLO with low bandwidth between the ISO image and the iLO

  In this scenario, the issue is due to low bandwidth: dracut-shutdown has only 90 seconds to complete and this may not be enough, causing systemd to kill the cpio command executing and having missing binaries in /run/initramfs, causing the issue after switching root

2. Upon restarting VMWare guests: it's unclear which condition effectively happens here but it's very likely related to unpacking being killed as well on 90 seconds timeout


Version-Release number of selected component (if applicable):

dracut-049-95.git20200804.el8.x86_64 / plymouth-0.9.4-7.20200615git1e36e30.el8.x86_64


How reproducible:

ALWAYS with a "slow" DVD during installation
ALWAYS using the hack below

Steps to Reproduce:
1. Edit /usr/lib/dracut/dracut-initramfs-restore to delete "reboot" binary after extraction happens

Replace "exit 0" at the end of the script by this:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
rm /run/initramfs/usr/sbin/reboot
exit 1
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This mimics a timeout while cpio is running (binaries in /usr/sbin are the last ones being extracted).

2. Reboot the system

Actual results:

Emergency prompt

Expected results:

Automatic reboot if possible

Comment 3 Pavel Valena 2022-02-15 20:29:26 UTC
Reviewed and pre-verified in the PR: https://github.com/redhat-plumbers/dracut-rhel9/pull/23

Comment 13 errata-xmlrpc 2022-05-17 16:04:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: dracut), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4070