Bug 2024169

Summary: [Regression] Network installation crashes 60% of the time due to not finding /dev/root
Product: Red Hat Enterprise Linux 8 Reporter: Renaud Métrich <rmetrich>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED WORKSFORME QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: high    
Version: 8.5CC: jstodola
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-18 09:19:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screenshot showing the crash
none
rd.debug output of a failing boot none

Description Renaud Métrich 2021-11-17 13:34:40 UTC
Created attachment 1842343 [details]
Screenshot showing the crash

Description of problem:

Trying to automatically install a QEMU/KVM over the network (PXE+HTTP) with RHEL8.5, I see that 60% of the time the switchroot to stage2 fails due to not having the /dev/root device.

In the rd.debug log (see attached), I can see that /dev/loop2 which is the "dmsetup overlay" has EXT4 errors:

[   28.178773] loop: Write error at byte offset 29511680, length 4096.
[   28.179856] loop: Write error at byte offset 29515776, length 4096.
[   28.181013] loop: Write error at byte offset 29519872, length 4096.
[   28.182207] loop: Write error at byte offset 29523968, length 4096.
[   28.184361] blk_update_request: I/O error, dev loop2, sector 57640 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   28.185819] blk_update_request: I/O error, dev loop2, sector 57648 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   28.187940] blk_update_request: I/O error, dev loop2, sector 57656 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   28.189591] blk_update_request: I/O error, dev loop2, sector 57664 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
...


Switchroot to /dev/loop2 then fails and we get the emergency prompt:

Warning: /dev/root does not exist
Entering emergency mode. Exit the shell to continue.
...


This is new with RHEL8.5.
I started 20 times the installation of the same system with RHEL8.4 and this was always a success.
With RHEL8.5, the failure happened ~60% of the time (runs 1, 2, 3, 4 and 7 were a success, all other failed similarly).


Version-Release number of selected component (if applicable):

RHEL8.5 DVD


How reproducible:

Very easily on my QEMU/KVM (BIOS mode)

Steps to Reproduce:

1. Boot over PXE

Actual results:

Crash as Switch Root to stage2


Expected results:

No crash

Comment 1 Renaud Métrich 2021-11-17 13:36:02 UTC
Created attachment 1842344 [details]
rd.debug output of a failing boot

Comment 2 Jan Stodola 2021-11-18 08:55:37 UTC
Can you please try increasing the memory to 2.5 or 3 GB? Does it solve the problem?

3 GiB of memory is the recommended minimum when installing from an http source:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_a_standard_rhel_installation/system-requirements-reference_installing-rhel#check-disk-and-memory-requirements_system-requirements-reference

Comment 3 Renaud Métrich 2021-11-18 09:19:06 UTC
Indeed, my bad, I wasn't aware so much memory was needed, and it seems to inflate with minor RHEL8 releases.
I executed 20 runs with 3GB and all seems fine.