Backport the fix for bug 2007085 to 4.9.
Original problem statement: ``` When booting the RHCOS live ISO image from a USB stick, boot sometimes fails with the following error: [ 9.410095] systemd[1]: Reached target Initrd Root Device. [ 9.415226] systemd[1]: Mounting /run/media/iso... [ 9.498191] ISOFS: Unable to identify CD-ROM format. [ 9.499552] mount[722]: mount: /run/media/iso: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or helper program, or other error. Failed to mount /run/media/iso. See 'systemctl status run-media-iso.mount' for details. 35coreos-live/live-generator creates run-media-iso.mount, which mounts the ISO from /dev/disk/by-label/rhcos-<version>. The latter should point to /dev/sda1, but in this case it's pointing to /dev/sda2, which is efiboot.img. /dev/sda2 does not have a filesystem label, but because of https://github.com/systemd/systemd/issues/14408, 60-persistent-storage.rules falls back to using the filesystem label from /dev/sda (the whole-disk device) when evaluating symlinks for /dev/sda2. As a result, /dev/sda1 and /dev/sda2 race to own the /dev/disk/by-label symlink, and if sda2 wins, the mount fails. The race occurs on BIOS, UEFI El Torito, and UEFI GPT boots; kola testiso iso-as-disk has reproduced this on both BIOS and UEFI. It does not affect CD or virtual CD boot because CD-ROM devices are not partitionable. It does not affect Fedora CoreOS because the udev rules on FCOS include https://github.com/systemd/systemd/pull/14485. The problem affects RHCOS 4.7 and up, but not the existing 4.6 ISO because the hybrid partition table in that release does not include a partition for efiboot.img. However, the fix for bug 2004679 (https://github.com/coreos/coreos-assembler/pull/2439) backports the hybrid ESP to 4.6, so currently the next bootimage bump will cause 4.6 to regress. To verify a fix for this bug, boot with a `udev.log_priority=debug` karg and ensure `journalctl -u systemd-udevd.service` has no mention of a `/dev/disk/by-label/rhcos-*` symlink being applied to `/dev/vda2` or `/dev/sda2`. ```
Since there is a chance that a customer would not be able to boot into the RHCOS live ISO and therefore unable to install RHCOS/OCP, I am erring on the side of caution and marking this `blocker+`
Fixed in 49.84.202109241334-0 on x86_64, and the bug does not affect other arches.
This bug has been reported fixed in a new RHCOS build. Do not move this bug to MODIFIED until the fix has landed in a new bootimage.
The fix for this bug has landed in a bootimage bump, as tracked in bug 2007086 (now in status MODIFIED). Moving this bug to MODIFIED.
Verified on 49.84.202109241334-0 Downloads live ISO and wrote to USB drive with `dd if=rhcos-49.84.202109241334-0-live.x86_64.iso of=/dev/sdb oflag=sync status=progress` Then booted USB drive and edited the kernel arguments to include `udev.log_priority=debug` `journalctl -u systemd-udevd.service --no-pager | grep rhcos` returned no symlinks pointing to /dev/sda2 or /dev/vda2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759