When booting the RHCOS live ISO image from a USB stick, boot sometimes fails with the following error: [ 9.410095] systemd[1]: Reached target Initrd Root Device. [ 9.415226] systemd[1]: Mounting /run/media/iso... [ 9.498191] ISOFS: Unable to identify CD-ROM format. [ 9.499552] mount[722]: mount: /run/media/iso: wrong fs type, bad option, bad superblock on /dev/sda2, missing codepage or helper program, or other error. Failed to mount /run/media/iso. See 'systemctl status run-media-iso.mount' for details. 35coreos-live/live-generator creates run-media-iso.mount, which mounts the ISO from /dev/disk/by-label/rhcos-<version>. The latter should point to /dev/sda1, but in this case it's pointing to /dev/sda2, which is efiboot.img. /dev/sda2 does not have a filesystem label, but because of https://github.com/systemd/systemd/issues/14408, 60-persistent-storage.rules falls back to using the filesystem label from /dev/sda (the whole-disk device) when evaluating symlinks for /dev/sda2. As a result, /dev/sda1 and /dev/sda2 race to own the /dev/disk/by-label symlink, and if sda2 wins, the mount fails. The race occurs on BIOS, UEFI El Torito, and UEFI GPT boots; kola testiso iso-as-disk has reproduced this on both BIOS and UEFI. It does not affect CD or virtual CD boot because CD-ROM devices are not partitionable. It does not affect Fedora CoreOS because the udev rules on FCOS include https://github.com/systemd/systemd/pull/14485. The problem affects RHCOS 4.7 and up, but not the existing 4.6 ISO because the hybrid partition table in that release does not include a partition for efiboot.img. However, the fix for bug 2004679 (https://github.com/coreos/coreos-assembler/pull/2439) backports the hybrid ESP to 4.6, so currently the next bootimage bump will cause 4.6 to regress. To verify a fix for this bug, boot with a `udev.log_priority=debug` karg and ensure `journalctl -u systemd-udevd.service` has no mention of a `/dev/disk/by-label/rhcos-*` symlink being applied to `/dev/vda2` or `/dev/sda2`.
*** Bug 1996882 has been marked as a duplicate of this bug. ***
Fixed in 410.84.202109241829-0 on x86_64, and other arches are not affected.
This bug has been reported fixed in a new RHCOS build. Do not move this bug to MODIFIED until the fix has landed in a new bootimage.
PreVerified on rhcos 410.84.202110081440-0 Downloads live ISO and wrote to USB drive with `dd if=rhcos-410.84.202110081440-0-live.x86_64.iso of=/dev/sdb oflag=sync status=progress` Then booted USB drive and edited the kernel arguments to include `udev.log_priority=debug` `journalctl -u systemd-udevd.service --no-pager | grep rhcos` returned no symlinks pointing to /dev/sdb2 or /dev/vdb2
The fix for this bug has landed in a bootimage bump, as tracked in bug 2004596 (now in status MODIFIED). Moving this bug to MODIFIED.
Moving to verified as the boot image has been landed and verified on https://bugzilla.redhat.com/show_bug.cgi?id=2004596
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056