Fedora Account System
Red Hat Associate
Red Hat Customer
1. Please describe the problem: Since Fedora-Rawhide-20260409.n.0 , where we went from kernel-7.0.0-0.rc7.260408g3036cd0d3328.56.fc45 to kernel-7.0.0-0.rc7.55.fc45 , all openQA USB boot tests fail on Rawhide. Boot fails in dracut, claiming that /dev/root does not exist. openQA emulates USB boot via qemu, this is the full qemu command: /usr/bin/qemu-system-x86_64 -device virtio-vga,edid=on,xres=1024,yres=768 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 4096 -machine q35,smm=on -cpu host -netdev user,id=qanet0,net=172.16.2.0/24 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 -device qemu-xhci -device usb-tablet -smp 2 -enable-kvm -no-shutdown -vnc :113,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console_user,path=virtio_console_user,logfile=virtio_console_user.log,logappend=on -device virtconsole,chardev=virtio_console_user,name=org.openqa.console.virtio_console_user -chardev socket,path=qmp_socket,server=on,wait=off,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -global driver=cfi.pflash01,property=secure,value=on -device virtio-scsi-pci,id=scsi0 -device qemu-xhci,id=xhci0 -blockdev driver=file,node-name=hd0-file,filename=/var/lib/openqa/pool/23/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on,discard=unmap -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=usbstick-overlay0-file,filename=/var/lib/openqa/pool/23/raid/usbstick-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=usbstick-overlay0,file=usbstick-overlay0-file,cache.no-flush=on,discard=unmap -device usb-storage,id=usbstick-device,drive=usbstick-overlay0,bootindex=0 -drive id=pflash-code-overlay0,if=pflash,file=/var/lib/openqa/pool/23/raid/pflash-code-overlay0,unit=0,readonly=on -drive id=pflash-vars-overlay0,if=pflash,file=/var/lib/openqa/pool/23/raid/pflash-vars-overlay0,unit=1 2. What is the Version-Release number of the kernel: kernel-7.0.0-0.rc7.55.fc45 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Yes, see above. Good: kernel-7.0.0-0.rc7.260408g3036cd0d3328.56.fc45 . Bad: kernel-7.0.0-0.rc7.55.fc45 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: openQA reproduces it multiple times a day. I haven't had time to try locally yet. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Yes. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No. 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Will see what I can get out of the dracut environment tomorrow, wanted to file this before I forgot.
Hmm, might be more than just USB, some other things stopped working in similar ways at the same time, like LVM thinp boot: https://openqa.fedoraproject.org/tests/4562805#step/_console_wait_login/5 and software RAID boot: https://openqa.fedoraproject.org/tests/4562753#step/_console_wait_login/11 also ARM disk image boot started failing at the same time which is likely also the same thing, though we have no video output at this point in boot so can't tell for sure: https://openqa.fedoraproject.org/tests/4581771#step/_console_wait_login/8
I'm seeing the same on hardware; here's an rdsosreport from a user: https://paste.centos.org/view/e7e75e5f
As an aside (it was a red herring in my case) `multipathd` has failed starting with that kernel as well complaining about the kernel module for it not being loaded.
jcline suggests this may be the same as https://bugzilla.redhat.com/show_bug.cgi?id=2457183 , that seems plausible. https://bodhi.fedoraproject.org/updates/FEDORA-2026-16d5de8bc9 reverts the glibc change but failed its own gating tests, I'm trying to fix its tests now - https://src.fedoraproject.org/rpms/glibc/pull-request/151
Looks like this was indeed that bug; it's fixed in the newest Rawhide compose with the updated glibc. *** This bug has been marked as a duplicate of bug 2457183 ***