1. Please describe the problem: I just upgraded to Fedora 42 Beta and with 6.14.0-0.rc7.56.fc42.x86_64 and any USB device plugged into a USB Hub no longer works. 2. What is the Version-Release number of the kernel: 6.14.0-0.rc7.56.fc42.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 6.14.0-0.rc7.56.fc42.x86_64 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Boot into 6.14.0-0.rc7.56.fc42.x86_64 and run lsusb and notice that any USB devices that are plugged into a USB hub are not present. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: I just installed kernel-0:6.14.0-0.rc7.20250318git76b6905c11fd.57.fc43.x86_64 and I will reboot in a few and report back. 6. Are you running any modules that not shipped with directly Fedora's kernel?: Nvidia akmod from RPMFusion 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Looking at the dmesg from 6.14.0-0.rc7.56.fc42.x86_64 these xhci_hcd module failures might be related. Mar 20 19:07:29 kernel: ahci 0000:26:00.0: enabling device (0000 -> 0001) Mar 20 19:07:29 kernel: ahci 0000:26:00.0: probe with driver ahci failed with error -12 Mar 20 19:07:29 kernel: ahci 0000:27:00.0: enabling device (0000 -> 0001) Mar 20 19:07:29 kernel: ahci 0000:27:00.0: probe with driver ahci failed with error -12 Mar 20 19:07:29 kernel: xhci_hcd 0000:23:00.0: init 0000:23:00.0 fail, -16 Mar 20 19:07:29 kernel: xhci_hcd 0000:23:00.0: probe with driver xhci_hcd failed with error -16 Mar 20 19:07:29 kernel: xhci_hcd 0000:28:00.1: init 0000:28:00.1 fail, -16 Mar 20 19:07:29 kernel: xhci_hcd 0000:28:00.1: probe with driver xhci_hcd failed with error -16 Mar 20 19:07:29 kernel: xhci_hcd 0000:28:00.3: init 0000:28:00.3 fail, -16 Mar 20 19:07:29 kernel: xhci_hcd 0000:28:00.3: probe with driver xhci_hcd failed with error -16 Reproducible: Always
Created attachment 2081178 [details] 6.13.7-200.fc41.x86_64 dmesg with working USB
Created attachment 2081188 [details] 6.14.0-0.rc7.56.fc42.x86_64 dmesg witih busted USB
Same results with 6.14.0-0.rc7.20250318git76b6905c11fd.57.fc43.x86_64.
Created attachment 2081191 [details] 6.14.0-0.rc7.20250318git76b6905c11fd.57.fc43.x86_64 dmesg with busted USB
Can you provide details of the actual HW, what's the make/model of the computer, what is the chip, what is the USB HUB. What sort of port is the hub plugged into (is it a A port or a Type-C, if the later is the Type-C part of the GPU). Is the USB port usb2 or usb3, what rev is the hub etc.
My workstation is a custom build with an Asus Pro WS WRX80E-SAGE SE WIFI motherboard which has 8 x USB 3.2 Gen 2 Type-A 1 x USB 3.2 Gen 2 Type-C port 1 x USB 3.2 Gen 2x2 Type-C port The USB Hub is an A8309 Anker 4 port with a USB C connector. It is plugged into the one of the back USB C ports of the motherboard. Also my Fractal Design Define 7 XL case has 1x USB Type-C and 2x USB 3 and 2x USB 2 ports on the front of it which are plugged into the USB 3.2 Gen 1 connector header on the motherboard. I am not sure what kind of hub this is to be honest but it also does not work.
Here's some lspci info running on 6.13.7-200.fc41.x86_64 $ lspci -nn |grep USB 06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] 23:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller [1b21:3242] 28:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 28:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 2c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
Same issue here. Installed the new Kernel, rebooted and hit a Kernel panic with Tux and an exclamation mark with tiny letter that I can barely read saying that the Kernel just panic'd. Unfortunately I can not provide any output because my entire Linux System is run from an M2 drive connected through USB3 over an USB Hub from Lionwei. The HUB is connected through USB-C. I had to revert back to the previous Kernels which works flawlessly. vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working
(In reply to Ali Akcaagac from comment #8) > vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working > vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working Forget to mention, that the tiny letters said something about that not being able to boot and that it can not find the root=<UUID> partition (which is only detectable, if the part of the kernel works). UEFI works of course and detects the UEFI Partition on the M2 drive.
> Installed the new Kernel, rebooted and hit a Kernel panic with Tux and an > exclamation mark with tiny letter that I can barely read saying that the > Kernel just panic'd. If you add drm.panic_screen=kmsg to the kernel cmdline in grub you can get the full crash output to get you debug there. > vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working > vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working Can you confirm the HW you have, is it also an AMD device?
> vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working > vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working So there were no USB changes at all in the kernel between rc7 and final. There were a bunch of AMD GPU changes though. I also got one of the Anker USB hubs as they were discounted and I had been considering getting something like it, it works on my QCom based aarch64 laptop and a Carbon X1, both on USB Type-C ports so this isn't a general issue. We're going to need more information on the USB controllers etc. You also mention in the header "USB devices plugged into USB hubs" so does a device plugged directly into the port, like a type-c ethernet/storage device work?
(In reply to Peter Robinson from comment #11) > > vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working > > vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working > > So there were no USB changes at all in the kernel between rc7 and final. Thanks for getting back to me again (even though I'm not the initial reporter of this issue). Thanks for giving the information for the kernel parameters that I used for hunting the issue. And yes I was able to solve the issue - that I initially thought to be an USB3 (USB-C) driver issue. What was it concrete? Short: It was dracut.... not generating an initrd for me... A few days ago there has been a dracut update (between the two Kernel version that I reported). The dracut update had a file called dracut-cpio in /usr/lib/dracut/ and there was no execution flag set. Long: After updating to the new Kernel versions and a bunch of failed booting attempts, I went back to the old Kernel which worked flawlessly. I went to the Fedora update page and looked at the karma points given and saw someone already opened a bug related to USB devices not found. Same what I thought happened for me. Last night I found some spare time testing properly and yet the new Kernel came back with the ASCII TUX and the exclamation mark. After I noted down the kernel parameter and entering grub, I figured out that there was something going on there. The vmlinuz line was there but the initrd line was missing inside the grub2 entry. I quickly booted back with the old Kernel and looked inside the grub.cfg file because I wanted to "cut and paste" (quick and dirty) the missing line in there with vim. Going back into the /boot directory I realized that the entire initrd for this specific Kernel was missing. I ran "dracut --xz --verbose --force" in bash and saw that dracut was trying to execute dracut-cpio but wasn't able to do so. The result was that the initrd was missing (once again). After changing dracut-cpio to 755 an re-running dracut, the initrd was written properly. Re-run grub2-mkconfig -o /boot/grub2/grub.cfg. The initrd entry (as well as the rest) was properly written (BLS_CFG = false). Reboot and everything was working properly. Maybe this "may" or "may not" help the initial bug owner. So what exactly happened was a causal chain, update dracut, update kernel days later, kernel not booting (caused by dracut .... or .... whatever caused the file dracut-cpio not to have the executable flags set, which then leads to no initrd, which leads to grub2-mkconfig writing half of the entries)... Hope this helps and thanks for the great job guys. I can understand getting through all these reports (false true, true falses) can be cumbersome at times. Regards.
dracut --xz --verbose --force --kver <the new kernel version> to be more specific.
Ok re-created the issue. It was not just the dracut.cpio, it was the ossl-config and the ossl-files, that had the execution flag missing... For whatever reasons (which I really don't know)... sudo dracut --force --verbose --xz dracut[I]: Executing: /usr/bin/dracut --force --verbose --xz dracut[I]: *** Including module: bash *** dracut[I]: *** Including module: shell-interpreter *** dracut[I]: *** Including module: systemd *** dracut[I]: *** Including module: fips *** dracut[I]: *** Including module: fips-crypto-policies *** dracut[I]: *** Including module: systemd-ask-password *** dracut[I]: *** Including module: systemd-battery-check *** dracut[I]: *** Including module: systemd-initrd *** dracut[I]: *** Including module: systemd-journald *** dracut[I]: *** Including module: systemd-modules-load *** dracut[I]: *** Including module: systemd-sysctl *** dracut[I]: *** Including module: systemd-sysusers *** dracut[I]: *** Including module: systemd-tmpfiles *** dracut[I]: *** Including module: systemd-udevd *** dracut[I]: *** Including module: nss-softokn *** dracut[I]: *** Including module: i18n *** dracut[I]: *** Including module: drm *** dracut[I]: *** Including module: plymouth *** dracut[I]: *** Including module: kernel-modules *** dracut[I]: *** Including module: kernel-modules-extra *** dracut[I]: *** Including module: rootfs-block *** dracut[I]: *** Including module: terminfo *** dracut[I]: *** Including module: udev-rules *** dracut[I]: *** Including module: dracut-systemd *** dracut[I]: *** Including module: usrmount *** dracut[I]: *** Including module: base *** dracut[I]: *** Including module: fs-lib *** dracut[I]: *** Including module: openssl *** /usr/lib/dracut/modules.d/99openssl/module-setup.sh: line 13: /usr/lib/dracut/ossl-files: Permission denied dracut[F]: '/usr/lib/dracut/ossl-files --config' does not return a path!! If you already have a matching initrd, this causes nothing. If you have a initrd missing, then the above issue won't generate one. firmware missing, modules missing, udev missing, ... Maybe components missing that may be interesting for the initial bug owner...
(In reply to Peter Robinson from comment #11) > > vmlinuz-6.14.0-0.rc7.56.fc42.x86_64 <--- Working > > vmlinuz-6.14.0-63.fc42.x86_64 <--- Not Working > > So there were no USB changes at all in the kernel between rc7 and final. > There were a bunch of AMD GPU changes though. > > I also got one of the Anker USB hubs as they were discounted and I had been > considering getting something like it, it works on my QCom based aarch64 > laptop and a Carbon X1, both on USB Type-C ports so this isn't a general > issue. We're going to need more information on the USB controllers etc. > > You also mention in the header "USB devices plugged into USB hubs" so does a > device plugged directly into the port, like a type-c ethernet/storage device > work? Yes, if I plug any of my devices directly into the back USB ports they work just fine. It is just an issue with anything plugged into a USB hub no longer shows up in lsusb or works as expected. I posted info about the controllers prior but posting again for clarity. They are AMD based controllers. $ lspci -nn |grep USB 06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] 23:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller [1b21:3242] 28:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 28:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 2c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
(In reply to Ali Akcaagac from comment #14) > Ok re-created the issue. It was not just the dracut.cpio, it was the > ossl-config and the ossl-files, that had the execution flag missing... For > whatever reasons (which I really don't know)... > *snip* > If you already have a matching initrd, this causes nothing. If you have a > initrd missing, then the above issue won't generate one. > > firmware missing, modules missing, udev missing, ... Maybe components > missing that may be interesting for the initial bug owner... I don't think this is impacting me at all. I just rebuilt my initrd and I don't have any permissions issues like you do. Maybe this is a red herring?
This issue is still happening with the 6.14.2-300.fc42.x86_64 kernel. $ uname -a Linux hadron 6.14.2-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 21:50:55 UTC 2025 x86_64 GNU/Linux $ lsusb Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 1532:0084 Razer USA, Ltd RZ01-0321 Gaming Mouse [DeathAdder V2] Bus 001 Device 003: ID 3434:0933 Keychron Keychron V3 Max Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 002: ID 21b4:0143 AudioQuest m9XX Bus 003 Device 003: ID 05ba:000a DigitalPersona, Inc. Fingerprint Reader Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Created attachment 2085093 [details] Good boot from sfalco machine
Created attachment 2085094 [details] Bad boot from sfalco machine
I'm having a similar issue. I just upgraded to f42 and my keyboard doesn't work with the 6.14.2-300.fc42 kernel. Stepping back to the 6.13.10-200.fc41 kernel everything works fine. I've attached two files: good.txt is using the 6.13.10-200.fc41 kernel; bad.txt is using the 6.14.2-300.fc42 kernel. There are quite a few missing items in bad.txt, including the keyboard. In particular, my keyboard has USB ID 0853:0100 and it only shows up in good.txt. My mouse has USB ID 046d:c051 and it shows up in both logs, and it works properly. I'll be happy to provide additional information. Here are a few hw details: Mobo: ASUS Pro WS WRX80E-SAGE SE WIFI II sWRX8 E-ATX Motherboard KB: HHKB Pro 2 Mouse: Logitech MX518 GUI: XFCE Two monitors, one ASUS and one HP, both connected over DP.
(In reply to Steven A. Falco from comment #20) > Mobo: ASUS Pro WS WRX80E-SAGE SE WIFI II sWRX8 E-ATX Motherboard We have almost identical motherboards. Mine is the ASUS Pro WS WRX80E-SAGE SE WIFI and they both use the AMD WRX80 Chipset.
Upstream BZ: https://bugzilla.kernel.org/show_bug.cgi?id=220016
I am now able to build a working 6.14 kernel. The only thing I had to change was to turn off CONFIG_PCI_REALLOC_ENABLE_AUTO. With CONFIG_PCI_REALLOC_ENABLE_AUTO off, all my USB controllers work. With CONFIG_PCI_REALLOC_ENABLE_AUTO=y, some of my USB controllers don't initialize.
Does just booting with pci=realloc=off help?
(In reply to Justin M. Forbes from comment #24) > Does just booting with pci=realloc=off help? Yep, it helps. Everything is back to normal now.
Setting pci=realloc=off works for me also. I started off by using https://gitlab.com/cki-project/kernel-ark.git to attempt to bisect the problem. While doing that I noticed that turning off the CONFIG_PCI_REALLOC_ENABLE_AUTO feature was the issue for me. Is that repo the correct one to use for bisection?
Yes, that is the correct repo.
Thanks. The question now is whether the root cause is a kernel bug or a motherboard bug or simply that this a very specialized option that shouldn't be enabled by default. According to MR 3622 and commit 510aeb84d99 this was enabled because: "This should help with automatically allocating resources for hotplug PCI devices (e.g. U.2 NVMe) when the BIOS/firmware hasn't made provisions for that." Not having a working keyboard or mouse is pretty tough to deal with, especially for a new user, while needing hot-plugged PCI support is likely a fairly niche situation, so I personally think it would be best to go back to the F41 setting.
FEDORA-2025-309c37bb06 (kernel-6.14.5-100.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2025-309c37bb06
FEDORA-2025-8bc830e072 (kernel-6.14.5-200.fc41) has been submitted as an update to Fedora 41. https://bodhi.fedoraproject.org/updates/FEDORA-2025-8bc830e072
FEDORA-2025-c5a31cf649 (kernel-6.14.5-300.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-c5a31cf649
FEDORA-2025-c5a31cf649 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-c5a31cf649` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-c5a31cf649 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-8bc830e072 has been pushed to the Fedora 41 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-8bc830e072` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-8bc830e072 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-309c37bb06 has been pushed to the Fedora 40 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-309c37bb06` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-309c37bb06 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-c5a31cf649 (kernel-6.14.5-300.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-8bc830e072 (kernel-6.14.5-200.fc41) has been pushed to the Fedora 41 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2025-309c37bb06 (kernel-6.14.5-100.fc40) has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report.