Created attachment 1713238 [details] sudo journalctl --no-hostname -k > dmesg.txt 1. Please describe the problem: When booting the 5.8.4-200.fc32.x86_64 kernel I have lost 3 USB device ports on a DELL Precicion T5610. 2. What is the Version-Release number of the kernel: kernel-5.8.4-200.fc32.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Previous kernel-5.7.17-200.fc32.x86_64 worked fine, problem appeared after an update to kernel-5.8.4-200.fc32.x86_64. If I boot the older 5.7.17 kernel the USB ports work again. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Booting the system reproduces the issue. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: This doesn't apply, we are just past a Fedora beta branch and Rawhide is still at 5.8.0-1. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Attached.
I tried 5.8.5-200.fc32.x86_64 downloaded from Koji and it has the same problem.
Same here. I need keyboard input at initrd time to unlock my disks but no usb input devices are available. Works fine with kernel-5.7.17-200.fc32.x86_64.
(In reply to Phil from comment #2) > Same here. I need keyboard input at initrd time to unlock my disks but no > usb input devices are available. Works fine with > kernel-5.7.17-200.fc32.x86_64. What hardware are you using ? (Most people are not seeing this, so I'm trying to find a pattern in which hardware is affected)
(In reply to Hans de Goede from comment #3) > What hardware are you using ? I'm using that hardware: 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 Bus 002 Device 002: ID 8087:8001 Intel Corp. Integrated Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:8009 Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 003: ID 05e3:0749 Genesys Logic, Inc. SD Card Reader and Writer Bus 004 Device 002: ID 05e3:0612 Genesys Logic, Inc. Hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 005: ID 0781:5583 SanDisk Corp. Ultra Fit Bus 003 Device 003: ID 0582:012a Roland Corp. UM-ONE Bus 003 Device 008: ID 056e:010d Elecom Co., Ltd M-HT1DRBK HUGE Wireless Optical TrackBall Bus 003 Device 007: ID 0d8c:000e C-Media Electronics, Inc. Audio Adapter (Planet UP-100, Genius G-Talk) Bus 003 Device 006: ID 0dc6:2011 Precision Squared Technology Corp. <--- that's my keyboard Bus 003 Device 004: ID 05e3:0608 Genesys Logic, Inc. Hub Bus 003 Device 002: ID 05e3:0610 Genesys Logic, Inc. Hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
(In reply to Phil from comment #4) > (In reply to Hans de Goede from comment #3) > > What hardware are you using ? > > I'm using that hardware: Thank you, but that was not what I had in mind, what I meant is what manufacturer and product, e.g. "Dell Latitude E6400" are you using ? Problems like the one from this bug are typically specific to how a specific product uses the involved chipsets, if it was broken on such a popular chipset regardless of the product using the chipset this problem would have been caught before the new kernel hit the updates repository.
(In reply to Hans de Goede from comment #5) > Thank you, but that was not what I had in mind, what I meant is what > manufacturer and product, e.g. "Dell Latitude E6400" are you using ? Oh sorry. It's a custom desktop pc with a fairly old Gigabyte H97-D3H board w/i5-4690, using the onboard usb controller. The latest bios updates (AMI), version f7, are applied.
Hmm, so that means this is happening on 2 quite different devices, what fun. I've been looking at the kernel history to see if anything stands out, but I've not spotted anything. I see that you have both XHCI and EHCI controllers. Can you add the output of "lsusb -t" with the working kernel; as well as attach a dmesg for the working kernel ?
Created attachment 1713312 [details] dmesg for 5.7.17-200
Created attachment 1713313 [details] lsusb -t for 5.7.17-200
Created attachment 1713314 [details] dmesg for 5.8.4-200
Created attachment 1713315 [details] lsusb -t for 5.8.4-200
@Hans this needs some explaining. I booted 5.7.17 which works, and did the "lsusb -t" on the working system, but I had already moved all devices to USB ports that work in 5.8.4 (ie nothing plugged into the ports 5.8.4 doesn't like). This is the result: zuke$ uname -r 5.7.17-200.fc32.x86_64 zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 5: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M I then moved the mouse to a port that is non-functional in 5.8.4 (but works fine in 5.7.17), this is what I got: zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 5: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M I then moved the keyboard to a port that is non-functional in 5.8.4, this is what I got: zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M Should there be 2 lines that say Port 3 on Bus 03 ????? In any case this was in 5.7.17 and regardless of the odd looking listing, everything does work.
Created attachment 1713383 [details] dmesg from working 5.7.17 kernel
Phil, (In reply to Phil from comment #10) > Created attachment 1713314 [details] > dmesg for 5.8.4-200 Thank you, I don't see any missing usb-devices here vs the 5.7 kernel ? Are you sure this is right ? I guess it is possible that devices are detected but not working. You say that 3 ports stopped working, can you describe which 3 devices have stopped working ?
Ian, (In reply to Ian Laurie from comment #12) > @Hans this needs some explaining. I booted 5.7.17 which works, and did the > "lsusb -t" on the working system, but I had already moved all devices to USB > ports that work in 5.8.4 (ie nothing plugged into the ports 5.8.4 doesn't > like). This is the result: <snip> Thank you for the info. So it seems that the 5.8.4 kernel has an issue with ports connected to the xhci controller, where as the ehci ports work fine, right ? > I then moved the mouse to a port that is non-functional in 5.8.4 (but works > fine in 5.7.17), this is what I got: Oh interesting, so the mouse (and later also the kbd) does get detected, as with Phil's "lsusb -t" output, but they don't send events. So the common pattern seems to be that: 1. You both have a "9 Series Chipset Family USB xHCI Controller" 2. You both have a lo-speed (1.5M) keyboard and mouse connected to that XHCI controller 3. The kbd/mouse gets detected but does not send events. So there seems to be an issue with lo-speed USB interrupt-transfers not working on the "9 Series Chipset Family USB xHCI Controller". I will report this upstream and then we will see from there. > Should there be 2 lines that say Port 3 on Bus 03 ????? In any case this > was in 5.7.17 and regardless of the odd looking listing, everything does > work. That is normal your keyboard registers itself as 2 HID interfaces (likely one for the standard keys and one for the multimedia keys) and each interface gets its own line, if you look closely: |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 3: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M You will see yhat it is Dev (USB address) 5 which gets listed twice, with "If 0", resp. "If 1". So yeah this part of "lsusb -t"'s output is a bit weird if you are not used to it I guess. But this is normal and this way of outputting things can actually be very helpful when debugging stuff.
Hi Hans, the lsusb is from when the system has finished booting and the gdm login screen is shown. My problem is that the usb devices are missing while running from initrd. When the PC powers on, all sata disks are security-locked. I'm using a dracut module that asks for the passphrases (read -s -r tmp < /dev/console), unlocks the disks, triggers a rescan and then continues booting. With kernel 5.8.4-200 I cannot enter my passphrase because at that point the keyboard is dead. With 5.7.17-200 it worked fine. Now I thought that there might be a module missing in the initrd, but alas: $ diff <(lsinitrd /boot/initramfs-5.7.17-200.fc32.x86_64.img | sed -r 's|[^/]+fc32\.x86_64|KVER|g' | cut -c 56-) \ <(lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | sed -r 's|[^/]+fc32\.x86_64|KVER|g' | cut -c 56-) 1931,1933d1930 < usr/lib/modules/KVER/kernel/crypto < usr/lib/modules/KVER/kernel/crypto/ecc.ko.xz < usr/lib/modules/KVER/kernel/crypto/ecdh_generic.ko.xz 2084a2082,2085 > usr/lib/modules/KVER/kernel/drivers/media > usr/lib/modules/KVER/kernel/drivers/media/cec > usr/lib/modules/KVER/kernel/drivers/media/cec/core > usr/lib/modules/KVER/kernel/drivers/media/cec/core/cec.ko.xz 2106a2108 > usr/lib/modules/KVER/kernel/drivers/pinctrl/intel/pinctrl-jasperlake.ko.xz 2127a2130 > usr/lib/modules/KVER/kernel/drivers/tty/serial/lantiq.ko.xz 2130a2134,2135 > usr/lib/modules/KVER/kernel/drivers/usb/host/xhci-pci.ko.xz > usr/lib/modules/KVER/kernel/drivers/usb/host/xhci-pci-renesas.ko.xz
Phil, (In reply to Phil from comment #16) > the lsusb is from when the system has finished booting and the gdm login > screen is shown. > > My problem is that the usb devices are missing while running from initrd. > > When the PC powers on, all sata disks are security-locked. I'm using a > dracut module that asks for the passphrases (read -s -r tmp < /dev/console), > unlocks the disks, triggers a rescan and then continues booting. With kernel > 5.8.4-200 I cannot enter my passphrase because at that point the keyboard is > dead. With 5.7.17-200 it worked fine. I do see in the dmesg for 5.8 that for some reason 5.8 takes significantly longer to initialize the USB stack. What happens if you boot 5.8 and then wait 10 seconds before trying to type the passphrase? And the keyboard does work once you are in gdm? Also how did you get into gdm with 5.8 if you cannot enter the passphrase?
Hans, (In reply to Hans de Goede from comment #17) > What happens if you boot 5.8 and then > wait 10 seconds before trying to type the passphrase? I'll give it a try. > And the keyboard does work once you are in gdm? Also how did you get into > gdm with 5.8 if you cannot enter the passphrase? It's simple: Boot with 5.7.17-200, enter passphrases, press sysrq-b, boot with 5.8.4-200 :)
(In reply to Phil from comment #18) > It's simple: Boot with 5.7.17-200, enter passphrases, press sysrq-b, boot > with 5.8.4-200 :) Nifty. And then once at gdm the keyboard and mouse do work?
(In reply to Hans de Goede from comment #19) > Nifty. And then once at gdm the keyboard and mouse do work? Yes, when the system is booted Just for fun I added a "sleep 10; lsusb -t >&2" to the initrd and that's all I get: /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M Bus 03 to 06 are missing. So ehci-pci is working, xhci_hcd not. Also, at that point the module is not loaded. The difference between kernel 5.7.17-200 and 5.8.4-200 config is: $ diff <(grep -i 'usb_.hci' config-5.7.17-200.fc32.x86_64) <(grep -i 'usb_.hci' config-5.8.4-200.fc32.x86_64 ) 4c4,5 < CONFIG_USB_XHCI_PCI=y --- > CONFIG_USB_XHCI_PCI=m > CONFIG_USB_XHCI_PCI_RENESAS=m Do I need some rdrdinsmodpost magic?
(In reply to Phil from comment #20) > (In reply to Hans de Goede from comment #19) > > Nifty. And then once at gdm the keyboard and mouse do work? > > Yes, when the system is booted > > Just for fun I added a "sleep 10; lsusb -t >&2" to the initrd and that's all > I get: > > /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M > |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M > /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M > |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M > > Bus 03 to 06 are missing. So ehci-pci is working, xhci_hcd not. > > Also, at that point the module is not loaded. The difference between kernel > 5.7.17-200 and 5.8.4-200 config is: > > $ diff <(grep -i 'usb_.hci' config-5.7.17-200.fc32.x86_64) <(grep -i > 'usb_.hci' config-5.8.4-200.fc32.x86_64 ) > 4c4,5 > < CONFIG_USB_XHCI_PCI=y > --- > > CONFIG_USB_XHCI_PCI=m > > CONFIG_USB_XHCI_PCI_RENESAS=m Hmm, I think that that as an unintended change, I will discuss this with the Fedora kernel team. > Do I need some rdrdinsmodpost magic? First thing would be to run lsinitrd on the 5.8 initrd, e.g.: sudo lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | grep xhci And check if the xhci-pci module is there. If it is there then the issue is the custom dracut magic you added for disk unlocking runs before udev gets started and modprobes the module; if it is not there, then this might be considered a dracut issue. I say might because I believe that the changing of the xhci-pci driver to no longer be builtin was unintentional. So the most likely fix here is to undo that change.
Phil, Ian, I've just started a scratchbuild of 5.8.5 with the XHCI PCI driver built into the kernel again (or at least it should be). Can you both give this a try please? ATM it is still building, this should be done in a couple of hours: https://koji.fedoraproject.org/koji/taskinfo?taskID=50639874 See here for some generic instructions for testing kernels directly from koji: https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Note you need to (temporarily) disable secureboot to test this kernel, either in your BIOS or run: sudo mokutil --disable-validation And follow the instructions.
FYI the scratch-build of 5.8.5 with XHCI builtin again has just completed building.
Hi Hans, (In reply to Hans de Goede from comment #21) > First thing would be to run lsinitrd on the 5.8 initrd, e.g.: > > sudo lsinitrd /boot/initramfs-5.8.4-200.fc32.x86_64.img | grep xhci > > And check if the xhci-pci module is there. yep, as shown in https://bugzilla.redhat.com/show_bug.cgi?id=1874300#c16 the module is there. > If it is there then the issue is the custom dracut magic you added for disk > unlocking runs before udev gets started and modprobes the module; Maybe. The priority of that dracut module is 6, I'll try setting it to a higher value. Thanks a lot!
Note the Fedora kernel team has agreed to set the XHCI module to be builtin again for the upcoming 5.8.6 kernel (whenever that comes out), matching what I've done for the 5.8.5 test / scratch-build.
Unfortunately the test kernel hasn't worked for me. Here is the "lsusb -t" from it: zuke$ lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M |__ Port 5: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.8.5-200.rhbz1874300.fc32.x86_64 zuke$
With everything plugged in the same way as comment #26, the working kernel looks like this: zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M |__ Port 5: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.7.17-200.fc32.x86_64 zuke$
Hans, thanks! Unfortunately, the 5.8.5-200 kernel didn't help as xhci_pci is still compiled in as a module: $ grep XHCI_PCI /boot/config-5.8.5-200.rhbz1874300.fc32.x86_64 CONFIG_USB_XHCI_PCI=m CONFIG_USB_XHCI_PCI_RENESAS=m Oh, regarding the dracut module's priority: I believe this issue also might affect users with LUKS-encrypted root devices, as the 'crypt' module gets started before the 'kernel-modules' module. $ ls -1d /lib/dracut/modules.d/*{crypt,kernel-modules} /lib/dracut/modules.d/90crypt /lib/dracut/modules.d/90kernel-modules
FWIW – rdloaddriver doesn't help as that boot argument is being evaluated too late (S90kernel-modules). It might help Ian, though. The only working solution for me so far (apart from compiling a custom kernel) is to include a 'modprobe xhci_pci' in my dracut module.
Ugh, I forgot to run "make config-files" which is necessary when changing the Fedora kernel config templates, sorry. Here is a new scratch-build where I did actually run "make config-files": https://koji.fedoraproject.org/koji/taskinfo?taskID=50680004 Can you both please give this a try (once it has completed building) ?
FWIW, the new, hopefully fixed, scratch-build is done now.
Thanks Hans, the new build fixed it :)
Phil, That is good to hear. Ian, can you give this new build a try to please? Ian, also I wonder if your problem like Phil's was with entering a password for disk-encryption at boot, or were you seeing issues once fully logged in too ? The xhci driver being a module should not cause issues once the boot is past the initrd-phase (it shouldn't but this still is the most likely cause of your issues too).
Hans, I do use LUKS on some VMs I share between work and home in case my car get ripped off with VMs on a portable drive, but this issue is on real metal and I am not using any encryption. Sadly we're victim of timezones, the issue I have is on a work machine and I'm home for the night. Will test as soon as I get in.
It still isn't working for me. I booted with the keyboard and mouse plugged into the sockets that don't work in 5.8.4. At the Grub kernel menu the keyboard works, and the LED on the mouse is on, but after I boot the keyboard and mouse were dead, and the LED on the mouse was off. Plugged both into working sockets and this is the result: zuke$ lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M |__ Port 5: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 6: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.8.5-200.rhbz1874300_2.fc32.x86_64 zuke$ zuke$ rpm -qa | grep rhbz1874300 kernel-modules-5.8.5-200.rhbz1874300_2.fc32.x86_64 kernel-devel-5.8.5-200.rhbz1874300_2.fc32.x86_64 kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64 kernel-modules-extra-5.8.5-200.rhbz1874300_2.fc32.x86_64 kernel-core-5.8.5-200.rhbz1874300_2.fc32.x86_64 zuke$
Hans, I should point out this problem is not a show stopper for me as it only takes out 3 USB ports on the back panel, they just happened by coincidence to be the ports I was using for the mouse and keyboard, so when the 5.8.4 kernel arrived it broke the box until I worked out what was going on. Technically I can live without these 3 ports, but obviously some other people may lose more than 3. Does upstream have any ideas? Willing to try more things if you want to pursue this.
FEDORA-2020-708b23f2ce has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce
FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-708b23f2ce` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
What happened between kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64 and kernel-5.8.6-201.fc32.x86_64 USB wise? zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 3: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 4: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.8.6-201.fc32.x86_64 zuke$ All my USB ports are back !! Keyboard and mouse are back in the previously unworking ports in the listing above.
(In reply to Ian Laurie from comment #39) > What happened between kernel-5.8.5-200.rhbz1874300_2.fc32.x86_64 and > kernel-5.8.6-201.fc32.x86_64 USB wise? The xhci controller driver got moved back to being built-in instead of being build as a module. > All my USB ports are back !! > > Keyboard and mouse are back in the previously unworking ports in the listing > above. That is good to know. From Fedora's pov that means that this bug is resolved now, as the building of the XHCI driver as a module was done by accident and the plan is to keep it built-in. The updates system will close this bug once kernel-5.8.6-201.fc32.x86_64 hits the stable updates repository. I will follow-up with upstream about the ports disappearing when xhci is built as a module, as that should not happen.
Ian one last question, upon reading your lsusb output, I now see that you did the lsusb-s all with the 5.7.17 kernel, right? What is the output of "lsusb -t" with the mouse/kbd in the troublesome ports and a non working kernel ?
Ian, one more question, do you have a usb device with a LED which always lights up when there is power (the red light at the bottom of a mouse typically turns off when not enumerated over USB) ? And if so can you try this in one of the trouble some ports with a non-working kernel? I think the 5V on the ports gets turned off when the xhci driver is built as a module.
(1) Comment #12, comment #27 and comment #39 were done with working kernels, the kernel appears in my "uname -r" in the comment. Comment #26 and comment #35 were non-working kernels, again with a "uname -r" showing which one it was. (2) I can test this in the morning, non-working kernel, keyboard and mouse in non-functional ports, I will get the "lsusb -t" over ssh. (3) I don't have a device to confirm if power is removed from a non-working port, but I can test if the 5V is there with a multimeter. Again I can do this in the morning.
FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.
garberw@electron> uname -a Linux electron.localdomain 5.8.4-200.fc32.x86_64 #1 SMP Wed Aug 26 22:28:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux I have two usb3.0gen1 cards with 4 ports each. model sunix USB4300NS chipset renesas/nec uPD720201 I get this in dmesg: [ 2.933892] xhci_hcd 0000:17:00.0: FW has invalid version :8224 [ 2.933908] xhci_hcd 0000:17:00.0: Direct firmware load for renesas_usb_fw.mem failed with error\ -2 [ 2.933910] xhci_hcd 0000:17:00.0: request_firmware failed: -2 [ 2.933988] xhci_hcd: probe of 0000:17:00.0 failed with error -2 [ 2.937826] xhci_hcd 0000:18:00.0: FW has invalid version :8224 [ 2.937838] xhci_hcd 0000:18:00.0: Direct firmware load for renesas_usb_fw.mem failed with error\ -2 [ 2.937840] xhci_hcd 0000:18:00.0: request_firmware failed: -2 [ 2.937910] xhci_hcd: probe of 0000:18:00.0 failed with error -2 root@electron# lsusb -t /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M |__ Port 1: Dev 2, If 1, Class=Vendor Specific Class, Driver=, 480M |__ Port 1: Dev 2, If 0, Class=Printer, Driver=usblp, 480M |__ Port 7: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 7: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 8: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 8: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M root@electron# root@electron# lsmod | grep xhci xhci_pci 20480 0 xhci_pci_renesas 20480 1 xhci_pci root@electron# the ports no longer work. rebooting into kernel 5.7.17-200.fc32.x86_64 they work again indicating sunix usb3 cards and connected external hard drive and usb3 enclosure are not broken. so it must be something to do with the kernel. this seems to be a report of something similar: https://bbs.archlinux.org/viewtopic.php?id=258184 root@electron# fdisk -l Disk /dev/sdb: 223.58 GiB, 240057409536 bytes, 468862128 sectors Disk model: SSD2SC240G1SA754 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x7639e137 Device Boot Start End Sectors Size Id Type /dev/sdb1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT /dev/sdb2 1026048 467868301 466842254 222.6G 7 HPFS/NTFS/exFAT /dev/sdb3 467869696 468856831 987136 482M 27 Hidden NTFS WinRE Disk /dev/sda: 476.96 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Crucial_CT512MX1 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 756AA145-ECFC-4044-8BBF-4AF4E15F61BB Device Start End Sectors Size Type /dev/sda1 2048 2099199 2097152 1G Linux filesystem /dev/sda2 2099200 6293503 4194304 2G EFI System /dev/sda3 6293504 1000214527 993921024 474G Linux LVM Disk /dev/mapper/main-root: 55 GiB, 59055800320 bytes, 115343360 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/main-00: 48 GiB, 51539607552 bytes, 100663296 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/main-home: 355.96 GiB, 382184980480 bytes, 746455040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/mapper/main-var: 15 GiB, 16106127360 bytes, 31457280 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes root@electron# fdisk -l shows disk was not recognized (/dev/sdc missing). attach dmesg.
Created attachment 1714009 [details] william.garber dmesg dmesg invalid firmware
Created attachment 1714010 [details] william.garber dmesg dmesg bug about firmware
Late yesterday I converted the box over to Fedora 33 beta branch, however the initial kernel is 5.8.3 which is problematic in Fedora 33 as well, so it shouldn't disturb the results. From comment #41 >What is the output of "lsusb -t" with the mouse/kbd in the troublesome ports and a non working kernel ? zuke$ lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.8.3-300.fc33.x86_64 zuke$ For the above, keyboard/mouse in non-functioning ports, mouse LED is off, results obtained through SSH, kernel-5.8.3-300.fc33.x86_64 from Fedora 33 beta branch.
From comment #42 >I think the 5V on the ports gets turned off when the xhci driver is built as a module. Not the results I was expecting, but I have 5V on the problematic ports, all 3 of them, even though the keyboard and mouse appear dead when plugged into them.
For the sake of completeness here is an "lsusb -t" from the [now] default kernel on the box which is 5.8.6-301: zuke$ lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=, 480M |__ Port 3: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 4: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M |__ Port 4: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M zuke$ uname -r 5.8.6-301.fc33.x86_64 zuke$ For this listing the keyboard and mouse were in the same usb sockets that are non-functional in the problematic kernels. Everything works well in kernel-5.8.6-301.fc33.x86_64. In other words, the only difference between this listing and the one in comment #49 is the kernel.
possibly this uefi setting could help? did not help me. https://askubuntu.com/questions/729558/how-do-i-get-usb-3-0-driver-working-or-check-that-it-is-already-working reposting from there: it's very simple, i struggled with this issue using Ubuntu and Ubuntu flavored distros for years (Mint, Elementary OS, etc). Go back into bios, have usb 3.0 turned on, an any other options turned on, but turn off legacy usb option. The description of legacy usb is that if you have it off, that will disable it for any os that's not "usb aware". But I thought flip the switch, because it's 2018 and most os's are usb aware now. It wasn't supposed to work, but it fixed the issue that has baffled me for years. My usb 3.0 works perfectly now. My theory is that usb legacy conflicts with the os's understanding of 3.0, so now there's no conflict. If it works for you, you're welcome. I googled this much, and no one else seemed to have tried or had the same conclusion. I hope this helps others who struggled with it.
garberw@electron> uname -a Linux electron.localdomain 5.8.6-201.fc32.x86_64 #1 SMP Fri Sep 4 03:27:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux at 2:30pst kernel was upgraded and now it works. external hard disks automounted. no error message about firmware in dmesg.
(In reply to Ian Laurie from comment #48) > zuke$ lsusb -t > /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M > |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M > /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M > |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M > zuke$ uname -r > 5.8.3-300.fc33.x86_64 Ok, thank you for all your patience in getting to the bottom of this. So the problem is the xhci driver not loading / binding to the xhci-controller at all when not building as a module. Now that I know that, if I go back to your original dmesg (first attachment, 5.8.4 kernel) I see: Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: FW has invalid version :8216 Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: Direct firmware load for renesas_usb_fw.mem failed with error -2 Sep 01 10:47:36 kernel: xhci_hcd 0000:05:00.0: request_firmware failed: -2 Sep 01 10:47:36 kernel: xhci_hcd: probe of 0000:05:00.0 failed with error -2 SO I now see that you are using a Renesas XHCI controller. I missed that at first, because I was looking at the "0000:00:05.0" pci-device-ids, where as your xhci controller is the "0000:05:00.0" pci-device, my bad. So you seem to have the same problem as william.garber, which is the new renasas xhci controller firmware loading support breaking things. The issue here is that the version check in the upstream code was no good, this has been fixed in this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d66a57be2f9a315fc10d0f524f670fec903e0fb4 This fix is available in the 5.8.6 kernel, which is in updates testing now, quoting from comment 38 : FEDORA-2020-708b23f2ce has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-708b23f2ce` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-708b23f2ce See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. Ian, you have already confirmed that 5.8.6 fixes things and the above actually explains why it fixes things, so I believe the issue on your system is fully resolved now. William, can you test the 5.8.6 update from updates-testing (see above)? That should fix this for you too.
see comment 52. it is working now. fixed in 5.8.6-201.fc32.x86_64 awesome; thank you :-)
>Ian, you have already confirmed that 5.8.6 fixes things and the above actually explains why it fixes things... The best conclusions are when in the end we understand fully the failure and fix scenario, this is a great result, thank you Hans.
I just ran into this with kernel 5.8.7. 5.8.6 works great, but USB devices stop working on 5.8.7.
(In reply to bryanhoop from comment #56) > I just ran into this with kernel 5.8.7. 5.8.6 works great, but USB devices > stop working on 5.8.7. This sounds like a different issue to me, please file a new bug for this. In this new bug please attach the following: 1. dmesg output of working kernel 2. lsusb output on working kernel 3. "lsusb -t" output on working kernel 4. dmesg output of non-working kernel 5. "lsusb -t" output on non-working kernel
I think this can be closed.
(In reply to Ian Laurie from comment #58) > I think this can be closed. Agreed, closing.