Created attachment 1711424 [details] kernel logs 1. Please describe the problem: Touchpad does not work at all. For reference, hardware is functional as touchpad works under Win10. USB and BT mice work. output of system info: # dmidecode System Information Manufacturer: LENOVO Product Name: 81WE Version: IdeaPad 3 15IIL05 Serial Number: PF25BWBN UUID: 58961da7-bdb8-2020-0425-103601000000 Wake-up Type: Power Switch SKU Number: LENOVO_MT_81WE_BU_idea_FM_IdeaPad 3 15IIL05 Family: IdeaPad 3 15IIL05 2. What is the Version-Release number of the kernel: 5.7.14-200.fc32.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : No previous Fedora releases tested - F32 only. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Boot laptop to desktop Wayland session. Touchpad non-functional at any time since F32 installation. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Not tested rawhide due to GPG key issue using the repo, however problem exists in latest updates-testing kernel 5.7.15-200.fc32.x86_64. 6. Are you running any modules that not shipped with directly Fedora's kernel?: No 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Attached. I believe the following info may be relevant from dmesg? This happens every boot and is reported as an kerneloops in kernel-core package via ABRT: [ 9.820184] irq 9: nobody cared (try booting with the "irqpoll" option) [ 9.820186] CPU: 1 PID: 844 Comm: rngd Not tainted 5.7.14-200.fc32.x86_64 #1 [ 9.820186] Hardware name: LENOVO 81WE/LNVNB161216, BIOS EMCN37WW 06/24/2020 [ 9.820187] Call Trace: [ 9.820189] <IRQ> [ 9.820194] dump_stack+0x64/0x88 [ 9.820197] __report_bad_irq+0x35/0xa7 [ 9.820198] note_interrupt.cold+0xb/0x6a [ 9.820199] handle_irq_event+0x88/0x8a [ 9.820201] handle_fasteoi_irq+0x78/0x1c0 [ 9.820203] do_IRQ+0x50/0xe0 [ 9.820205] common_interrupt+0xf/0xf [ 9.820206] </IRQ> [ 9.820207] RIP: 0033:0x7f01133a49df [ 9.820208] Code: 48 8b 45 e0 48 c1 e8 1b 83 e0 01 48 31 45 f0 48 8b 45 e0 48 c1 e8 16 83 e0 01 48 31 45 f0 48 d1 65 e0 48 8b 45 f0 48 31 45 e0 <83> 45 d4 01 83 7d d4 40 0f 86 72 ff ff ff 48 83 45 d8 01 48 8b 45 [ 9.820209] RSP: 002b:00007f0111cb0bc0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde [ 9.820210] RAX: 0000000000000001 RBX: 000000000000000a RCX: 000000000000002e [ 9.820210] RDX: 099c400000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 9.820211] RBP: 00007f0111cb0c10 R08: 0000000000000000 R09: 0000000000000035 [ 9.820211] R10: 00007f0111cb0a77 R11: 0000000000000000 R12: 00007f00fc004c00 [ 9.820211] R13: 00007fff3323cf5f R14: 00007f0111cb0d10 R15: 000055d8d442b4c0 [ 9.820213] handlers: [ 9.820215] [<0000000094f280d8>] acpi_irq [ 9.820216] Disabling IRQ #9
FYI, latest kernel 5.8.4-200.fc32 of the new 5.8 branch still contains this touchpad issue. Are there any temporary fixes or official patches forthcoming to support this hardware functionality available here or upstream?
Updated Fedora version to the current 33rd edition as problem still exists.
Updated to latest Fedora release 34. Is there a timeline that could be conjected by anyone cc'd in this bug to illuminate when one on average sees hardware such as this touchpad reach supported status in Fedora once a new device is released to market? Are there some proactive steps I could take to expedite the process toward driver support for this hardware in Fedora?
(In reply to Matt from comment #3) > Updated to latest Fedora release 34. > > Is there a timeline that could be conjected by anyone cc'd in this bug to > illuminate when one on average sees hardware such as this touchpad reach > supported status in Fedora once a new device is released to market? > > Are there some proactive steps I could take to expedite the process toward > driver support for this hardware in Fedora? Sorry that it is taking so long to resolve this issue. I've been working on fixing touchpad issues on various Lenovo Ideapad laptops, but there are multiple issues at play. Twice now I thought I had fixed things and I did fix things for some Ideapad models, but others still do not work. Let me copy and paste a large comment which I added to another bug. I'll put this in a separate comment here for clarity.
So this is a bit of a mess which I'm trying to sort out ATM, there are at least 4 different issues with Ideapad touchpads that I'm aware of: 1. There was a Linux ACPI issue where the resource-list (_CRS) for the touchpad given to Linux was not correct, this has been fixed for a while now by these 2 commits from me: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=21653a4181ff292480599dad996a2b759ccf050f https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8058d69905058ec8f467a120b5ec5bb831ea67f3 These commits should be in recent Ubuntu kernels so if you are still seeing issues then this issue was not the root cause for your laptop model. 2. On some Ideapad models the elants_i2c driver takes control of handling the touchpad, but the touchpad does not talk the elants protocol at all, it uses the standard I2C-HID protocol so this does not work. If adding "initcall_blacklist=elants_i2c_driver_init" to your kernel commandline fixes things, then your Ideapad model is hit by this. This is fixed now by this commit from me: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=65299e8bfb24774e6340e93ae49f6626598917c8 This commit was added to the 5.10.y and 5.12.y stable kernel series a couple of days ago, so hopefully it will get added to the Ubuntu kernels soon too. 3. On some Ideapads the BIOS seems to corrupt its own settings (often after a BIOS update) and as a result of this the BIOS is returning bogus addresses (NULL or 0xffffffff) as base-address from the ACPI _CRS call for some PCI devices, including the I2C controller connected to the touchpad. If adding "pci=nocrs" as a workaround helps then your model Ideapad is affected by this issue. 4. On some Ideapads something goes wrong with the OSYS code in the ACPI tables, this code is responsible for detecting which Windows version the ACPI tables think they are dealing with (or which Windows version Linux mimicks in case you are running Linux). To see if your Ideapad is affected by this issue try running: "cat /sys/bus/acpi/devices/XXXX0000\:00/status" from a terminal, if the output of this command is "15" then your model is affected by this issue. At least the "Lenovo Thinkbook 14-ILL" is known to be affected by this. The exact root cause of this is unknown atm. If your laptop is affected by this please run "acpidump -o acpidump" and attach the acpidump here, in combination with specifying the exact model of your laptop.
Translating this to your situation / laptop: 1. Is fixed in Fedora 34, so you are not hit by 1. 2. Is fixed in kernel 5.12.6, 5.12.7 is in the Fedora 34 updates repo now, so if you run "sudo dnf update 'kernel*'" then you should get a kernel >= 5.12.6 installed. If you reboot after this and things work now, then 2. was the root cause and you can stop reading here :) 3. To test if you are hit by this run: "sudo grubby --update-kernel=ALL --args=pci=nocrs" and then reboot. If this works around the issue please let me know, I'm still investigating this variant of the problem. If this does not work, run the following command to undo the change: "sudo grubby --update-kernel=ALL --remove-args=pci=nocrs" 4. As the comment says (and if 1-3 do not help), try running: "cat /sys/bus/acpi/devices/XXXX0000\:00/status" from a terminal, if the output of this command is "15" then your model is affected by this issue
(In reply to Hans de Goede from comment #6) > Translating this to your situation / laptop: > > <snip> > > 3. To test if you are hit by this run: > "sudo grubby --update-kernel=ALL --args=pci=nocrs" > and then reboot. If this works around the issue please let me know, I'm > still investigating this variant of the problem. > If this does not work, run the following command to undo the change: > "sudo grubby --update-kernel=ALL --remove-args=pci=nocrs" Hi Hans, Apologies for the delay in responding. I can happily confirm this 3rd workaround fixes the problem for my laptop touchpad functionality! :-) I also tested gestures and the 2-fingered combo works however the new 3-finger Gnome 40 gesturing doesn't seem to work. (perhaps I'm just not doing it correctly) > > 4. As the comment says (and if 1-3 do not help), try running: > "cat /sys/bus/acpi/devices/XXXX0000\:00/status" > from a terminal, if the output of this command is "15" then your model is > affected by this issue Note, this device path doesn't exist on my system. Is this out of the oridinary? -Matt
(In reply to Matt from comment #7) > Apologies for the delay in responding. I can happily confirm this > 3rd workaround fixes the problem for my laptop touchpad functionality! :-) That is good news. I think I know what is going on here and I've written a kernel fix which should make things work without the workaround. I've build a test-kernel with the fix which I wrote added: https://koji.fedoraproject.org/koji/taskinfo?taskID=70106533 Here are some generic instructions for installing a kernel directly from koji (the Fedora buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Can you please undo the workaround by running: sudo grubby --update-kernel=ALL --remove-args=pci=nocrs And then give this new kernel a try. After booting the new kernel please run: dmesg > dmesg-patched-kernel.txt And attach the generated dmesg-patched-kernel.txt. Also please let me know if the touchpad still works with the patches kernel without the workaround. > > 4. As the comment says (and if 1-3 do not help), try running: > > "cat /sys/bus/acpi/devices/XXXX0000\:00/status" > > from a terminal, if the output of this command is "15" then your model is > > affected by this issue > Note, this device path doesn't exist on my system. Is this out of the > oridinary? The path NOT existing is normal. If the path exists then that actually is indicative of another bug which is also causing touchpads to not work on some models.
Created attachment 1792465 [details] dmesg output
(In reply to Hans de Goede from comment #8) > (In reply to Matt from comment #7) > > Apologies for the delay in responding. I can happily confirm this > > 3rd workaround fixes the problem for my laptop touchpad functionality! :-) > > That is good news. I think I know what is going on here and I've written a > kernel fix which should make things work without the workaround. > > I've build a test-kernel with the fix which I wrote added: > https://koji.fedoraproject.org/koji/taskinfo?taskID=70106533 > > Here are some generic instructions for installing a kernel directly from > koji (the Fedora buildsystem): > https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt > > Can you please undo the workaround by running: > > sudo grubby --update-kernel=ALL --remove-args=pci=nocrs > > And then give this new kernel a try. After booting the new kernel please run: > > dmesg > dmesg-patched-kernel.txt > > And attach the generated dmesg-patched-kernel.txt. Also please let me know > if the touchpad still works with the patches kernel without the workaround. Hi Hans, Unfortunately this patched kernel does not fix the bug for my touchpad. The dmesg output is attached. Regards, Matt
Thank you, I've been discussing this problem upstream and my initial patch was no good, sorry about that. We think we have some idea what is going on, but we need some more info. So I've prepared another test kernel with a patch which prints some extra debugging included: https://koji.fedoraproject.org/koji/taskinfo?taskID=70539682 ATM this is still building, this takes a couple of hours. Can you please install this kernel once it is done building and collect dmesg output both with and without "pci=nocrs" on the kernel commandline; and then attach both dmesg files here ? Note this is not expected to work without "pci=nocrs", the goal is purely to gather some information.
Created attachment 1794948 [details] dmesg output (with pci cmdline arg)
Created attachment 1794960 [details] dmesg output (without pci cmdline arg)
(In reply to Hans de Goede from comment #11) > Thank you, I've been discussing this problem upstream and my initial patch > was no good, sorry about that. > > We think we have some idea what is going on, but we need some more info. So > I've prepared another test kernel with a patch which prints some extra > debugging included: > https://koji.fedoraproject.org/koji/taskinfo?taskID=70539682 > > ATM this is still building, this takes a couple of hours. > > Can you please install this kernel once it is done building and collect > dmesg output both with and without "pci=nocrs" on the kernel commandline; > and then attach both dmesg files here ? Note this is not expected to work > without "pci=nocrs", the goal is purely to gather some information. Hi Hans, I've attached the two dmesg files as requested. HTH the debugging and patching process. -Matt
Thank you for the logs, here is another test-kernel with more debugging added: https://koji.fedoraproject.org/koji/taskinfo?taskID=71078980 ATM this is still building, this takes a couple of hours. Can you please install this kernel once it is done building and collect dmesg output with "log_buf_len=50M pci=nocrs" on the kernel commandline ?
Created attachment 1796782 [details] dmesg output for 5.12.13-300 Hi Hans, dmesg attached. HTH. -Matt.
Matt, thank you. I'm afraid I made a mistake I actually need the dmesg output without the "pci=nocrs", so with just "log_buf_len=50M", note the log_buf_len is probably not necessary but better safe then sorry. If you can grab the dmesg without the "pci=nocrs", that would be great.
Created attachment 1796799 [details] dmesg output for 5.12.13-300 (In reply to Hans de Goede from comment #17) > Matt, thank you. > > I'm afraid I made a mistake I actually need the dmesg output without the > "pci=nocrs", so with just "log_buf_len=50M", note the log_buf_len is > probably not necessary but better safe then sorry. > > If you can grab the dmesg without the "pci=nocrs", that would be great. No worries Hans. I've attached the updated dmesg without the pci kernel arg. -Matt
Thanks, the last set of logs has helped to finally understand what is going on here. Your BIOS claims that any memory between 0x000000004bc50000-0x00000000cfffffff is reserved: [ 0.000000] BIOS-e820: [mem 0x000000004bc50000-0x00000000cfffffff] reserved But then continuous with placing the iomem window for PCI devices inside that reserved range: [ 0.609988] pci_bus 0000:00: root bus resource [mem 0x65400000-0xbfffffff window] This confuses the kernel and makes it unable to assign PCI iomem to devices which have not already been assigned iomem by the BIOS, like the I2C-controller used for the touchpad. If you are interested in / curious about the low level details, I'm discussing this with the kernel PCI developers (including figuring out how to deal with this) here: https://lore.kernel.org/linux-pci/7d80f639-0768-850b-5313-3bdedf0a5579@redhat.com/
(In reply to Hans de Goede from comment #19) > Thanks, the last set of logs has helped to finally understand what is going > on here. > > Your BIOS claims that any memory between > 0x000000004bc50000-0x00000000cfffffff is reserved: > > [ 0.000000] BIOS-e820: [mem 0x000000004bc50000-0x00000000cfffffff] > reserved > > But then continuous with placing the iomem window for PCI devices inside > that reserved range: > > [ 0.609988] pci_bus 0000:00: root bus resource [mem 0x65400000-0xbfffffff > window] > > This confuses the kernel and makes it unable to assign PCI iomem to devices > which have not already been assigned iomem by the BIOS, like the > I2C-controller used for the touchpad. > > If you are interested in / curious about the low level details, I'm > discussing this with the kernel PCI developers (including figuring out how > to deal with this) here: > https://lore.kernel.org/linux-pci/7d80f639-0768-850b-5313- > 3bdedf0a5579/ Hi Hans, thanks for the informative summary of the low-level details from the discussion at the linux-pci list. It's always interesting to read about a bug being unravelled and eventually squashed. I will try to respond promptly over the next few days if further logs/data are required. -Matt
Hi, Sorry for the long silence on this bug. I was hoping one of the upstream pci-subsys devs would help out with this... Since that did not happen I've been looking into fixing this myself now. I've come up with a solution which still involves a kernel cmdline option for now, but this one is a much smaller hammer (almost a scalpel) and once it is confirmed that this works I can enable this by default through a DMI quirk. I've build a test-kernel with support for the new kernel cmdline option here: https://koji.fedoraproject.org/koji/taskinfo?taskID=76687852 Here are some generic instructions for installing a kernel directly from koji (the Fedora buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Please undo the pci=nocrs workaround by running: sudo grubby --update-kernel=ALL --remove-args=pci=nocrs And then enable the new workaround by: sudo grubby --update-kernel=ALL --args=pci=no_e820 Also please run the following command *as normal user* to collect the DMI strings for your laptop, so that they can be used to add a quirk: grep . /sys/class/dmi/id/* 2> /dev/null And copy and paste the output here.
p.s. Please also provide dmesg output for the new kernel after booting it with pci=no_e820 on the kernel cmdline.
(In reply to Hans de Goede from comment #21) > Hi, > > Sorry for the long silence on this bug. I was hoping one of the upstream > pci-subsys devs would help out with this... Since the nocrs cmdline arg solution has been available to use as a workaround, there's no problems with waiting. :) > Since that did not happen I've been looking into fixing this myself now. > I've come up with a solution which still involves a kernel cmdline option > for now, but this one is a much smaller hammer (almost a scalpel) and once > it is confirmed that this works I can enable this by default through a DMI > quirk. Can confirm the more refined no_e820 cmdline arg works and imo it seems to offer a more responsive/smoother engagement when using the touchpad. > Also please run the following command *as normal user* to collect the DMI > strings for your laptop, so that they can be used to add a quirk: > > grep . /sys/class/dmi/id/* 2> /dev/null > > And copy and paste the output here. /sys/class/dmi/id/bios_date:12/23/2020 /sys/class/dmi/id/bios_release:1.44 /sys/class/dmi/id/bios_vendor:LENOVO /sys/class/dmi/id/bios_version:EMCN44WW /sys/class/dmi/id/board_asset_tag:NO Asset Tag /sys/class/dmi/id/board_name:LNVNB161216 /sys/class/dmi/id/board_vendor:LENOVO /sys/class/dmi/id/board_version:SDK0J40700 WIN /sys/class/dmi/id/chassis_asset_tag:NO Asset Tag /sys/class/dmi/id/chassis_type:10 /sys/class/dmi/id/chassis_vendor:LENOVO /sys/class/dmi/id/chassis_version:IdeaPad 3 15IIL05 /sys/class/dmi/id/ec_firmware_release:1.44 /sys/class/dmi/id/modalias:dmi:bvnLENOVO:bvrEMCN44WW:bd12/23/2020:br1.44:efr1.44:svnLENOVO:pn81WE:pvrIdeaPad315IIL05:rvnLENOVO: rnLNVNB161216:rvrSDK0J40700WIN:cvnLENOVO:ct10:cvrIdeaPad315IIL05:skuLENOVO_MT_81WE_BU_idea_FM_IdeaPad315IIL05: /sys/class/dmi/id/product_family:IdeaPad 3 15IIL05 /sys/class/dmi/id/product_name:81WE /sys/class/dmi/id/product_sku:LENOVO_MT_81WE_BU_idea_FM_IdeaPad 3 15IIL05 /sys/class/dmi/id/product_version:IdeaPad 3 15IIL05 /sys/class/dmi/id/sys_vendor:LENOVO /sys/class/dmi/id/uevent:MODALIAS=dmi:bvnLENOVO:bvrEMCN44WW:bd12/23/2020:br1.44:efr1.44:svnLENOVO:pn81WE:pvrIdeaPad315IIL05:rvnLENOVO: rnLNVNB161216:rvrSDK0J40700WIN:cvnLENOVO:ct10:cvrIdeaPad315IIL05:skuLENOVO_MT_81WE_BU_idea_FM_IdeaPad315IIL05: Cheers, Matt.
Created attachment 1829283 [details] dmesg output with no_e820 pci cmdline arg
Matt, thank you for the quick turn-around time on testing this. I've one more kernel for you to test. This automatically activates the pci=no_e820 behavior on your model laptop, so this one should work without needing any kernel cmdline options at all: https://koji.fedoraproject.org/koji/taskinfo?taskID=76736063 (note this is still building ATM) After booting this (without special kernel cmdline options), 'dmesg | grep "E820 reservations"' should still show: [ 0.558144] PCI: Ignoring E820 reservations for host bridge windows As it does with yesterdays test kernel when adding pci=no_e820 to the kernel cmdline. And your touchpad should still work :) If you can confirm this then I'll submit the new version with the DMI quirk to automatically enable this upstream and then cross my fingers that my workaround for this is acceptable upstream. I'll also try to get testing feedback from people seeing the same issue on a "Lenovo IdeaPad 5 14IIL05" and other related models. Hopefully we will be able to fix this for those other models by adding DMI quirks for them too. Thank you very much for all your help with testing things to get this resolved.
(In reply to Hans de Goede from comment #25) > After booting this (without special kernel cmdline options), 'dmesg | grep > "E820 reservations"' should still show: > > [ 0.558144] PCI: Ignoring E820 reservations for host bridge windows > > As it does with yesterdays test kernel when adding pci=no_e820 to the kernel > cmdline. > > And your touchpad should still work :) > > If you can confirm this then I'll submit the new version with the DMI quirk > to automatically enable this upstream and then cross my fingers that my > workaround for this is acceptable upstream. Hi Hans, I'm happy to confirm your latest kernel build with the new DMI quirk works as expected without the cmdline arg and the dmesg output still includes the above message regarding ignoring E820 reservations. :) > I'll also try to get testing feedback from people seeing the same issue on a > "Lenovo IdeaPad 5 14IIL05" and other related models. Hopefully we will be > able to fix this for those other models by adding DMI quirks for them too. > > Thank you very much for all your help with testing things to get this > resolved. Happy to help - we got there in the end! :) Thanks for your work on this bug and finding a solution. Hopefully it will be suitable upstream and will be useful for applying a fix to similar laptops to mine. Regards, Matt.
Thank you, I've submitted the patch upstream now: https://lore.kernel.org/linux-pci/20211005150956.303707-1-hdegoede@redhat.com/T/#u
Hi Matt, Upstream review/discussion has noted that many different models are impacted by this problem. And the conclusion is that the no_e820 behavior should be the default, but to avoid causing regressions with some older boards which needed the use_e820 behavior the plan (my plan) for now is to keep the old use_e820 behavior for any devices where the cat /sys/class/dmi/id/bios_date year is less then 2018 (so the latest BIOS update is from 2017 or older). This replaces the per model DMI matching from my previous patch. Leading to this new version of the patch: https://lore.kernel.org/linux-pci/20211011090531.244762-1-hdegoede@redhat.com/T/#u I've started a new scratch/test-kernel build with the new version of the patch: https://koji.fedoraproject.org/koji/taskinfo?taskID=77061333 (note this is still building ATM) After booting this (without special kernel cmdline options), 'dmesg | grep "E820 reservations"' should still show: [ 0.558144] PCI: Ignoring E820 reservations for host bridge windows as before and your touchpad should still work :) If you can find some time to test this and confirm that I did not break things, that would be great. Regards, Hans
(In reply to Hans de Goede from comment #28) > Hi Matt, > > Upstream review/discussion has noted that many different models are impacted > by this problem. And the conclusion is that the no_e820 behavior should be > the default, but to avoid causing regressions with some older boards which > needed the use_e820 behavior the plan (my plan) for now is to keep the old > use_e820 behavior for any devices where the cat /sys/class/dmi/id/bios_date > year is less then 2018 (so the latest BIOS update is from 2017 or older). > This replaces the per model DMI matching from my previous patch. Hi Hans, This BIOS date matching logic definitely seems like the more efficient way to scale the application of this patch. :) > > After booting this (without special kernel cmdline options), 'dmesg | grep > "E820 reservations"' should still show: > > [ 0.558144] PCI: Ignoring E820 reservations for host bridge windows > > as before and your touchpad should still work :) > > If you can find some time to test this and confirm that I did not break > things, that would be great. > > Regards, > > Hans The dmesg output still correctly logs the E820 reservations as being ignored and touchpad functions as expected - you did not break things. :) Regards, Matt.
Great, thank you for testing!
Hi Matt, Unfortunately it seems that my fix to ignore e820 reservations for PCI bar allocations to fix the touchpad issue on your laptop is causing issues on other machines, see bug 2029207 and: https://lore.kernel.org/linux-pci/a7ad05fe-c2ab-a6d9-b66e-68e8c5688420@redhat.com/ So I have to come up with a different fix. Can you please let me know if you will be able to test new test-kernel-builds with a different fix for me (once I have them ready) ? Regards, Hans
I'm still discussing alternative fixes upstream. To figure things out, it would really help if you can boot your Lenovo Ideapad Slim 3 with "efi=debug" added to the kernel commandline and then directly after booting do: dmesg > dmesg-efi-debug.txt And attach the dmesg-efi-debug.txt file here, it does not matter which kernel you do this with.
I have come up with an alternative fix for the issue which is causing your touchpad to not work. I have started a test kernel-build with this new patch (replacing the previous one) here: https://koji.fedoraproject.org/koji/taskinfo?taskID=82606916 Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Can you please give this one a test and let me know if the touchpad still works with this one ? (I fully expect it will)
Hi Hans, Apologies for the delay in getting back to you. Had HDD partitioning problems recently, resulting in a full re-format of my laptop and haven't had time to setup the device again yet unfortunately. I therefore only have access to a live image setup via USB. There's no way to test kernels while in a live setup I'm assuming? If there is, I'll test the kernel over the weekend, else my testing will have to wait until the laptop is setup again. Thanks, Matt.
Hi Matt, No worries, I'm happy that you are still willing to help with further testing with this bug, which is turning out much harder to fix then it really should be. The new fix from the test kernel should work fine for your laptop, so there is no big hurry with testing this. You may want to save the rpms from the test-kernel though, since koji removes the files from test builds after about 10 days to save diskspace. What would be very helpful is if you can boot the livecd with "efi=debug" added to the kernel commandline (press e at the livecd boot menu to edit the cmdline) and then collect a dmesg in the livecd env. and somehow (e.g. https://paste.centos.org/, save to a second usb-drive) save that dmesg and attach it here. Regards, Hans
Created attachment 1860924 [details] dmesg output with efi=debug cmdline addition Hi Hans, Here's the requested dmesg with required debug command-line. -Matt.
(In reply to Matt from comment #36) > Here's the requested dmesg with required debug command-line. Thank you. Note there is no need anymore to test the new test-kernel. The approach taken there is still causing issues on some other devices, so I'm working on a new approach/patch. I'll get back to you when I have a test-kernel with a new patch.
OK, I've prepared a series of 2 patches which tries to fix the touchpad problem in a new way, which will hopefully not break suspend/resume for others: https://lore.kernel.org/linux-pci/20220214151759.98267-1-hdegoede@redhat.com/T/ A Fedora test-kernel with these patches is building here: https://koji.fedoraproject.org/koji/taskinfo?taskID=82813003 this should be done in a couple of hours. Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Can you please give this one a test and check if the touchpad still works with this new test-kernel? Please add efi=debug to the kernel commandline when testing this, collect dmesg output with this kernel and attach it here.
Created attachment 1861152 [details] dmesg output for new test-kernel with efi=debug cmdline Hi Hans, I can confirm the touchpad functions correctly with your new patch in this test-kernel. :-) I do hope this latest attempt at fixing this issue is resilient enough to keep all other devices from breaking again. It's been a test of patience for you, I'm sure, trying to fully resolve this and similar issues related to these EFI memmap quirks on mine and similar devices. I'll continue to assist you in testing and helping out with anything further, if you need. I'm optimistic you're getting close to the end here! :-D Regards, Matt.
(In reply to Matt from comment #39) > I can confirm the touchpad functions correctly with your new patch in this > test-kernel. :-) > > I do hope this latest attempt at fixing this issue is resilient enough to > keep all other devices from breaking again. It's been a test of patience for > you, I'm sure, trying to fully resolve this and similar issues related to > these EFI memmap quirks on mine and similar devices. > > I'll continue to assist you in testing and helping out with anything > further, if you need. Thanks much appreciated. > I'm optimistic you're getting close to the end here! I'm optimistic too, but unfortunately we are not quite there yet. The last fix you tested is still causing some potential issues on the laptop from bug 2029207. So into to the recycle bin that fix goes and I've written another workaround for the PCI BAR assignment issue on your laptop (and others like it). I'm hopeful that this really will be the last one. Here is a new test-kernel with the new workaround: https://koji.fedoraproject.org/koji/taskinfo?taskID=82854981 This time the kernel is already fully build, so you can grab it right away. Please add efi=debug to the kernel commandline when testing this, collect dmesg output with this kernel and attach it here, thanks. Note there is a small chance that your touchpad stops working with this one as the fix is taking a different approach trying to only chance kernel behavior on a very narrow set of laptops this time around. It should identify your laptop as one needing to ignore e820 reservations, but if the identifying code somehow fails then your touchpad will stop working.
Created attachment 1861407 [details] dmesg output for latest test-kernel with efi=debug cmdline Hi Hans, Sorry to hear there are still issues fully resolving this bug whilst not breaking other patches for the related bugs. Quite the cat-and-mouse game for you it seems! Good news on my end, however; this latest test-kernel hasn't regressed my touchpad to a state of not functioning again. The dmesg is attached. -Matt.
Hi Matt, (In reply to Matt from comment #41) > Created attachment 1861407 [details] > dmesg output for latest test-kernel with efi=debug cmdline > > Hi Hans, > > Sorry to hear there are still issues fully resolving this bug whilst not > breaking other patches for the related bugs. Quite the cat-and-mouse game > for you it seems! > > Good news on my end, however; this latest test-kernel hasn't regressed my > touchpad to a state of not functioning again. The dmesg is attached. Great, thank you for testing; and I see this new log message which is part of the new patch: [ 0.321640] acpi PNP0A08:00: host bridge window [mem 0x65400000-0xbfffffff window] is marked by EFI as MMIO Which means that the new patch is working as it should (as is shown by your touchpad still working) so I've posted it upstream now: https://lore.kernel.org/linux-pci/20220216150121.9400-2-hdegoede@redhat.com/T/ This will hopefully finally resolve this without causing issues on other systems, thank you for your patience with this.
(In reply to Hans de Goede from comment #42) > Hi Matt, > > > Great, thank you for testing; and I see this new log message which is part > of the new patch: > > [ 0.321640] acpi PNP0A08:00: host bridge window [mem > 0x65400000-0xbfffffff window] is marked by EFI as MMIO > > Which means that the new patch is working as it should (as is shown by your > touchpad still working) so I've posted it upstream now: > > https://lore.kernel.org/linux-pci/20220216150121.9400-2-hdegoede@redhat.com/ > T/ > > This will hopefully finally resolve this without causing issues on other > systems, thank you for your patience with this. Hi Hans, No problem - happy to help if it creates a successful outcome long-term for mine and others' devices that may be experiencing this. Thanks for your perseverance in working on this bug to get to the point where (hopefully) this will result in a final stable fix. :-) Regards, Matt.
Hi Matt, Bjorn, the upstream PCI subsystem maintainer has come up with a slightly different version of my final final fix. I've done a Fedora test kernel build of 5.16.12 with the fix from Bjorn added: https://koji.fedoraproject.org/koji/taskinfo?taskID=83634323 Building is already done, so you can grab it right away. As always if you need install instructions they are here: https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Please give this a new kernel a try and please collect dmesg output after booting it and attach the dmesg output here. Thanks & Regards, Hans
Created attachment 1864198 [details] dmesg output for Bjorn's kernel patch Hi Hans, Touchpad is working as usual with this latest test kernel featuring the patches from Bjorn. Regards, Matt.
Thank you for the quick testing. The dmesg shows that Bjorn's patches are working as intended (as does the touchpad working as usual). What is interesting is that the dmesg output for the workaround triggers *twice*: [ 0.327837] acpi PNP0A08:00: resource [mem 0x000a0000-0x000bffff window] fully covered by e820 entry [mem 0x0009f000-0x000fffff] [ 0.327843] acpi PNP0A08:00: resource [mem 0x65400000-0xbfffffff window] fully covered by e820 entry [mem 0x4bc50000-0xcfffffff] The first line is about the mem-window for ISA memory-mapped io (behind a PCI to ISA bridge) getting fully clipped without the workaround, which I did not realize also was a potential issue.
Matt, Bjorn would like to acknowledge all your testing by adding a: Tested-by: Matt <xxxxx> Tag to the commit msg, using e.g. the same email as you use for your bugzilla account here, the downside of this is that it exposes your email address to the entire word. Can you please let us know if this is ok and what name (e.g. do you want your lastname added?) + email address you want us to use for this?
(In reply to Hans de Goede from comment #47) > Matt, Bjorn would like to acknowledge all your testing by adding a: > > Tested-by: Matt <xxxxx> > > Tag to the commit msg, using e.g. the same email as you use for your > bugzilla account here, the downside of this is that it exposes your email > address to the entire word. > > Can you please let us know if this is ok and what name (e.g. do you want > your lastname added?) + email address you want us to use for this? Hi Hans, Please relay my appreciation to Bjorn in recognising my help with this bug. Although, it is you (and Bjorn) who deserve the credit here; you guys did the debugging and coding. So thank you both for fixing this issue. :-) I'm happy to have my name attached to the testing tag. You can use my full name - Matt Hansen. The duck.com forwarding address I have attached to my Bugzilla account is fine. Can I get the link to the final patch commits please for reference? They're in the lore.kernel.org list archive? Also, should I update the bug status tag from new to closed or will this happen automatically when it's the right time? I'm not well-versed in the Bugzilla process. Regards, Matt.
(In reply to Matt from comment #48) > I'm happy to have my name attached to the testing tag. You can use my full > name - Matt Hansen. The duck.com forwarding address I have attached to my > Bugzilla account is fine. Ok, I've passed this info along to Bjorn. > Can I get the link to the final patch commits please for reference? They're > in the lore.kernel.org list archive? The thread discussing the patches is here: https://lore.kernel.org/linux-pci/20220304035110.988712-1-helgaas@kernel.org/ I'll provide you with a link to the final patches when they have been merged. > Also, should I update the bug status tag from new to closed or will this > happen automatically when it's the right time? If the bug gets added to the Fedora specific kernel-update changelog then it will automatically be closed, but that does not always happen. I'll keep an eye on this and close it when the patches have landed.
Unfortunately the last attempt fix this did not stick upstream because it caused regressions on some chromebooks. So there is a new patch to attempt to fix this here: https://lore.kernel.org/linux-pci/20220512202511.34197-2-hdegoede@redhat.com/ A Fedora test-kernel with these patches is building here: https://koji.fedoraproject.org/koji/taskinfo?taskID=86998438 this should be done in a couple of hours. Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Can you please give this one a test and check if the touchpad still works with this new test-kernel? When running the new kernel please run: dmesg | grep "E820 res" This should output: "PCI: ignoring E820 reservations for host bridge windows" and then your touchpad should work normally.
(In reply to Hans de Goede from comment #50) > Unfortunately the last attempt fix this did not stick upstream because it > caused regressions on some chromebooks. > Hi Hans, apologies on the delay with testing - I noticed you'd built a kernel for the newly released F36 and I hadn't yet upgraded my system. Unfortunate to hear about regressions again! > > Can you please give this one a test and check if the touchpad still works > with this new test-kernel? Can confirm touchpad functions correctly with this test-kernel. > > When running the new kernel please run: > > dmesg | grep "E820 res" > > This should output: > > "PCI: ignoring E820 reservations for host bridge windows" > > and then your touchpad should work normally. Log output is as expected: [mh@fedora ~]$ dmesg | grep "E820 res" [ 0.273196] PCI: Ignoring E820 reservations for host bridge windows [mh@fedora ~]$ I am optimistic this bug may now see final resolution. These DMI quirks for systems <2023 seem like the most reliable method to satisfy all test-cases for all systems affected by this and related PCI issues. Regards, Matt.
(In reply to Matt from comment #51) > (In reply to Hans de Goede from comment #50) > > Unfortunately the last attempt fix this did not stick upstream because it > > caused regressions on some chromebooks. > > > Hi Hans, apologies on the delay with testing - I noticed you'd built a kernel > for the newly released F36 and I hadn't yet upgraded my system. No worries and as always thank you for the testing. Note that kernels are pretty much release version independent, so you could have just installed the F36 kernel on your F35.
(In reply to Hans de Goede from comment #52) > No worries and as always thank you for the >testing. Always happy to help and see this bug through to closure. Hopefully we are close after almost 2 years now. :-D > Note that kernels are pretty much release version independent, so you could > have just installed the F36 kernel on your F35. Thanks for the tip Hans. The upgrade was planned anyway, so it was no bother. Especially these days, with the simple and stable Fedora upgrade process :-) Regards, Matt.
Hi Hans, It seems I've had a fully working touchpad for a few recent kernel releases now. Can you confirm the bug is now fully fixed? Also, if so, can you please provide the changelogs for the Fedora-specific kernel package the final patches were first applied? Have the fixes gone upstream to the Linux mainline kernel or are they RH/Fedora specific patches only? Is this bug now ready to be marked close? Regards, Matt.
Hi Matt, Good to hear that this works in standard Fedora kernels now. Yes I can confirm that this is now fully fixed. The Fedora kernels do not carry any specific patches for this. This has been fixed in the upstream / mainline kernel with this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d341838d776abadb3ac48abdd2f1f40df5a4fc10 Which is in essence the same fix as you tested earlier: https://lore.kernel.org/linux-pci/20220512202511.34197-2-hdegoede@redhat.com/ except that the addition of the use_e820 / no_e820 kernel commandline options was split out into a separate patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fa6dae5d82081e8d9f8e6a2baf7149442a6c1ba5 Regards, Hans
Hi Hans, Thanks so much for all your work on this bug and eventual patches! It's great to see the fixes accepted into mainline kernels as well. :) I'll go ahead and mark this bug as status closed. (upstream tag is best?) TC & regards, Matt.
Created attachment 1926437 [details] experimental patch to remove EfiMemoryMappedIO The current upstream resolution to this problem is https://git.kernel.org/linus/d341838d776a ("x86/PCI: Disable E820 reserved region clipping via quirks"), which relies on quirks that match DMI Vendor, Product Version, Product Name, and Board Name. This isn't an ideal solution because there are likely other systems we don't know about that need the a similar fix. The patch I'm attaching here is an experimental idea to work around this issue without the maintenance burden of the quirks. If anybody would be willing to test this patch, I would be very grateful. To test it, apply this patch to your kernel, boot with "pci=use_e820 efi=debug", and collect the dmesg log.
Hi Bjorn Happy to continue testing patches and will make an effort to apply this patch and build a new Fedora testing kernel via fedpkg within the next few days. That is unless/if Hans has planned to build one via Koji? Regards, Matt. PS. I've reopened the bug to status "MODIFIED". Please let me know if it should be changed.
(In reply to Matt from comment #58) > Hi Bjorn > > Happy to continue testing patches and will make an effort to apply this > patch and build a new Fedora testing kernel via fedpkg within the next few > days. > That is unless/if Hans has planned to build one via > Koji? Hi Matt, I really appreciate your willingness to help Bjorn test alternative methods of fixing this. Unfortunately my workload does not allow me to go and do koji scratch-builds for this. I would likely only be getting in the way of any dialog between you and Bjorn because I would cause you to wait for a long time before I get around to submitting any koji kernel test builds. So it would be best if you just build the kernel locally with fedpkg. Regards, Hans
Created attachment 1927740 [details] dmesg output for remove-mmio patch Good morning all, Hans, no problem - I managed to build a local kernel via fedpkg. That process was quite an experience as I've never built a kernel package via fedpkg before and it took quite a lot of tinkering with the git cloning process due to some possibly network-related issues that caused the process to fail with error: fetch-pack: unexpected disconnect while reading sideband packetB/s fatal: early EOF fatal: fetch-pack: invalid index-pack output Seems to be a potential Fedora Infrastructure problem (https://pagure.io/fedora-infrastructure/issue/9677). I managed to work-around the issue with a partial clone followed by an unshallow command and then fetching all branches. Have you ever come across this type of problem before with fedpkg? Bjorn, I've booted into a test kernel with the two required kernel command-line args and attached the dmesg here. Touchpad functions as expected. Your patch seems to have been applied correctly as seen in the dmesg showing your newly added kernel boot info: [ 0.000000] efi: removing MMIO range=[0x0000000065400000-0x00000000cfffffff] (1708MB) from E820 reservations [ 0.000000] e820: remove [mem 0x65400000-0xcfffffff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000fc800000-0x00000000fe7fffff] (32MB) from E820 reservations [ 0.000000] e820: remove [mem 0xfc800000-0xfe7fffff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000fed00000-0x00000000fed00fff] (0MB) from E820 reservations [ 0.000000] e820: remove [mem 0xfed00000-0xfed00fff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000fed10000-0x00000000fed17fff] (0MB) from E820 reservations [ 0.000000] e820: remove [mem 0xfed10000-0xfed17fff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000feda0000-0x00000000feda1fff] (0MB) from E820 reservations [ 0.000000] e820: remove [mem 0xfeda0000-0xfeda1fff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000fee00000-0x00000000fee00fff] (0MB) from E820 reservations [ 0.000000] e820: remove [mem 0xfee00000-0xfee00fff] reserved [ 0.000000] efi: removing MMIO range=[0x00000000ff600000-0x00000000ffffffff] (10MB) from E820 reservations [ 0.000000] e820: remove [mem 0xff600000-0xffffffff] reserved Hopefully I've done this all correctly! Let me know what else you may need for debugging purposes. Regards, Matt.
Thank you very much, Matt! The fact that the touchpad works is very promising. I do want to figure out these messages because they really look bogus and should be improved or removed: [ 0.413903] clipped [mem size 0x00000000 64bit] to [mem size 0xfffffffffffa0000 64bit] for e820 entry [mem 0x0009f000-0x000fffff]
(In reply to Bjorn Helgaas from comment #61) > Thank you very much, Matt! The fact that the touchpad works is very > promising. I do want to figure out these messages because they really look > bogus and should be improved or removed: > > [ 0.413903] clipped [mem size 0x00000000 64bit] to [mem size > 0xfffffffffffa0000 64bit] for e820 entry [mem 0x0009f000-0x000fffff] No problem at all Bjorn. Feel free to provide another patch if you decide to alter/remove the message output mentioned above and need further testing. -Matt
Created attachment 1929651 [details] dmesg output for latest Bjorn patchset(4/4) Hi Bjorn, I've built a 6.0.11 based Fedora kernel with the 4 patches applied. Great news - the touchpad remains functioning as expected and the dmesg output looks great with the new or cleaned-up message formats. :) Regards, Matt PS, my Duck remailer seems to have trouble replying to your email list, so I haven't cross-posted this there.
Matt, Thank you for testing. I'm a bit worried that Bjorn's patches are not going to work in BIOS (CSM) compatibility mode (which quite a few Linux users use mostly due to inertia). IIRC last time I dived into this the troublesome memory reservations also showed up in the E820 memory map when in CSM mode and in that case the E820 map is actually made by the BIOS and we cannot tell the difference between reservations which come from EfiMemoryMappedIO and other reservations. To confirm this, can you boot a Fedora 37 live USB in BIOS compatibility mode and then collect dmesg output ? I think just getting the dmesg output with a standard F37 kernel from the live USB should give us enough info to confirm (or deny) my worries about Born's patches not helping for the BIOS (CSM) compatibility mode. Regards, Hans
(In reply to Hans de Goede from comment #64) > Matt, > > Thank you for testing. I'm a bit worried that Bjorn's patches are not going > to work in BIOS (CSM) compatibility mode (which quite a few Linux users use > mostly due to inertia). > > IIRC last time I dived into this the troublesome memory reservations also > showed up in the E820 memory map when in CSM mode and in that case the E820 > map is actually made by the BIOS and we cannot tell the difference between > reservations which come from EfiMemoryMappedIO and other reservations. > > To confirm this, can you boot a Fedora 37 live USB in BIOS compatibility > mode and then collect dmesg output ? > > I think just getting the dmesg output with a standard F37 kernel from the > live USB should give us enough info to confirm (or deny) my worries about > Born's patches not helping for the BIOS (CSM) compatibility mode. > > Regards, > > Hans Hi Hans, Yeah, I'll test that tomorrow no worries. I'll have to download a F37 image as I think my live USB still contains F35. :D I'll update with a dmesg as requested. Regards, Matt PS Is there a way to speed up kernel package building with fedpkg? Perhaps a way to not build all the auxillary kernel rpms such as kernel-debug*, kernel-devel* or the extra kernel-modules-* ones? Limit the build to just kernel/kernel-core/kernel-modules?
Created attachment 1929753 [details] dmesg output while booting in Legacy BIOS/CSM mode via LiveUSB Good morning Hans, I've booted into a F37 LiveUSB system after setting the boot to "Legacy BIOS" mode. The dmesg seems to suggest it's using the CSM as there's no usual references to "efi" in the output. Touchpad still functions. Hope this helps. Regards, Matt.
(In reply to Matt from comment #66) > Created attachment 1929753 [details] > dmesg output while booting in Legacy BIOS/CSM mode via LiveUSB > > Good morning Hans, > > I've booted into a F37 LiveUSB system after setting the boot to "Legacy > BIOS" mode. > The dmesg seems to suggest it's using the CSM as there's no usual references > to "efi" in the output. Touchpad still functions. Hope this helps. > > Regards, Matt. Hi Matt, Thank you. So as I was afraid Bjorn's patches (which are an attempt to fix the issue in a generic way) won't work when booting the system in "Legacy BIOS" mode. Note this does not affect your system since your system has a model-string matching based quirk to avoid the problem already, so it will work even in CSM mode. But it does mean that Bjorn's patches won't help for as of yet unknown models with this issue when those are booted in CSM mode, which is good to know. Regards, Hans
Matt, Bjorn has posted a v2 of his patch series for this. I have started a test Fedora kernel build with the v2 patches added: https://koji.fedoraproject.org/koji/taskinfo?taskID=95100621 Note this is still building atm. It should be finished in a couple of hours. It would be great if you can give this a test spin once it has finished building. Like with the kernel you build yourself, please collect a dmesg with "pci=use_e820 efi=debug" added to the kernel commandline Your touchpad should keep working with this (even if you pass "pci=use_e820" to override the no_e820 quirk for your model). I don't think you need these anymore, but just incase here are some generic instructions for installing a kernel directly from koji (Fedora's buildsystem): https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt Regards, Hans
Created attachment 1931253 [details] dmesg output for v2 of Bjorn's patchset(4/4) Hi Hans, Thanks for building a test-kernel for this v2 patchset. I enjoyed undertaking the kernel building and patch-applying process, but having one pre-built and ready to install is much easier. :) Dmesg attached with the required kernel cmdline args. Bjorn has CC'd me into the current email-list discussion as well, which is interesting to follow. Regards, Matt.
This message is a reminder that Fedora Linux 36 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '36'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 36 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
I think this issue should be resolved by 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") (https://git.kernel.org/linus/07eab0901ede), which appeared in v6.2. Let me know if you still see this issue with a kernel that includes 07eab0901ede.
(In reply to Bjorn Helgaas from comment #71) > I think this issue should be resolved by 07eab0901ede ("efi/x86: Remove > EfiMemoryMappedIO from E820 map") > (https://git.kernel.org/linus/07eab0901ede), which appeared in v6.2. > > Let me know if you still see this issue with a kernel that includes > 07eab0901ede. Hi Bjorn, Yes, I can confirm this issue has been fixed for me in recent kernels. :) Thanks to Hans and Bjorn for all the great work in resolving this issue. It's great to also see it incorporated as a generic solution into the upstream kernel tree without having to rely on any Fedora-specific patches. It's been a fun experience helping test patches and provide debugging data over the last almost 3 years. :) I'll go ahead and close this bug with resolution: upstream. Regards, Matt.