Bug 1868899 - Touchpad not working on Lenovo Ideapad Slim 3
Summary: Touchpad not working on Lenovo Ideapad Slim 3
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 36
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-14 07:05 UTC by Matt
Modified: 2023-04-26 12:38 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-26 12:38:44 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
kernel logs (84.29 KB, text/plain)
2020-08-14 07:05 UTC, Matt
no flags Details
dmesg output (75.27 KB, text/plain)
2021-06-20 05:52 UTC, Matt
no flags Details
dmesg output (with pci cmdline arg) (75.41 KB, text/plain)
2021-06-27 07:11 UTC, Matt
no flags Details
dmesg output (without pci cmdline arg) (77.03 KB, text/plain)
2021-06-27 07:13 UTC, Matt
no flags Details
dmesg output for 5.12.13-300 (85.55 KB, text/plain)
2021-07-01 12:41 UTC, Matt
no flags Details
dmesg output for 5.12.13-300 (94.68 KB, text/plain)
2021-07-01 13:15 UTC, Matt
no flags Details
dmesg output with no_e820 pci cmdline arg (74.50 KB, text/plain)
2021-10-05 08:05 UTC, Matt
no flags Details
dmesg output with efi=debug cmdline addition (96.96 KB, text/plain)
2022-02-14 07:17 UTC, Matt
no flags Details
dmesg output for new test-kernel with efi=debug cmdline (87.18 KB, text/plain)
2022-02-15 07:52 UTC, Matt
no flags Details
dmesg output for latest test-kernel with efi=debug cmdline (88.54 KB, text/plain)
2022-02-16 07:03 UTC, Matt
no flags Details
dmesg output for Bjorn's kernel patch (75.16 KB, text/plain)
2022-03-05 00:49 UTC, Matt
no flags Details
experimental patch to remove EfiMemoryMappedIO (1.35 KB, patch)
2022-11-22 15:59 UTC, Bjorn Helgaas
no flags Details | Diff
dmesg output for remove-mmio patch (89.29 KB, text/plain)
2022-11-26 23:52 UTC, Matt
no flags Details
dmesg output for latest Bjorn patchset(4/4) (89.02 KB, text/plain)
2022-12-03 10:43 UTC, Matt
no flags Details
dmesg output while booting in Legacy BIOS/CSM mode via LiveUSB (76.61 KB, text/plain)
2022-12-04 00:53 UTC, Matt
no flags Details
dmesg output for v2 of Bjorn's patchset(4/4) (89.63 KB, text/plain)
2022-12-09 07:56 UTC, Matt
no flags Details

Description Matt 2020-08-14 07:05:32 UTC
Created attachment 1711424 [details]
kernel logs

1. Please describe the problem:
Touchpad does not work at all. For reference, hardware is functional as touchpad works under Win10. USB and BT mice work. 

output of system info:
# dmidecode 
System Information
	Manufacturer: LENOVO
	Product Name: 81WE
	Version: IdeaPad 3 15IIL05
	Serial Number: PF25BWBN
	UUID: 58961da7-bdb8-2020-0425-103601000000
	Wake-up Type: Power Switch
	SKU Number: LENOVO_MT_81WE_BU_idea_FM_IdeaPad 3 15IIL05
	Family: IdeaPad 3 15IIL05



2. What is the Version-Release number of the kernel:
5.7.14-200.fc32.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
No previous Fedora releases tested - F32 only.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
Boot laptop to desktop Wayland session. Touchpad non-functional at any time since F32 installation.


5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
Not tested rawhide due to GPG key issue using the repo, however problem exists in latest updates-testing kernel 5.7.15-200.fc32.x86_64.


6. Are you running any modules that not shipped with directly Fedora's kernel?:
No


7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
Attached.


I believe the following info may be relevant from dmesg? This happens every boot and is reported as an kerneloops in kernel-core package via ABRT:
[    9.820184] irq 9: nobody cared (try booting with the "irqpoll" option)
[    9.820186] CPU: 1 PID: 844 Comm: rngd Not tainted 5.7.14-200.fc32.x86_64 #1
[    9.820186] Hardware name: LENOVO 81WE/LNVNB161216, BIOS EMCN37WW 06/24/2020
[    9.820187] Call Trace:
[    9.820189]  <IRQ>
[    9.820194]  dump_stack+0x64/0x88
[    9.820197]  __report_bad_irq+0x35/0xa7
[    9.820198]  note_interrupt.cold+0xb/0x6a
[    9.820199]  handle_irq_event+0x88/0x8a
[    9.820201]  handle_fasteoi_irq+0x78/0x1c0
[    9.820203]  do_IRQ+0x50/0xe0
[    9.820205]  common_interrupt+0xf/0xf
[    9.820206]  </IRQ>
[    9.820207] RIP: 0033:0x7f01133a49df
[    9.820208] Code: 48 8b 45 e0 48 c1 e8 1b 83 e0 01 48 31 45 f0 48 8b 45 e0 48 c1 e8 16 83 e0 01 48 31 45 f0 48 d1 65 e0 48 8b 45 f0 48 31 45 e0 <83> 45 d4 01 83 7d d4 40 0f 86 72 ff ff ff 48 83 45 d8 01 48 8b 45
[    9.820209] RSP: 002b:00007f0111cb0bc0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffde
[    9.820210] RAX: 0000000000000001 RBX: 000000000000000a RCX: 000000000000002e
[    9.820210] RDX: 099c400000000000 RSI: 0000000000000000 RDI: 0000000000000001
[    9.820211] RBP: 00007f0111cb0c10 R08: 0000000000000000 R09: 0000000000000035
[    9.820211] R10: 00007f0111cb0a77 R11: 0000000000000000 R12: 00007f00fc004c00
[    9.820211] R13: 00007fff3323cf5f R14: 00007f0111cb0d10 R15: 000055d8d442b4c0
[    9.820213] handlers:
[    9.820215] [<0000000094f280d8>] acpi_irq
[    9.820216] Disabling IRQ #9

Comment 1 Matt 2020-09-02 01:07:49 UTC
FYI, latest kernel 5.8.4-200.fc32 of the new 5.8 branch still contains this touchpad issue. 
Are there any temporary fixes or official patches forthcoming to support this hardware 
functionality available here or upstream?

Comment 2 Matt 2020-12-06 09:37:18 UTC
Updated Fedora version to the current 33rd edition as problem still exists.

Comment 3 Matt 2021-05-29 05:33:15 UTC
Updated to latest Fedora release 34.

Is there a timeline that could be conjected by anyone cc'd in this bug to 
illuminate when one on average sees hardware such as this touchpad reach 
supported status in Fedora once a new device is released to market?

Are there some proactive steps I could take to expedite the process toward 
driver support for this hardware in Fedora?

Comment 4 Hans de Goede 2021-05-30 11:12:46 UTC
(In reply to Matt from comment #3)
> Updated to latest Fedora release 34.
> 
> Is there a timeline that could be conjected by anyone cc'd in this bug to 
> illuminate when one on average sees hardware such as this touchpad reach 
> supported status in Fedora once a new device is released to market?
> 
> Are there some proactive steps I could take to expedite the process toward 
> driver support for this hardware in Fedora?

Sorry that it is taking so long to resolve this issue. I've been working on fixing touchpad issues on various Lenovo Ideapad laptops, but there are multiple issues at play. Twice now I thought I had fixed things and I did fix things for some Ideapad models, but others still do not work.

Let me copy and paste a large comment which I added to another bug. I'll put this in a separate comment here for clarity.

Comment 5 Hans de Goede 2021-05-30 11:13:16 UTC
So this is a bit of a mess which I'm trying to sort out ATM, there are at least 4 different issues with Ideapad touchpads that I'm aware of:

1. There was a Linux ACPI issue where the resource-list (_CRS) for the touchpad given to Linux was not correct, this has been fixed for a while now by these 2 commits from me:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=21653a4181ff292480599dad996a2b759ccf050f
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8058d69905058ec8f467a120b5ec5bb831ea67f3
These commits should be in recent Ubuntu kernels so if you are still seeing issues then this issue was not the root cause for your laptop model.

2. On some Ideapad models the elants_i2c driver takes control of handling the touchpad, but the touchpad does not talk the elants protocol at all, it uses the standard I2C-HID protocol so this does not work. If adding "initcall_blacklist=elants_i2c_driver_init" to your kernel commandline fixes things, then your Ideapad model is hit by this. This is fixed now by this commit from me:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=65299e8bfb24774e6340e93ae49f6626598917c8
This commit was added to the 5.10.y and 5.12.y stable kernel series a couple of days ago, so hopefully it will get added to the Ubuntu kernels soon too.

3. On some Ideapads the BIOS seems to corrupt its own settings (often after a BIOS update) and as a result of this the BIOS is returning bogus addresses (NULL or 0xffffffff) as base-address from the ACPI _CRS call for some PCI devices, including the I2C controller connected to the touchpad. If adding "pci=nocrs" as a workaround helps then your model Ideapad is affected by this issue.

4. On some Ideapads something goes wrong with the OSYS code in the ACPI tables, this code is responsible for detecting which Windows version the ACPI tables think they are dealing with (or which Windows version Linux mimicks in case you are running Linux). To see if your Ideapad is affected by this issue try running:
"cat /sys/bus/acpi/devices/XXXX0000\:00/status"
from a terminal, if the output of this command is "15" then your model is affected by this issue. At least the "Lenovo Thinkbook 14-ILL" is known to be affected by this. The exact root cause of this is unknown atm. If your laptop is affected by this please run "acpidump -o acpidump" and attach the acpidump here, in combination with specifying the exact model of your laptop.

Comment 6 Hans de Goede 2021-05-30 11:20:29 UTC
Translating this to your situation / laptop:

1. Is fixed in Fedora 34, so you are not hit by 1.

2. Is fixed in kernel 5.12.6, 5.12.7 is in the Fedora 34 updates repo now, so if you run "sudo dnf update 'kernel*'" then you should get a kernel >= 5.12.6 installed. If you reboot after this and things work now, then 2. was the root cause and you can stop reading here :)

3. To test if you are hit by this run:
"sudo grubby --update-kernel=ALL --args=pci=nocrs"
and then reboot. If this works around the issue please let me know, I'm still investigating this variant of the problem.
If this does not work, run the following command to undo the change:
"sudo grubby --update-kernel=ALL --remove-args=pci=nocrs"

4. As the comment says (and if 1-3 do not help), try running:
"cat /sys/bus/acpi/devices/XXXX0000\:00/status"
from a terminal, if the output of this command is "15" then your model is affected by this issue

Comment 7 Matt 2021-06-13 07:26:39 UTC
(In reply to Hans de Goede from comment #6)
> Translating this to your situation / laptop:
> 
> <snip>
> 
> 3. To test if you are hit by this run:
> "sudo grubby --update-kernel=ALL --args=pci=nocrs"
> and then reboot. If this works around the issue please let me know, I'm
> still investigating this variant of the problem.
> If this does not work, run the following command to undo the change:
> "sudo grubby --update-kernel=ALL --remove-args=pci=nocrs"

Hi Hans, 

Apologies for the delay in responding. I can happily confirm this 
3rd workaround fixes the problem for my laptop touchpad functionality! :-)
I also tested gestures and the 2-fingered combo works however the new 3-finger 
Gnome 40 gesturing doesn't seem to work. (perhaps I'm just not doing it correctly) 
> 
> 4. As the comment says (and if 1-3 do not help), try running:
> "cat /sys/bus/acpi/devices/XXXX0000\:00/status"
> from a terminal, if the output of this command is "15" then your model is
> affected by this issue
Note, this device path doesn't exist on my system. Is this out of the oridinary?

-Matt

Comment 8 Hans de Goede 2021-06-15 10:13:39 UTC
(In reply to Matt from comment #7)
> Apologies for the delay in responding. I can happily confirm this 
> 3rd workaround fixes the problem for my laptop touchpad functionality! :-)

That is good news. I think I know what is going on here and I've written a kernel fix which should make things work without the workaround.

I've build a test-kernel with the fix which I wrote added:
https://koji.fedoraproject.org/koji/taskinfo?taskID=70106533

Here are some generic instructions for installing a kernel directly from koji (the Fedora buildsystem):
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Can you please undo the workaround by running:

sudo grubby --update-kernel=ALL --remove-args=pci=nocrs

And then give this new kernel a try. After booting the new kernel please run:

dmesg > dmesg-patched-kernel.txt

And attach the generated dmesg-patched-kernel.txt. Also please let me know if the touchpad still works with the patches kernel without the workaround.

> > 4. As the comment says (and if 1-3 do not help), try running:
> > "cat /sys/bus/acpi/devices/XXXX0000\:00/status"
> > from a terminal, if the output of this command is "15" then your model is
> > affected by this issue
> Note, this device path doesn't exist on my system. Is this out of the
> oridinary?

The path NOT existing is normal. If the path exists then that actually is indicative of another bug which is also causing touchpads to not work on some models.

Comment 9 Matt 2021-06-20 05:52:33 UTC
Created attachment 1792465 [details]
dmesg output

Comment 10 Matt 2021-06-20 05:56:54 UTC
(In reply to Hans de Goede from comment #8)
> (In reply to Matt from comment #7)
> > Apologies for the delay in responding. I can happily confirm this 
> > 3rd workaround fixes the problem for my laptop touchpad functionality! :-)
> 
> That is good news. I think I know what is going on here and I've written a
> kernel fix which should make things work without the workaround.
> 
> I've build a test-kernel with the fix which I wrote added:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=70106533
> 
> Here are some generic instructions for installing a kernel directly from
> koji (the Fedora buildsystem):
> https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt
> 
> Can you please undo the workaround by running:
> 
> sudo grubby --update-kernel=ALL --remove-args=pci=nocrs
> 
> And then give this new kernel a try. After booting the new kernel please run:
> 
> dmesg > dmesg-patched-kernel.txt
> 
> And attach the generated dmesg-patched-kernel.txt. Also please let me know
> if the touchpad still works with the patches kernel without the workaround.

Hi Hans,

Unfortunately this patched kernel does not fix the bug for my touchpad. The 
dmesg output is attached.

Regards, Matt

Comment 11 Hans de Goede 2021-06-21 11:36:48 UTC
Thank you, I've been discussing this problem upstream and my initial patch was no good, sorry about that.

We think we have some idea what is going on, but we need some more info. So I've prepared another test kernel with a patch which prints some extra debugging included:
https://koji.fedoraproject.org/koji/taskinfo?taskID=70539682

ATM this is still building, this takes a couple of hours.

Can you please install this kernel once it is done building and collect dmesg output both with and without "pci=nocrs" on the kernel commandline; and then attach both dmesg files here ?   Note this is not expected to work without "pci=nocrs", the goal is purely to gather some information.

Comment 12 Matt 2021-06-27 07:11:16 UTC
Created attachment 1794948 [details]
dmesg output (with pci cmdline arg)

Comment 13 Matt 2021-06-27 07:13:31 UTC
Created attachment 1794960 [details]
dmesg output (without pci cmdline arg)

Comment 14 Matt 2021-06-27 07:18:41 UTC
(In reply to Hans de Goede from comment #11)
> Thank you, I've been discussing this problem upstream and my initial patch
> was no good, sorry about that.
> 
> We think we have some idea what is going on, but we need some more info. So
> I've prepared another test kernel with a patch which prints some extra
> debugging included:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=70539682
> 
> ATM this is still building, this takes a couple of hours.
> 
> Can you please install this kernel once it is done building and collect
> dmesg output both with and without "pci=nocrs" on the kernel commandline;
> and then attach both dmesg files here ?   Note this is not expected to work
> without "pci=nocrs", the goal is purely to gather some information.

Hi Hans, I've attached the two dmesg files as requested. HTH the debugging and patching
process.

-Matt

Comment 15 Hans de Goede 2021-06-30 13:43:12 UTC
Thank you for the logs, here is another test-kernel with more debugging added:
https://koji.fedoraproject.org/koji/taskinfo?taskID=71078980

ATM this is still building, this takes a couple of hours.

Can you please install this kernel once it is done building and collect dmesg output with "log_buf_len=50M pci=nocrs" on the kernel commandline ?

Comment 16 Matt 2021-07-01 12:41:32 UTC
Created attachment 1796782 [details]
dmesg output for 5.12.13-300

Hi Hans, dmesg attached. HTH.

-Matt.

Comment 17 Hans de Goede 2021-07-01 12:55:31 UTC
Matt, thank you.

I'm afraid I made a mistake I actually need the dmesg output without the "pci=nocrs", so with just "log_buf_len=50M", note the log_buf_len is probably not necessary but better safe then sorry.

If you can grab the dmesg without the "pci=nocrs", that would be great.

Comment 18 Matt 2021-07-01 13:15:09 UTC
Created attachment 1796799 [details]
dmesg output for 5.12.13-300

(In reply to Hans de Goede from comment #17)
> Matt, thank you.
> 
> I'm afraid I made a mistake I actually need the dmesg output without the
> "pci=nocrs", so with just "log_buf_len=50M", note the log_buf_len is
> probably not necessary but better safe then sorry.
> 
> If you can grab the dmesg without the "pci=nocrs", that would be great.

No worries Hans. I've attached the updated dmesg without the pci kernel arg. 

-Matt

Comment 19 Hans de Goede 2021-07-01 14:26:51 UTC
Thanks, the last set of logs has helped to finally understand what is going on here.

Your BIOS claims that any memory between 0x000000004bc50000-0x00000000cfffffff is reserved:

[    0.000000] BIOS-e820: [mem 0x000000004bc50000-0x00000000cfffffff] reserved

But then continuous with placing the iomem window for PCI devices inside that reserved range:

[    0.609988] pci_bus 0000:00: root bus resource [mem 0x65400000-0xbfffffff window]

This confuses the kernel and makes it unable to assign PCI iomem to devices which have not already been assigned iomem by the BIOS, like the I2C-controller used for the touchpad.

If you are interested in / curious about the low level details, I'm discussing this with the kernel PCI developers (including figuring out how to deal with this) here:
https://lore.kernel.org/linux-pci/7d80f639-0768-850b-5313-3bdedf0a5579@redhat.com/

Comment 20 Matt 2021-07-02 09:35:34 UTC
(In reply to Hans de Goede from comment #19)
> Thanks, the last set of logs has helped to finally understand what is going
> on here.
> 
> Your BIOS claims that any memory between
> 0x000000004bc50000-0x00000000cfffffff is reserved:
> 
> [    0.000000] BIOS-e820: [mem 0x000000004bc50000-0x00000000cfffffff]
> reserved
> 
> But then continuous with placing the iomem window for PCI devices inside
> that reserved range:
> 
> [    0.609988] pci_bus 0000:00: root bus resource [mem 0x65400000-0xbfffffff
> window]
> 
> This confuses the kernel and makes it unable to assign PCI iomem to devices
> which have not already been assigned iomem by the BIOS, like the
> I2C-controller used for the touchpad.
> 
> If you are interested in / curious about the low level details, I'm
> discussing this with the kernel PCI developers (including figuring out how
> to deal with this) here:
> https://lore.kernel.org/linux-pci/7d80f639-0768-850b-5313-
> 3bdedf0a5579/

Hi Hans, thanks for the informative summary of the low-level details from the 
discussion at the linux-pci list. It's always interesting to read about a bug 
being unravelled and eventually squashed. 

I will try to respond promptly over the next few days if further logs/data
are required.

-Matt

Comment 21 Hans de Goede 2021-10-04 16:05:36 UTC
Hi,

Sorry for the long silence on this bug. I was hoping one of the upstream pci-subsys devs would help out with this...

Since that did not happen I've been looking into fixing this myself now. I've come up with a solution which still involves a kernel cmdline option for now, but this one is a much smaller hammer (almost a scalpel) and once it is confirmed that this works I can enable this by default through a DMI quirk.

I've build a test-kernel with support for the new kernel cmdline option here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=76687852

Here are some generic instructions for installing a kernel directly from koji (the Fedora buildsystem):
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Please undo the pci=nocrs workaround by running:

sudo grubby --update-kernel=ALL --remove-args=pci=nocrs

And then enable the new workaround by:

sudo grubby --update-kernel=ALL --args=pci=no_e820

Also please run the following command *as normal user* to collect the DMI strings for your laptop, so that they can be used to add a quirk:

grep . /sys/class/dmi/id/* 2> /dev/null

And copy and paste the output here.

Comment 22 Hans de Goede 2021-10-04 16:12:11 UTC
p.s.

Please also provide dmesg output for the new kernel after booting it with pci=no_e820 on the kernel cmdline.

Comment 23 Matt 2021-10-05 08:02:19 UTC
(In reply to Hans de Goede from comment #21)
> Hi,
> 
> Sorry for the long silence on this bug. I was hoping one of the upstream
> pci-subsys devs would help out with this...

Since the nocrs cmdline arg solution has been available to use as a workaround, 
there's no problems with waiting. :)

> Since that did not happen I've been looking into fixing this myself now.
> I've come up with a solution which still involves a kernel cmdline option
> for now, but this one is a much smaller hammer (almost a scalpel) and once
> it is confirmed that this works I can enable this by default through a DMI
> quirk.

Can confirm the more refined no_e820 cmdline arg works and imo it seems to 
offer a more responsive/smoother engagement when using the touchpad.

 
> Also please run the following command *as normal user* to collect the DMI
> strings for your laptop, so that they can be used to add a quirk:
> 
> grep . /sys/class/dmi/id/* 2> /dev/null
> 
> And copy and paste the output here.

/sys/class/dmi/id/bios_date:12/23/2020
/sys/class/dmi/id/bios_release:1.44
/sys/class/dmi/id/bios_vendor:LENOVO
/sys/class/dmi/id/bios_version:EMCN44WW
/sys/class/dmi/id/board_asset_tag:NO Asset Tag
/sys/class/dmi/id/board_name:LNVNB161216
/sys/class/dmi/id/board_vendor:LENOVO
/sys/class/dmi/id/board_version:SDK0J40700 WIN
/sys/class/dmi/id/chassis_asset_tag:NO Asset Tag
/sys/class/dmi/id/chassis_type:10
/sys/class/dmi/id/chassis_vendor:LENOVO
/sys/class/dmi/id/chassis_version:IdeaPad 3 15IIL05
/sys/class/dmi/id/ec_firmware_release:1.44
/sys/class/dmi/id/modalias:dmi:bvnLENOVO:bvrEMCN44WW:bd12/23/2020:br1.44:efr1.44:svnLENOVO:pn81WE:pvrIdeaPad315IIL05:rvnLENOVO:
rnLNVNB161216:rvrSDK0J40700WIN:cvnLENOVO:ct10:cvrIdeaPad315IIL05:skuLENOVO_MT_81WE_BU_idea_FM_IdeaPad315IIL05:
/sys/class/dmi/id/product_family:IdeaPad 3 15IIL05
/sys/class/dmi/id/product_name:81WE
/sys/class/dmi/id/product_sku:LENOVO_MT_81WE_BU_idea_FM_IdeaPad 3 15IIL05
/sys/class/dmi/id/product_version:IdeaPad 3 15IIL05
/sys/class/dmi/id/sys_vendor:LENOVO
/sys/class/dmi/id/uevent:MODALIAS=dmi:bvnLENOVO:bvrEMCN44WW:bd12/23/2020:br1.44:efr1.44:svnLENOVO:pn81WE:pvrIdeaPad315IIL05:rvnLENOVO:
rnLNVNB161216:rvrSDK0J40700WIN:cvnLENOVO:ct10:cvrIdeaPad315IIL05:skuLENOVO_MT_81WE_BU_idea_FM_IdeaPad315IIL05:


Cheers, Matt.

Comment 24 Matt 2021-10-05 08:05:25 UTC
Created attachment 1829283 [details]
dmesg output with no_e820 pci cmdline arg

Comment 25 Hans de Goede 2021-10-05 09:55:47 UTC
Matt, thank you for the quick turn-around time on testing this.

I've one more kernel for you to test. This automatically activates the pci=no_e820 behavior on your model laptop, so this one should work without needing any kernel cmdline options at all:

https://koji.fedoraproject.org/koji/taskinfo?taskID=76736063

(note this is still building ATM)

After booting this (without special kernel cmdline options), 'dmesg | grep "E820 reservations"' should still show:

[    0.558144] PCI: Ignoring E820 reservations for host bridge windows

As it does with yesterdays test kernel when adding pci=no_e820 to the kernel cmdline.

And your touchpad should still work :)

If you can confirm this then I'll submit the new version with the DMI quirk to automatically enable this upstream and then cross my fingers that my workaround for this is acceptable upstream.

I'll also try to get testing feedback from people seeing the same issue on a "Lenovo IdeaPad 5 14IIL05" and other related models. Hopefully we will be able to fix this for those other models by adding DMI quirks for them too.

Thank you very much for all your help with testing things to get this resolved.

Comment 26 Matt 2021-10-05 12:13:49 UTC
(In reply to Hans de Goede from comment #25)

> After booting this (without special kernel cmdline options), 'dmesg | grep
> "E820 reservations"' should still show:
> 
> [    0.558144] PCI: Ignoring E820 reservations for host bridge windows
> 
> As it does with yesterdays test kernel when adding pci=no_e820 to the kernel
> cmdline.
> 
> And your touchpad should still work :)
> 
> If you can confirm this then I'll submit the new version with the DMI quirk
> to automatically enable this upstream and then cross my fingers that my
> workaround for this is acceptable upstream.

Hi Hans,

I'm happy to confirm your latest kernel build with the new DMI quirk works as 
expected without the cmdline arg and the dmesg output still includes the above
message regarding ignoring E820 reservations. :)

> I'll also try to get testing feedback from people seeing the same issue on a
> "Lenovo IdeaPad 5 14IIL05" and other related models. Hopefully we will be
> able to fix this for those other models by adding DMI quirks for them too.
> 
> Thank you very much for all your help with testing things to get this
> resolved.

Happy to help - we got there in the end! :) Thanks for your work on this bug and
finding a solution. Hopefully it will be suitable upstream and will be useful for 
applying a fix to similar laptops to mine.

Regards, Matt.

Comment 27 Hans de Goede 2021-10-05 15:16:13 UTC
Thank you, I've submitted the patch upstream now:

https://lore.kernel.org/linux-pci/20211005150956.303707-1-hdegoede@redhat.com/T/#u

Comment 28 Hans de Goede 2021-10-11 09:36:38 UTC
Hi Matt,

Upstream review/discussion has noted that many different models are impacted by this problem. And the conclusion is that the no_e820 behavior should be the default, but to avoid causing regressions with some older boards which needed the use_e820 behavior the plan (my plan) for now is to keep the old use_e820 behavior for any devices where the cat /sys/class/dmi/id/bios_date year is less then 2018 (so the latest BIOS update is from 2017 or older). This replaces the per model DMI matching from my previous patch.

Leading to this new version of the patch:
https://lore.kernel.org/linux-pci/20211011090531.244762-1-hdegoede@redhat.com/T/#u

I've started a new scratch/test-kernel build with the new version of the patch:
https://koji.fedoraproject.org/koji/taskinfo?taskID=77061333

(note this is still building ATM)

After booting this (without special kernel cmdline options), 'dmesg | grep "E820 reservations"' should still show:

[    0.558144] PCI: Ignoring E820 reservations for host bridge windows

as before and your touchpad should still work :)

If you can find some time to test this and confirm that I did not break things, that would be great.

Regards,

Hans

Comment 29 Matt 2021-10-12 07:15:03 UTC
(In reply to Hans de Goede from comment #28)
> Hi Matt,
> 
> Upstream review/discussion has noted that many different models are impacted
> by this problem. And the conclusion is that the no_e820 behavior should be
> the default, but to avoid causing regressions with some older boards which
> needed the use_e820 behavior the plan (my plan) for now is to keep the old
> use_e820 behavior for any devices where the cat /sys/class/dmi/id/bios_date
> year is less then 2018 (so the latest BIOS update is from 2017 or older).
> This replaces the per model DMI matching from my previous patch.

Hi Hans,

This BIOS date matching logic definitely seems like the more efficient way to 
scale the application of this patch. :)

> 
> After booting this (without special kernel cmdline options), 'dmesg | grep
> "E820 reservations"' should still show:
> 
> [    0.558144] PCI: Ignoring E820 reservations for host bridge windows
> 
> as before and your touchpad should still work :)
> 
> If you can find some time to test this and confirm that I did not break
> things, that would be great.
> 
> Regards,
> 
> Hans

The dmesg output still correctly logs the E820 reservations as being ignored
and touchpad functions as expected - you did not break things. :)

Regards, 
Matt.

Comment 30 Hans de Goede 2021-10-12 07:55:19 UTC
Great, thank you for testing!

Comment 31 Hans de Goede 2022-02-08 15:41:02 UTC
Hi Matt,

Unfortunately it seems that my fix to ignore e820 reservations for PCI bar allocations to fix the touchpad issue on your laptop is causing issues on other machines, see bug 2029207 and:

https://lore.kernel.org/linux-pci/a7ad05fe-c2ab-a6d9-b66e-68e8c5688420@redhat.com/

So I have to come up with a different fix.

Can you please let me know if you will be able to test new test-kernel-builds with a different fix for me (once I have them ready) ?

Regards,

Hans

Comment 32 Hans de Goede 2022-02-09 15:16:37 UTC
I'm still discussing alternative fixes upstream. To figure things out, it would really help if you can boot your Lenovo Ideapad Slim 3 with "efi=debug" added to the kernel commandline and then directly after booting do:

dmesg > dmesg-efi-debug.txt

And attach the dmesg-efi-debug.txt file here, it does not matter which kernel you do this with.

Comment 33 Hans de Goede 2022-02-09 17:15:12 UTC
I have come up with an alternative fix for the issue which is causing your touchpad to not work. I have started a test kernel-build with this new patch (replacing the previous one) here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=82606916

Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem):
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Can you please give this one a test and let me know if the touchpad still works with this one ? (I fully expect it will)

Comment 34 Matt 2022-02-11 07:28:57 UTC
Hi Hans,

Apologies for the delay in getting back to you.  Had HDD partitioning problems recently, resulting in a full re-format of my laptop and haven't had time to setup the device again yet unfortunately. I therefore only have access to a live image setup via USB. There's no way to test kernels while in a live setup I'm assuming? If there is, I'll test the kernel over the weekend, else my testing will have to wait until the laptop is setup again.  

Thanks, Matt.

Comment 35 Hans de Goede 2022-02-11 08:44:09 UTC
Hi Matt,

No worries, I'm happy that you are still willing to help with further testing with this bug, which is turning out much harder to fix then it really should be.

The new fix from the test kernel should work fine for your laptop, so there is no big hurry with testing this. You may want to save the rpms from the test-kernel though, since koji removes the files from test builds after about 10 days to save diskspace.

What would be very helpful is if you can boot the livecd with "efi=debug" added to the kernel commandline (press e at the livecd boot menu to edit the cmdline) and then collect a dmesg in the livecd env. and somehow (e.g. https://paste.centos.org/, save to a second usb-drive) save that dmesg and attach it here.

Regards,

Hans

Comment 36 Matt 2022-02-14 07:17:33 UTC
Created attachment 1860924 [details]
dmesg output with efi=debug cmdline addition

Hi Hans,

Here's the requested dmesg with required debug command-line.


-Matt.

Comment 37 Hans de Goede 2022-02-14 11:29:29 UTC
(In reply to Matt from comment #36)
> Here's the requested dmesg with required debug command-line.

Thank you. Note there is no need anymore to test the new test-kernel. The approach taken there is still causing issues on some other devices, so I'm working on a new approach/patch.

I'll get back to you when I have a test-kernel with a new patch.

Comment 38 Hans de Goede 2022-02-14 15:24:17 UTC
OK, I've prepared a series of 2 patches which tries to fix the touchpad problem in a new way, which will hopefully not break suspend/resume for others:
https://lore.kernel.org/linux-pci/20220214151759.98267-1-hdegoede@redhat.com/T/

A Fedora test-kernel with these patches is building here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=82813003
this should be done in a couple of hours.

Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem):
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Can you please give this one a test and check if the touchpad still works with this new test-kernel?

Please add efi=debug to the kernel commandline when testing this, collect dmesg output with this kernel and attach it here.

Comment 39 Matt 2022-02-15 07:52:22 UTC
Created attachment 1861152 [details]
dmesg output for new test-kernel with efi=debug cmdline

Hi Hans,

I can confirm the touchpad functions correctly with your new patch in this test-kernel. :-) 

I do hope this latest attempt at fixing this issue is resilient enough to keep all other devices from breaking again. It's been a test of patience for you, I'm sure, trying to fully resolve this and similar issues related to these EFI memmap quirks on mine and similar devices.

I'll continue to assist you in testing and helping out with anything further, if you need. I'm optimistic you're getting close to the end here! :-D

Regards, Matt.

Comment 40 Hans de Goede 2022-02-15 20:00:15 UTC
(In reply to Matt from comment #39)
> I can confirm the touchpad functions correctly with your new patch in this
> test-kernel. :-) 
> 
> I do hope this latest attempt at fixing this issue is resilient enough to
> keep all other devices from breaking again. It's been a test of patience for
> you, I'm sure, trying to fully resolve this and similar issues related to
> these EFI memmap quirks on mine and similar devices.
> 
> I'll continue to assist you in testing and helping out with anything
> further, if you need.

Thanks much appreciated.

> I'm optimistic you're getting close to the end here!

I'm optimistic too, but unfortunately we are not quite there yet.

The last fix you tested is still causing some potential issues on the laptop from bug 2029207. So into to the recycle bin that fix goes and I've written another workaround for the PCI BAR assignment issue on your laptop (and others like it). I'm hopeful that this really will be the last one.

Here is a new test-kernel with the new workaround:

https://koji.fedoraproject.org/koji/taskinfo?taskID=82854981

This time the kernel is already fully build, so you can grab it right away. Please add efi=debug to the kernel commandline when testing this, collect dmesg output with this kernel and attach it here, thanks.

Note there is a small chance that your touchpad stops working with this one as the fix is taking a different approach trying to only chance kernel behavior on a very narrow set of laptops this time around. It should identify your laptop as one needing to ignore e820 reservations, but if the identifying code somehow fails then your touchpad will stop working.

Comment 41 Matt 2022-02-16 07:03:35 UTC
Created attachment 1861407 [details]
dmesg output for latest test-kernel with efi=debug cmdline

Hi Hans,

Sorry to hear there are still issues fully resolving this bug whilst not breaking other patches for the related bugs. Quite the cat-and-mouse game for you it seems! 

Good news on my end, however; this latest test-kernel hasn't regressed my touchpad to a state of not functioning again. The dmesg is attached.

-Matt.

Comment 42 Hans de Goede 2022-02-16 15:08:00 UTC
Hi Matt,

(In reply to Matt from comment #41)
> Created attachment 1861407 [details]
> dmesg output for latest test-kernel with efi=debug cmdline
> 
> Hi Hans,
> 
> Sorry to hear there are still issues fully resolving this bug whilst not
> breaking other patches for the related bugs. Quite the cat-and-mouse game
> for you it seems! 
> 
> Good news on my end, however; this latest test-kernel hasn't regressed my
> touchpad to a state of not functioning again. The dmesg is attached.

Great, thank you for testing; and I see this new log message which is part of the new patch:

[    0.321640] acpi PNP0A08:00: host bridge window [mem 0x65400000-0xbfffffff window] is marked by EFI as MMIO

Which means that the new patch is working as it should (as is shown by your touchpad still working) so I've posted it upstream now:

https://lore.kernel.org/linux-pci/20220216150121.9400-2-hdegoede@redhat.com/T/

This will hopefully finally resolve this without causing issues on other systems, thank you for your patience with this.

Comment 43 Matt 2022-02-17 06:37:38 UTC
(In reply to Hans de Goede from comment #42)
> Hi Matt,
>
> 
> Great, thank you for testing; and I see this new log message which is part
> of the new patch:
> 
> [    0.321640] acpi PNP0A08:00: host bridge window [mem
> 0x65400000-0xbfffffff window] is marked by EFI as MMIO
> 
> Which means that the new patch is working as it should (as is shown by your
> touchpad still working) so I've posted it upstream now:
> 
> https://lore.kernel.org/linux-pci/20220216150121.9400-2-hdegoede@redhat.com/
> T/
> 
> This will hopefully finally resolve this without causing issues on other
> systems, thank you for your patience with this.

Hi Hans,

No problem - happy to help if it creates a successful outcome long-term for mine
and others' devices that may be experiencing this.

Thanks for your perseverance in working on this bug to get to the point where 
(hopefully) this will result in a final stable fix. :-)

Regards, Matt.

Comment 44 Hans de Goede 2022-03-04 14:13:10 UTC
Hi Matt,

Bjorn, the upstream PCI subsystem maintainer has come up with a slightly different version of my final final fix. I've done a Fedora test kernel build of 5.16.12 with the fix from Bjorn added:

https://koji.fedoraproject.org/koji/taskinfo?taskID=83634323

Building is already done, so you can grab it right away. As always if you need install instructions they are here:

https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Please give this a new kernel a try and please collect dmesg output after booting it and attach the dmesg output here.

Thanks & Regards,

Hans

Comment 45 Matt 2022-03-05 00:49:12 UTC
Created attachment 1864198 [details]
dmesg output for Bjorn's kernel patch

Hi Hans,

Touchpad is working as usual with this latest test kernel featuring the patches from Bjorn.

Regards, Matt.

Comment 46 Hans de Goede 2022-03-05 10:40:09 UTC
Thank you for the quick testing.

The dmesg shows that Bjorn's patches are working as intended (as does the touchpad working as usual).

What is interesting is that the dmesg output for the workaround triggers *twice*:

[    0.327837] acpi PNP0A08:00: resource [mem 0x000a0000-0x000bffff window] fully covered by e820 entry [mem 0x0009f000-0x000fffff]
[    0.327843] acpi PNP0A08:00: resource [mem 0x65400000-0xbfffffff window] fully covered by e820 entry [mem 0x4bc50000-0xcfffffff]

The first line is about the mem-window for ISA memory-mapped io (behind a PCI to ISA bridge) getting fully clipped without the workaround, which I did not realize also was a potential issue.

Comment 47 Hans de Goede 2022-03-10 12:27:01 UTC
Matt, Bjorn would like to acknowledge all your testing by adding a:

Tested-by: Matt <xxxxx>

Tag to the commit msg, using e.g. the same email as you use for your bugzilla account here, the downside of this is that it exposes your email address to the entire word.

Can you please let us know if this is ok and what name (e.g. do you want your lastname added?) + email address you want us to use for this?

Comment 48 Matt 2022-03-11 07:12:53 UTC
(In reply to Hans de Goede from comment #47)
> Matt, Bjorn would like to acknowledge all your testing by adding a:
> 
> Tested-by: Matt <xxxxx>
> 
> Tag to the commit msg, using e.g. the same email as you use for your
> bugzilla account here, the downside of this is that it exposes your email
> address to the entire word.
> 
> Can you please let us know if this is ok and what name (e.g. do you want
> your lastname added?) + email address you want us to use for this?

Hi Hans,

Please relay my appreciation to Bjorn in recognising my help with this bug.  Although, it is you (and Bjorn) who deserve the credit here; you guys did the debugging and coding. So thank you both for fixing this issue. :-)

I'm happy to have my name attached to the testing tag. You can use my full name - Matt Hansen. The duck.com forwarding address I have attached to my Bugzilla account is fine.

Can I get the link to the final patch commits please for reference? They're in the lore.kernel.org list archive?

Also, should I update the bug status tag from new to closed or will this happen automatically when it's the right time? I'm not well-versed in the Bugzilla process. 

Regards, Matt.

Comment 49 Hans de Goede 2022-03-11 09:19:31 UTC
(In reply to Matt from comment #48)
> I'm happy to have my name attached to the testing tag. You can use my full
> name - Matt Hansen. The duck.com forwarding address I have attached to my
> Bugzilla account is fine.

Ok, I've passed this info along to Bjorn.

> Can I get the link to the final patch commits please for reference? They're
> in the lore.kernel.org list archive?

The thread discussing the patches is here:

https://lore.kernel.org/linux-pci/20220304035110.988712-1-helgaas@kernel.org/

I'll provide you with a link to the final patches when they have been merged.

> Also, should I update the bug status tag from new to closed or will this
> happen automatically when it's the right time?

If the bug gets added to the Fedora specific kernel-update changelog then it will automatically be closed, but that does not always happen. I'll keep an eye on this and close it when the patches have landed.

Comment 50 Hans de Goede 2022-05-13 16:04:16 UTC
Unfortunately the last attempt fix this did not stick upstream because it caused regressions on some chromebooks.

So there is a new patch to attempt to fix this here:
https://lore.kernel.org/linux-pci/20220512202511.34197-2-hdegoede@redhat.com/

A Fedora test-kernel with these patches is building here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=86998438
this should be done in a couple of hours.

Here are some generic instructions for installing a Fedora kernel directly from koji (the Fedora buildsystem):
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Can you please give this one a test and check if the touchpad still works with this new test-kernel?

When running the new kernel please run:

dmesg | grep "E820 res"

This should output:

"PCI: ignoring E820 reservations for host bridge windows"

and then your touchpad should work normally.

Comment 51 Matt 2022-05-15 09:19:37 UTC
(In reply to Hans de Goede from comment #50)
> Unfortunately the last attempt fix this did not stick upstream because it
> caused regressions on some chromebooks.
>
Hi Hans, apologies on the delay with testing - I noticed you'd built a kernel
for the newly released F36 and I hadn't yet upgraded my system. 
Unfortunate to hear about regressions again!
> 
> Can you please give this one a test and check if the touchpad still works
> with this new test-kernel?

Can confirm touchpad functions correctly with this test-kernel.
> 
> When running the new kernel please run:
> 
> dmesg | grep "E820 res"
> 
> This should output:
> 
> "PCI: ignoring E820 reservations for host bridge windows"
> 
> and then your touchpad should work normally.

Log output is as expected:
[mh@fedora ~]$ dmesg | grep "E820 res"
[    0.273196] PCI: Ignoring E820 reservations for host bridge windows
[mh@fedora ~]$ 

I am optimistic this bug may now see final resolution. These DMI quirks
for systems <2023 seem like the most reliable method to satisfy all 
test-cases for all systems affected by this and related PCI issues.

Regards, Matt.

Comment 52 Hans de Goede 2022-05-16 07:28:12 UTC
(In reply to Matt from comment #51)
> (In reply to Hans de Goede from comment #50)
> > Unfortunately the last attempt fix this did not stick upstream because it
> > caused regressions on some chromebooks.
> >
> Hi Hans, apologies on the delay with testing - I noticed you'd built a kernel
> for the newly released F36 and I hadn't yet upgraded my system. 

No worries and as always thank you for the testing.

Note that kernels are pretty much release version independent, so you could have just installed the F36 kernel on your F35.

Comment 53 Matt 2022-05-16 08:46:50 UTC
(In reply to Hans de Goede from comment #52)
> No worries and as always thank you for the >testing.

Always happy to help and see this bug through to closure. Hopefully we are close after almost 2 years now. :-D

> Note that kernels are pretty much release version independent, so you could
> have just installed the F36 kernel on your F35.

Thanks for the tip Hans.  The upgrade was planned anyway, so it was no bother. Especially these days, with the simple and stable Fedora upgrade process :-)

Regards, Matt.

Comment 54 Matt 2022-10-30 09:14:36 UTC
Hi Hans,

It seems I've had a fully working touchpad for a few recent kernel releases now. Can you confirm the bug is now
fully fixed? 
Also, if so, can you please provide the changelogs for the Fedora-specific kernel package the final patches were
first applied?
Have the fixes gone upstream to the Linux mainline kernel or are they RH/Fedora specific patches only? Is this 
bug now ready to be marked close?

Regards, Matt.

Comment 55 Hans de Goede 2022-10-30 10:10:19 UTC
Hi Matt,

Good to hear that this works in standard Fedora kernels now. Yes I can confirm that this is now fully fixed.

The Fedora kernels do not carry any specific patches for this. This has been fixed in the upstream / mainline kernel with this commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d341838d776abadb3ac48abdd2f1f40df5a4fc10

Which is in essence the same fix as you tested earlier:

https://lore.kernel.org/linux-pci/20220512202511.34197-2-hdegoede@redhat.com/

except that the addition of the use_e820 / no_e820 kernel commandline options was split out into a separate patch:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fa6dae5d82081e8d9f8e6a2baf7149442a6c1ba5

Regards,

Hans

Comment 56 Matt 2022-10-30 11:37:28 UTC
Hi Hans,

Thanks so much for all your work on this bug and eventual patches! It's great to see the fixes accepted 
into mainline kernels as well. :)

I'll go ahead and mark this bug as status closed. (upstream tag is best?)

TC & regards, Matt.

Comment 57 Bjorn Helgaas 2022-11-22 15:59:36 UTC
Created attachment 1926437 [details]
experimental patch to remove EfiMemoryMappedIO

The current upstream resolution to this problem is https://git.kernel.org/linus/d341838d776a ("x86/PCI: Disable E820 reserved region clipping via quirks"), which relies on quirks that match DMI Vendor, Product Version, Product Name, and Board Name.  This isn't an ideal solution because there are likely other systems we don't know about that need the a similar fix.

The patch I'm attaching here is an experimental idea to work around this issue without the maintenance burden of the quirks.

If anybody would be willing to test this patch, I would be very grateful.  To test it, apply this patch to your kernel, boot with "pci=use_e820 efi=debug", and collect the dmesg log.

Comment 58 Matt 2022-11-23 08:02:07 UTC
Hi Bjorn

Happy to continue testing patches and will make an effort to apply this patch and build a new Fedora testing kernel via fedpkg within the next few days. 
That is unless/if Hans has planned to build one via 
Koji?

Regards, Matt.

PS. I've reopened the bug to status "MODIFIED". Please let me know if it should be changed.

Comment 59 Hans de Goede 2022-11-23 14:44:50 UTC
(In reply to Matt from comment #58)
> Hi Bjorn
> 
> Happy to continue testing patches and will make an effort to apply this
> patch and build a new Fedora testing kernel via fedpkg within the next few
> days. 
> That is unless/if Hans has planned to build one via 
> Koji?

Hi Matt,

I really appreciate your willingness to help Bjorn test alternative methods of fixing this. Unfortunately my workload does not allow me to go and do koji scratch-builds for this. I would likely only be getting in the way of any dialog between you and Bjorn because I would cause you to wait for a long time before I get around to submitting any koji kernel test builds.

So it would be best if you just build the kernel locally with fedpkg.

Regards,

Hans

Comment 60 Matt 2022-11-26 23:52:48 UTC
Created attachment 1927740 [details]
dmesg output for remove-mmio patch

Good morning all,

Hans, no problem - I managed to build a local kernel via fedpkg. That process was quite an experience as I've never built a kernel package via fedpkg before and it took quite a lot of tinkering with the git cloning process due to some possibly network-related issues that caused the process to fail with error: 

fetch-pack: unexpected disconnect while reading sideband packetB/s
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Seems to be a potential Fedora Infrastructure problem (https://pagure.io/fedora-infrastructure/issue/9677). I managed to work-around the issue with a partial clone followed by an unshallow command and then fetching all branches. Have you ever come across this type of problem before with fedpkg? 


Bjorn, I've booted into a test kernel with the two required kernel command-line args and attached the dmesg here. Touchpad functions as expected. Your patch seems to have been applied correctly as seen in the dmesg showing your newly added kernel boot info:

[    0.000000] efi: removing MMIO range=[0x0000000065400000-0x00000000cfffffff] (1708MB) from E820 reservations
[    0.000000] e820: remove [mem 0x65400000-0xcfffffff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000fc800000-0x00000000fe7fffff] (32MB) from E820 reservations
[    0.000000] e820: remove [mem 0xfc800000-0xfe7fffff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000fed00000-0x00000000fed00fff] (0MB) from E820 reservations
[    0.000000] e820: remove [mem 0xfed00000-0xfed00fff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000fed10000-0x00000000fed17fff] (0MB) from E820 reservations
[    0.000000] e820: remove [mem 0xfed10000-0xfed17fff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000feda0000-0x00000000feda1fff] (0MB) from E820 reservations
[    0.000000] e820: remove [mem 0xfeda0000-0xfeda1fff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000fee00000-0x00000000fee00fff] (0MB) from E820 reservations
[    0.000000] e820: remove [mem 0xfee00000-0xfee00fff] reserved
[    0.000000] efi: removing MMIO range=[0x00000000ff600000-0x00000000ffffffff] (10MB) from E820 reservations
[    0.000000] e820: remove [mem 0xff600000-0xffffffff] reserved

Hopefully I've done this all correctly! Let me know what else you may need for debugging purposes.

Regards, Matt.

Comment 61 Bjorn Helgaas 2022-11-28 21:22:41 UTC
Thank you very much, Matt!  The fact that the touchpad works is very promising.  I do want to figure out these messages because they really look bogus and should be improved or removed:

[    0.413903] clipped [mem size 0x00000000 64bit] to [mem size 0xfffffffffffa0000 64bit] for e820 entry [mem 0x0009f000-0x000fffff]

Comment 62 Matt 2022-12-02 08:04:51 UTC
(In reply to Bjorn Helgaas from comment #61)
> Thank you very much, Matt!  The fact that the touchpad works is very
> promising.  I do want to figure out these messages because they really look
> bogus and should be improved or removed:
> 
> [    0.413903] clipped [mem size 0x00000000 64bit] to [mem size
> 0xfffffffffffa0000 64bit] for e820 entry [mem 0x0009f000-0x000fffff]

No problem at all Bjorn. Feel free to provide another patch if you decide to alter/remove the message output mentioned above and need further testing.

-Matt

Comment 63 Matt 2022-12-03 10:43:32 UTC
Created attachment 1929651 [details]
dmesg output for latest Bjorn patchset(4/4)

Hi Bjorn,

I've built a 6.0.11 based Fedora kernel with the 4 patches applied. Great news -  the touchpad remains functioning as expected and the dmesg output looks great with the new or cleaned-up message formats. :)

Regards, Matt

PS, my Duck remailer seems to have trouble replying to your email list, so I haven't cross-posted this there.

Comment 64 Hans de Goede 2022-12-03 11:56:04 UTC
Matt,

Thank you for testing. I'm a bit worried that Bjorn's patches are not going to work in BIOS (CSM) compatibility mode (which quite a few Linux users use mostly due to inertia).

IIRC last time I dived into this the troublesome memory reservations also showed up in the E820 memory map when in CSM mode and in that case the E820 map is actually made by the BIOS and we cannot tell the difference between reservations which come from EfiMemoryMappedIO and other reservations.

To confirm this, can you boot a Fedora 37 live USB in BIOS compatibility mode and then collect dmesg output ?

I think just getting the dmesg output with a standard F37 kernel from the live USB should give us enough info to confirm (or deny) my worries about Born's patches not helping for the BIOS (CSM) compatibility mode.

Regards,

Hans

Comment 65 Matt 2022-12-03 14:08:54 UTC
(In reply to Hans de Goede from comment #64)
> Matt,
> 
> Thank you for testing. I'm a bit worried that Bjorn's patches are not going
> to work in BIOS (CSM) compatibility mode (which quite a few Linux users use
> mostly due to inertia).
> 
> IIRC last time I dived into this the troublesome memory reservations also
> showed up in the E820 memory map when in CSM mode and in that case the E820
> map is actually made by the BIOS and we cannot tell the difference between
> reservations which come from EfiMemoryMappedIO and other reservations.
> 
> To confirm this, can you boot a Fedora 37 live USB in BIOS compatibility
> mode and then collect dmesg output ?
> 
> I think just getting the dmesg output with a standard F37 kernel from the
> live USB should give us enough info to confirm (or deny) my worries about
> Born's patches not helping for the BIOS (CSM) compatibility mode.
> 
> Regards,
> 
> Hans

Hi Hans,

Yeah, I'll test that tomorrow no worries. I'll have to download a F37 image as I think my live USB still contains F35. :D
I'll update with a dmesg as requested.

Regards, Matt

PS Is there a way to speed up kernel package building with fedpkg? Perhaps a way to
not build all the auxillary kernel rpms such as kernel-debug*, kernel-devel* or the extra kernel-modules-* ones? 
Limit the build to just kernel/kernel-core/kernel-modules?

Comment 66 Matt 2022-12-04 00:53:54 UTC
Created attachment 1929753 [details]
dmesg output while booting in Legacy BIOS/CSM mode via LiveUSB

Good morning Hans,

I've booted into a F37 LiveUSB system after setting the boot to "Legacy BIOS" mode.
The dmesg seems to suggest it's using the CSM as there's no usual references to "efi" in the output. Touchpad still functions. Hope this helps.

Regards, Matt.

Comment 67 Hans de Goede 2022-12-04 09:35:19 UTC
(In reply to Matt from comment #66)
> Created attachment 1929753 [details]
> dmesg output while booting in Legacy BIOS/CSM mode via LiveUSB
> 
> Good morning Hans,
> 
> I've booted into a F37 LiveUSB system after setting the boot to "Legacy
> BIOS" mode.
> The dmesg seems to suggest it's using the CSM as there's no usual references
> to "efi" in the output. Touchpad still functions. Hope this helps.
> 
> Regards, Matt.


Hi Matt,

Thank you.

So as I was afraid Bjorn's patches (which are an attempt to fix the issue in a generic way) won't work when booting the system in "Legacy BIOS" mode. Note this does not affect your system since your system has a model-string matching based quirk to avoid the problem already, so it will work even in CSM mode.

But it does mean that Bjorn's patches won't help for as of yet unknown models with this issue when those are booted in CSM mode, which is good to know.

Regards,

Hans

Comment 68 Hans de Goede 2022-12-08 19:57:07 UTC
Matt,

Bjorn has posted a v2 of his patch series for this. I have started a test Fedora kernel build with the v2 patches added:

https://koji.fedoraproject.org/koji/taskinfo?taskID=95100621

Note this is still building atm. It should be finished in a couple of hours.

It would be great if you can give this a test spin once it has finished building.

Like with the kernel you build yourself, please collect a dmesg with "pci=use_e820 efi=debug" added to the kernel commandline

Your touchpad should keep working with this (even if you pass "pci=use_e820" to override the no_e820 quirk for your model).

I don't think you need these anymore, but just incase here are some generic instructions for installing a kernel directly from koji (Fedora's buildsystem):

https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Regards,

Hans

Comment 69 Matt 2022-12-09 07:56:21 UTC
Created attachment 1931253 [details]
dmesg output for v2 of Bjorn's patchset(4/4)

Hi Hans,

Thanks for building a test-kernel for this v2 patchset. I enjoyed undertaking the kernel building and patch-applying process, but having one pre-built and ready to install is much easier. :)

Dmesg attached with the required kernel cmdline args. Bjorn has CC'd me into the current email-list discussion as well, which is interesting to follow.

Regards, Matt.

Comment 70 Ben Cotton 2023-04-25 18:27:00 UTC
This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 71 Bjorn Helgaas 2023-04-25 19:19:41 UTC
I think this issue should be resolved by 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") (https://git.kernel.org/linus/07eab0901ede), which appeared in v6.2.

Let me know if you still see this issue with a kernel that includes 07eab0901ede.

Comment 72 Matt 2023-04-26 12:38:44 UTC
(In reply to Bjorn Helgaas from comment #71)
> I think this issue should be resolved by 07eab0901ede ("efi/x86: Remove
> EfiMemoryMappedIO from E820 map")
> (https://git.kernel.org/linus/07eab0901ede), which appeared in v6.2.
> 
> Let me know if you still see this issue with a kernel that includes
> 07eab0901ede.

Hi Bjorn,

Yes, I can confirm this issue has been fixed for me in recent kernels. :)

Thanks to Hans and Bjorn for all the great work in resolving this issue.
It's great to also see it incorporated as a generic solution into the 
upstream kernel tree without having to rely on any Fedora-specific patches.

It's been a fun experience helping test patches and provide debugging data 
over the last almost 3 years. :)

I'll go ahead and close this bug with resolution: upstream.  

Regards, Matt.


Note You need to log in before you can comment on or make changes to this bug.