Created attachment 1520153 [details] dmesg output after system boot 1. Please describe the problem: no wireless interfaces and thus no wifi support... In dmesg I can see: rtl8723bs: probe of mmc1:0001:1 failed with error -16 The device selection as such seems to be the correct: # cat /sys/bus/mmc/devices/mmc1\:0001/type SDIO On this unit mmc0 is internal flash. There are also some bluetooth error messages, not sure if this is somehow related (see dmesg attachment). 2. What is the Version-Release number of the kernel: 4.19.13-300.fc29.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Wifi never worked on this hardware for me, that being said, its the first time I installed anything on it. 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: booting is enough, wifi does not work consistently 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: yes, the behavior on 5.0.0-0.rc1.git3.2.fc30.x86_64 is the same 6. Are you running any modules that not shipped with directly Fedora's kernel?: no 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. dmesg attached
Hi, Ok, so looking at the logs, this is the only error about the problem indeed is: rtl8723bs: probe of mmc1:0001:1 failed with error -16 With -16 being -EBUSY. This is quite weird. Can you do: sudo dnf install acpica-tools sudo acpidump -o acpidump.acer-switch10-byt And attach the generated idump.acer-switch10-byt file here, And also run: sudo alsa-info.sh And attach the file it generates here? I realize this is not an audio issue but alsa-info collects a bunch of useful info regardless. Also what model do you have? There should be a sticker at the edge of the tablet which is only visible when you undock it from the keyboard. Thanks & Regards, Hans
Created attachment 1522608 [details] output of acpidump
Created attachment 1522609 [details] output of alsa-info.sh
Hi, thank you for looking into this! I attached both logs, the sticker says: NTG8ZEC00154478BC7200 if I got the numbers/letters right, the size and quality of the print on the sticker makes 8 vs B vs 6 hard to read. It was manufactured in 2015.10 Kind regards, Sergey
Hmm, I might have an idea what is going on here, can you please do: cat /sys/bus/sdio/devices/mmc1\:0001\:?/vendor cat /sys/bus/sdio/devices/mmc1\:0001\:?/device And copy and paste the output of both commands here please?
Here it is: # cat /sys/bus/sdio/devices/mmc1\:0001\:?/vendor 0x024c # cat /sys/bus/sdio/devices/mmc1\:0001\:?/device 0x0623
Ok, so this is weird, can you do (as root): rmmod r8723bs echo "file drivers/base/* +p" > /sys/kernel/debug/dynamic_debug/control modprobe r8732bs dmesg > dmesg.debug And then attach dmesg.debug here please?
Created attachment 1522634 [details] dmesg output after tuning /sys/kernel/debug/dynamic_debug/control
Sure, attached.
Btw all todays logs created running 5.0.0-0.rc3.git0.1.fc30.x86_64 (updated today).
(In reply to Sergey Bostandzhyan from comment #10) > Btw all todays logs created running 5.0.0-0.rc3.git0.1.fc30.x86_64 (updated > today). That is fine. Still no clue unfortunately, can you do: echo "file drivers/acpi/* +p" > /sys/kernel/debug/dynamic_debug/control echo "file drivers/gpio/* +p" > /sys/kernel/debug/dynamic_debug/control echo "file drivers/pinctrl/* +p" > /sys/kernel/debug/dynamic_debug/control And then do the rmmod + modprobe and gather dmesg again ?
Ah and also add: echo "file drivers/mmc/core/* +p" > /sys/kernel/debug/dynamic_debug/control That might get a bit verbose with the system running of an eMMC but hopefully it won't be too bad.
It is indeed verbose, so I did it in two steps, dmeg02.debug with the additoinal three commands from comment 11 and dmesg03.debug with the mmc core messages. Had to modprobe && dmesg to catch the insertion messages as the dmesg buffer was overfilling quickly, I hope all relevant stuff is in there.
Created attachment 1522641 [details] dmesg output with options from comment 11
Created attachment 1522642 [details] dmesg output with options from comment 11 and 12
Thank you for the log. Unfortunately nothing stands out. I've a theory for where the EBUSY is coming from, but not for why yet. However there also is a problem with Linux and your ACPI tables not liking each other. How to properly fix this is a bit of a head-scratcher, But as a workaround we can "simply" override the ACPI tables. I've prepared an initrd with ACPI table override for your device (and for your device only) here: https://fedorapeople.org/~jwrdegoede/initrd-dsdt-override.acer-switch10-TP-SW3-016-15NE Download this file and then as root do: cd /boot mv initramfs-$(uname -r).img initramfs-$(uname -r).img.org cat initrd-dsdt-override.acer-switch10-TP-SW3-016-15NE initramfs-$(uname -r).img.org > initramfs-$(uname -r).img This prepends the override-initrd to your existing initrd. and then reboot into the kernel you were running when executing this commands. I'm not sure if this will help with your wifi (it might, it does contain wifi relevant changes) but it should definitively help with the bluetooth related errors. If you have the latest linux-firmware package installed I expect you to have working bluetooth after this. Either way please collect a dmesg shortly after boot with the new initrd, and if possible please test bluetooth (and wifi of course).
Created attachment 1522687 [details] dmesg output with new initrd I followed the instructions, but I still see those bluetooth errors in dmesg, I think something did not work out. Also, hcitool dev does not show any devices. I am no good with driver/lowlevel coding, I am not a hardware guy, but I know C and I can set up a kernel build and add additional logs/patches where instructed if that would be of any help to you.
Created attachment 1522690 [details] dmesg output with new initrd plus exsra debug options Not sure if it helps, but here's another dmesg with reloading the wifi driver and adding the extra options (except for the chatty emmc stuff) while running the new initrd image.
The new dmesg contains: [ 0.012284] ACPI: ACPI OVERRIDE: Unknown signature [kernel/firmware/acpi/dsdt So I think you have secureboot enabled in the BIOS, can you try disabling it and see if that message goes away? (In reply to Sergey Bostandzhyan from comment #17) > I am no good with driver/lowlevel coding, I am not a hardware guy, but I > know C and I can set up a kernel build and add additional logs/patches where > instructed if that would be of any help to you. If the DSDT override does not help (once it sticks and fixes the bluetooth errors), it would be good if you can verify my theory where the EBUSY is coming from. I'll attach a patch for that, if you can build your own kernel, starting with the Fedora /boot/config-5.xxxx as .config, with that patch applied and then see if the error message from the patch triggers then that would be useful as a starting point. After that please keep the kernel-tree with the build around so we can add more debugging easily.
Created attachment 1522718 [details] PATCH adding a log message to prove my theory of the EBUSY
(In reply to Hans de Goede from comment #19) > The new dmesg contains: > > [ 0.012284] ACPI: ACPI OVERRIDE: Unknown signature > [kernel/firmware/acpi/dsdt > > So I think you have secureboot enabled in the BIOS, can you try disabling it > and see if that message goes away? BIOS and UEFI is a real pain point on this device, I actually had to send it in to Acer because at some point it was only able to boot Kali Linux from USB, but nothing else, not even a Winblows 10 recovery. I now know better that deleting all partitions on emmc including EFI and recreating them was not a good idea ;) I'll try to change the BIOS settings and see if accepts that... > If the DSDT override does not help (once it sticks and fixes the bluetooth > errors), it would be good if you can verify my theory where the EBUSY is > coming from. > > I'll attach a patch for that, if you can build your own kernel, starting > with the Fedora /boot/config-5.xxxx as .config, with that patch applied and > then see if the error message from the patch triggers then that would be > useful as a starting point. After that please keep the kernel-tree with the > build around so we can add more debugging easily. Sure, we can do that. Keeping the tree around is a good idea, when I tried compiling the kernel a year ago on that hardware it took several hours... I'll get back to you on both points (BIOS + patch), latter may take a while since I will have to find a time slot where the device is free for longer experiments. Thank you for debugging this issue, I really appreciate the effort on getting this problem solved!
(In reply to Hans de Goede from comment #19) > So I think you have secureboot enabled in the BIOS, can you try disabling it > and see if that message goes away? To my surprise secure boot was already disabled in BIOS... what else could prevent the initrd override from loading?
Created attachment 1522736 [details] secure boot setting in Acer Switch 10 BIOS
My bad, that error has nothing to do with secure boot and I put the DSDT source-code in the initrd instead of the compiled byte-code which is why it is barfing. Let me prepare a new version.
Ok a new version is now in the same place: https://fedorapeople.org/~jwrdegoede/initrd-dsdt-override.acer-switch10-TP-SW3-016-15NE After downloading this should be 94702 bytes big. Since you already moved your original initrd to .orig, now all you need to do is: cd /boot cat initrd-dsdt-override.acer-switch10-TP-SW3-016-15NE initramfs-$(uname -r).img.org > initramfs-$(uname -r).img Otherwise you end up with 2 prepended initrds :)
Created attachment 1522780 [details] dmesg output with newer initrd Cool, thank you! WiFi works now, I did not have any BT devices around to test but hcitool dev does list a BT device as well. I'll get back to you with that kernel build, but as I mentioned that may take a while. Can I attach your ACPI code to any newer initrd on this device? I.e. it should still be applicable after each new kernel update on this particular machine?
Hi, (In reply to Sergey Bostandzhyan from comment #26) > Created attachment 1522780 [details] > dmesg output with newer initrd > > Cool, thank you! WiFi works now, I did not have any BT devices around to > test but hcitool dev does list a BT device as well. That is both good news. I was hoping for that, but I did not really expect this to fix wifi too. > I'll get back to you with that kernel build, but as I mentioned that may take a while. Given that the DSDT override fixes wifi, that kernel build is no longer necessary. > Can I attach your ACPI code to any newer initrd on this device? I.e. it should still > be applicable after each new kernel update on this particular machine? Yes, you can prepend the initrd I gave you to the initrd for any kernel. We really should have a way to specify a dsdt override initrd in a config file and have this happen automatically. Unfortunately there is no way to do this automatically atm. I will discuss this with a couple of colleges. As for a fix which will make things work OOTB, as mentioned already how to properly fix this is a bit of a head-scratcher. Now that I've confirmation that the problem is what I thought it was (a difference in in which order Windows parses the DSDT vs Linux more or less) I can start discussing trying to fix this in a better way with other upstream kernel devs. In the mean time you can keep using the DSDT override as a workaround. Thank you for your cooperation in getting to the bottom of this. Regards, Hans
> Thank you for your cooperation in getting to the bottom of this. Thank *you*! It's you who actually made it work :) And I'm happy to hear that you are in the right spot to discuss and help with a permanent solution in the stock kernel, thank you for taking care of this! There are some other probems with this device (freezes once in a while, touchpad stops working), I'm trying to gather something debuggable first before submitting the issues, but you'll probably see more Acer Switch 10 related bug reports from me in the future :) You can also contact me if any further testing on this device is required when a permanent fix is in sight. Kind regards, Sergey
We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora XX has now been rebased to 5.0.6 Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
We have a workaround by overriding the DSTD, but that is not really a solution. Unfortunately properly fixing this is quite tricky so the root cause is still present, clearing needinfo.
Hans, thank you for keeping an eye on this, I was actually about to ask how to proceed with the ticket, but I see you have already answered.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 5.2.9-100.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
I retested the issue with 5.2.7-100.fc29.x86_64 without the dtb append from Hans and it actually worked! Hans, so it seems you did manage to solve this problem? Can we close this one?
I did not make any changes which would explain this being fixed. But if some other changes fixes this (that is not impossible) then I'm happy to hear that. Likely this does mean that we need a small tweak to also make bluetooth work, can you do the following for me (with the new kernel) to check: 1) Run "ls /sys/bus/acpi/devices" and copy and paste the output here 2) Run "cat /sys/bus/acpi/devices/OBDA0623:00/status" and copy and paste the output here 3) Run "dmesg > dmesg.txt" and attach the generated dmesg.txt file here Thanks.
Created attachment 1607710 [details] latest dmesg output with 5.2.7-100.fc29.x86_64
You quite right... I did not notice it, because I was not using bluetooth, but log messages indicate that it would not work, so I guess the issue just got swapped between bluetooth and wifi... 1) # ls /sys/bus/acpi/devices 10EC5640:00 device:01 device:21 INT33E1:00 LNXCPU:02 LNXSYBUS:00 80860F14:00 device:02 device:22 INT33F4:00 LNXCPU:03 LNXSYBUS:01 80860F14:01 device:03 device:23 INT33F5:00 LNXPOWER:00 LNXSYSTM:00 80860F14:02 device:04 device:24 INT33FB:00 LNXPOWER:01 LNXTHERM:00 80860F14:03 device:05 device:25 INT33FB:01 LNXPOWER:02 LNXVIDEO:00 80862286:00 device:06 device:26 INT33FD:00 LNXPOWER:03 MSFT0101:00 80862288:00 device:07 device:27 INT33FF:00 LNXPOWER:04 MSFT9001:00 80862289:00 device:08 device:28 INT33FF:01 LNXPOWER:05 MSFT9002:00 8086228A:00 device:09 device:29 INT33FF:02 LNXPOWER:06 PNP0000:00 8086228A:01 device:0a device:2a INT33FF:03 LNXPOWER:07 PNP0100:00 8086228E:00 device:0b device:2b INT33FF:04 LNXPOWER:08 PNP0103:00 8086228E:01 device:0c device:2c INT3400:00 LNXPOWER:09 PNP0501:00 8086228E:02 device:0d device:2d INT3403:00 LNXPOWER:0a PNP0A08:00 8086229C:00 device:0e device:2e INT3403:01 LNXPOWER:0b PNP0B00:00 808622A8:00 device:0f device:2f INT3403:02 LNXPOWER:0c PNP0C02:00 808622C0:00 device:10 device:30 INT3403:03 LNXPOWER:0d PNP0C02:01 808622C1:00 device:11 device:31 INT3403:04 LNXPOWER:0e PNP0C02:02 808622C1:01 device:12 device:32 INT3406:00 LNXPOWER:0f PNP0C0A:00 808622C1:02 device:13 device:33 INT3407:00 LNXPOWER:10 PNP0C0C:00 808622C1:03 device:14 DMY0001:00 INT3408:00 LNXPOWER:11 PNP0C0D:00 808622C1:04 device:15 ELAN1001:00 INT3409:00 LNXPOWER:12 PNP0C0F:00 808622C1:05 device:16 GPTC0001:00 INT3477:00 LNXPOWER:13 PNP0C0F:01 808622C1:06 device:17 IMPJ0003:00 INT34D0:00 LNXPOWER:14 PNP0C0F:02 808622D8:00 device:18 IMPJ0003:01 INT34D3:00 LNXPOWER:15 PNP0C0F:03 ACPI0003:00 device:19 INT0002:00 INTCF1C:00 LNXPOWER:16 PNP0C0F:04 ACPI000C:00 device:1a INT0800:00 INTCF1D:00 LNXPOWER:17 PNP0C0F:05 ACPI0011:00 device:1b INT339A:00 INTCFD9:00 LNXPOWER:18 PNP0C0F:06 AUTH2750:00 device:1c INT33A2:00 INTL9C60:00 LNXPOWER:19 PNP0C0F:07 BCM2E5B:00 device:1d INT33A4:00 INTL9C60:01 LNXPOWER:1a PNP0C14:00 BCM4356:00 device:1e INT33BD:00 KIOX0009:00 LNXPOWER:1b PNP0C14:01 CPLM3218:00 device:1f INT33CF:00 LNXCPU:00 LNXPOWER:1c device:00 device:20 INT33D5:00 LNXCPU:01 LNXPWRBN:00 2) # cat /sys/bus/acpi/devices/OBDA0623:00/status cat: '/sys/bus/acpi/devices/OBDA0623:00/status': No such file or directory Did you mean to look at one of the BCM devices? I'm just guessing... # cat BCM4356\:00/status 0 # cat BCM2E5B\:00/status 15 3) attached
Thank you for providing the debug output, so it seems that the root cause of the problem (ACPI initialization order) still exits and because of this the _STA methods which read some GPIOs to determine the type of wifi module used still get things wrong and misidentifies your Realtek RTL8723BS wifi/bt module as a Broadcom device. For some reason this no longer causes the -EBUSY errors on the wifi we were seeing before and since the *actual* manufacturer and device id are read over SDIO, the wifi device being called BCMXXXX by the ACPI code does not bother things as the sdio subsys will still load the right wifi driver based on the actual manufacturer and device id. But for bluetooth things are still broken, as for UART attached devices we do rely on the ID provided by the ACPI code. TL;DR: nothing has changed really and you still need to preprend the DSTD override to the initrd to get things to work properly.
This message is a reminder that Fedora 29 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '29'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 29 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
Hans, shall I try to update the device to Fedora 31 and reopen the issue? In other words - are the discussions still ongoing and is there some effort under way to find a proper fix in the kernel, or shall we just leave it be? As long as the workaround that you provided works - I'm fine, but of course I would prefer if there was a fix for all users.
(In reply to Sergey Bostandzhyan from comment #40) > Hans, > > shall I try to update the device to Fedora 31 and reopen the issue? In other > words - are the discussions still ongoing and is there some effort under way > to find a proper fix in the kernel, or shall we just leave it be? As long as > the workaround that you provided works - I'm fine, but of course I would > prefer if there was a fix for all users. Nothing has changed wrt this, so re-testing will not lead to any different results. It seems that your device-model is the only model we have encountered sofar which suffers from this ordering issue, so the chances of someone reworking ACPI probe ordering upstream just for this are small. So I'm afraid we will just have to live with the workaround and I believe that it is best to leave this bug closed.
OK, understood; thank you for the info and of course also for the workaround :)
(In reply to Sergey Bostandzhyan from comment #42) > OK, understood; thank you for the info and of course also for the workaround :) So a quick update on this, I bought a second hand Acer Switch 10E myself, without realizing it actually was the model from this bug. Since I now have access to the affected hardware and since we already had a discussion upstream about how to implement this, I've made some time to actually implement a fix on the kernel side for this. The patches fixing this have been posted here: https://lore.kernel.org/linux-acpi/20201121203040.146252-1-hdegoede@redhat.com/ I'll also attach them to this bug in case you want to build your own kernel with the patches added.
Created attachment 1732018 [details] Set of kernel patches fixing this without needing a DSDT override
Hello Hans, wow, cool, thank you for looking into this! Can't promise that I will test it soon, truth be told I did not update that notebook in a while (my wife hates distro updates), so I'd need to install something recent first, before I can try your patches on a new kernel. I'll report back once I get there. Thank you!!!
(In reply to Sergey Bostandzhyan from comment #45) > Hello Hans, > > wow, cool, thank you for looking into this! You're welcome. > Can't promise that I will test > it soon, truth be told I did not update that notebook in a while (my wife > hates distro updates), so I'd need to install something recent first, before > I can try your patches on a new kernel. My comment was mostly just FYI (for your information), I have access to the exact same model as you have now, so there is no need to test. You can still test if you curious, but there is really no need for it to help with the patches moving forward.