Bug 1690852 - 5 to 20 minutes shutdowns on Cherry Trail machine [NEEDINFO]
Summary: 5 to 20 minutes shutdowns on Cherry Trail machine
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 29
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 10:56 UTC by Jeffrey Walton
Modified: 2019-09-17 20:03 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-17 20:03:18 UTC
jforbes: needinfo? (noloader)


Attachments (Terms of Use)
Image of screen capture during shutdown (3.80 MB, image/jpeg)
2019-03-20 10:56 UTC, Jeffrey Walton
no flags Details
Output from dmidecode (3.31 KB, application/zip)
2019-03-20 10:57 UTC, Jeffrey Walton
no flags Details
Output of dmesg after reboot (16.17 KB, application/zip)
2019-03-21 01:30 UTC, Jeffrey Walton
no flags Details
brcmfmac43455-sdio.txt (1.73 KB, text/plain)
2019-03-21 14:28 UTC, Hans de Goede
no flags Details
UEFI v1.0 dumps (316.36 KB, application/zip)
2019-03-22 11:12 UTC, Jeffrey Walton
no flags Details
Output of dmesg using HDG kernel (16.21 KB, application/zip)
2019-04-20 06:36 UTC, Jeffrey Walton
no flags Details
Output from dmidecode using HDG kernel (3.29 KB, application/zip)
2019-04-20 06:36 UTC, Jeffrey Walton
no flags Details

Description Jeffrey Walton 2019-03-20 10:56:34 UTC
Created attachment 1546006 [details]
Image of screen capture during shutdown

I have a minipc based on Cherry Trail. The machine is https://www.amazon.com/gp/product/B07D9YX3W6 .

After a dnf update I often reboot the machine. I issue 'shutdown -r' over shh.

The problem is, the machine experiences some sort of bug on shutdown and often takes 15 or 20 minutes to reboot. Message are provided to a monitor as shown below, and it is repeated numerous times. The message eventually fills the screen.

    watchdog: watchdog0: watchdog did not stop!
    watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-shutdow:1]
    watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-shutdow:1]
    watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd-shutdow:1]
    ...

In this instance of the problem, I issued 'reboot -r' at 6:27 AM and the machine rebooted at 6:46 AM. That's 19 minutes to reboot.

==========

I really don't know what you need or how to provide it since this is a shutdown bug. If anyone has suggestions for capturing a log file during shutdown before the subsequent rewrite then I would be happy to provide it.

==========

The machine is running Fedora 29 and fully patched. The machine is updated several times a week using 'dnf upgrade'.

    $ uname -a
    Linux callboot 4.20.16-200.fc29.x86_64 #1 SMP Thu Mar 14 15:10:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

And:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 76
model name      : Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz
stepping        : 4
microcode       : 0x40a
cpu MHz         : 1439.954
cache size      : 1024 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm arat
bugs            : cpu_meltdown spectre_v1 spectre_v2
bogomips        : 2880.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Comment 1 Jeffrey Walton 2019-03-20 10:57:37 UTC
Created attachment 1546008 [details]
Output from dmidecode

Comment 2 Jeffrey Walton 2019-03-20 11:10:21 UTC
There are some other unusual things going on with this machine. As a first example, GNOME reports batter status like a cell phone or laptop even though there is no battery. And more amazingly, it decrements the charge over time. It will start at 97% or so and slowly decrease to a minimum like 5%.

A second example is, I cannot sustain a SSH connection unless I (1) disable "put computer to sleep when battery is low" and (2) physically log into the machine. Without disabling sleep the network is dropped after about 15 minutes. Without a physical login SSH is dropped after 15 or 30 minutes.

Comment 3 Jeffrey Walton 2019-03-21 01:30:03 UTC
Created attachment 1546278 [details]
Output of dmesg after reboot

Comment 4 Hans de Goede 2019-03-21 09:31:47 UTC
(In reply to Jeffrey Walton from comment #2)
> There are some other unusual things going on with this machine. As a first
> example, GNOME reports batter status like a cell phone or laptop even though
> there is no battery. And more amazingly, it decrements the charge over time.
> It will start at 97% or so and slowly decrease to a minimum like 5%.

Yes, well the vendor took a tablet design and then put it in a different enclosure
and they did not bother to disable the charger/fuel-gauge part of the PMIC
in there BIOS, from your dmidecode:

Chassis Information
        Manufacturer: To be filled by O.E.M.
        Type: Tablet

The axp288 fuel-gauge driver reads the charger/fuel-gauge enable register, as
some vendors actually bother to set the charger / fg enable bits to 0 there.

Normally we blacklist the fuel-gauge on systems where there is no battery but
the firmware does not bother to disable things, but this is done based on
DMI strings and your systems DMI strings are not usable for blacklisting:

System Information
        Manufacturer: Default string
        Product Name: Default string
        Version: Default string

Base Board Information
        Manufacturer: To be filled by O.E.M.
        Product Name: Cherry Trail CR
        Version: 1.0

Chassis Information
        Manufacturer: To be filled by O.E.M.

All way to generic, so nothing we can do here.

> A second example is, I cannot sustain a SSH connection unless I (1) disable
> "put computer to sleep when battery is low" and (2) physically log into the
> machine. Without disabling sleep the network is dropped after about 15
> minutes. Without a physical login SSH is dropped after 15 or 30 minutes.

This is because GNOME identifies your device as a tablet and thus defaults
to somewhat aggressive power-saving.

Comment 5 Hans de Goede 2019-03-21 09:53:38 UTC
Now as for your shutdown problem, that is weird.

The only thing which comes to mind is that there seem to be some issues with the wifi/bt:

[   12.759815] hci_uart_bcm: probe of serial0-0 failed with error -16
[   15.697798] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.txt failed with error -2

Do you have an image of the original Windows install which came with the machine ?

There should be a bcm or similar prefixed directory under Windows/System32/DriverStore
there (or multiple such directories) if you can make a zip of those available somewhere
(please do not attach here) I can look up the necessary file (e.g. something like
43455r1nvram.txt) and then we can get your wifi to work, no idea if that will
help with the shutdown, but it is good to see if we can fix this regardless.

Can you confirm that that amazon link is the exact machine you bought ?
In that case it is the same machine as this one:
https://www.aliexpress.com/item/wintel-pro-MINI-PC-intel-atom-X5-Z8350-1-44Ghz-quad-core-2GB-32GB-with-WIFI/32968469903.html

And I might get myself one to play around with.

Comment 6 Jeffrey Walton 2019-03-21 10:21:58 UTC
(In reply to Hans de Goede from comment #5)
> Now as for your shutdown problem, that is weird.
> 
> The only thing which comes to mind is that there seem to be some issues with
> the wifi/bt:
> 
> [   12.759815] hci_uart_bcm: probe of serial0-0 failed with error -16
> [   15.697798] brcmfmac mmc1:0001:1: Direct firmware load for
> brcm/brcmfmac43455-sdio.txt failed with error -2
> 
> Do you have an image of the original Windows install which came with the
> machine ?

No. Windows 10 was preloaded and summarily removed.

I'm trying to figure out how to reload Windows 10 again. Windows appears to have a really twisted process. Apparently one cannot simply download an ISO. Cf., https://www.microsoft.com/en-us/software-download/windows10.

> There should be a bcm or similar prefixed directory under
> Windows/System32/DriverStore
> there (or multiple such directories) if you can make a zip of those
> available somewhere
> (please do not attach here) I can look up the necessary file (e.g. something
> like
> 43455r1nvram.txt) and then we can get your wifi to work, no idea if that will
> help with the shutdown, but it is good to see if we can fix this regardless.

I _think_ this may be the APEC page with the drivers: https://www.iacepc.com/download/.

I am less sure about which one I need. They seem to conflate T5 (a thumb drive pc) and the T8 (minipc in this report). At times they say a warez applies to both; at other times they say a warez applies to T5 but do not offer a separate T8 program. 

> Can you confirm that that amazon link is the exact machine you bought ?

Yes. That was the actual page I ordered it from. (And the page now says, "You purchased this item on ...").

> In that case it is the same machine as this one:
> https://www.aliexpress.com/item/wintel-pro-MINI-PC-intel-atom-X5-Z8350-1-
> 44Ghz-quad-core-2GB-32GB-with-WIFI/32968469903.html

Yes, that looks the same. This site may be helpful, too: https://www.iacepc.com/

> And I might get myself one to play around with.

It is not a bad machine for price and performance. I picked it up for testing when I found a RPI-3B+ and Tinkerboard were not meeting expectations.

I would recommend it for a entry level Linux machine (sans these problems).

Comment 7 Jeffrey Walton 2019-03-21 10:32:54 UTC
(In reply to Hans de Goede from comment #4)
> (In reply to Jeffrey Walton from comment #2)
> > There are some other unusual things going on with this machine. As a first
> > example, GNOME reports batter status like a cell phone or laptop even though
> > there is no battery. And more amazingly, it decrements the charge over time.
> > It will start at 97% or so and slowly decrease to a minimum like 5%.
> 
> Yes, well the vendor took a tablet design and then put it in a different
> enclosure
> and they did not bother to disable the charger/fuel-gauge part of the PMIC
> in there BIOS, from your dmidecode:
> 
> Chassis Information
>         Manufacturer: To be filled by O.E.M.
>         Type: Tablet
> 
> The axp288 fuel-gauge driver reads the charger/fuel-gauge enable register, as
> some vendors actually bother to set the charger / fg enable bits to 0 there.
> 
> Normally we blacklist the fuel-gauge on systems where there is no battery but
> the firmware does not bother to disable things, but this is done based on
> DMI strings and your systems DMI strings are not usable for blacklisting:
> 
> System Information
>         Manufacturer: Default string
>         Product Name: Default string
>         Version: Default string
> 
> Base Board Information
>         Manufacturer: To be filled by O.E.M.
>         Product Name: Cherry Trail CR
>         Version: 1.0
> 
> Chassis Information
>         Manufacturer: To be filled by O.E.M.
> 
> All way to generic, so nothing we can do here.

Ack, thanks.

I see a lot of "To be filled by O.E.M." that go un-filled by folks like ACEPC. I think this is the first time it caused a lot of problems. (I'm guessing I missed a lot of little problems in the past).

I contacted the company and requested an updated BIOS/UEFI with the Chassis Information and System Information filled-in as expected. The email cites this issue report. I can post the request here, if it would be helpful.

> > A second example is, I cannot sustain a SSH connection unless I (1) disable
> > "put computer to sleep when battery is low" and (2) physically log into the
> > machine. Without disabling sleep the network is dropped after about 15
> > minutes. Without a physical login SSH is dropped after 15 or 30 minutes.
> 
> This is because GNOME identifies your device as a tablet and thus defaults
> to somewhat aggressive power-saving.

I'm looking for a switch to override GNOME's default handling of the device. I'm hoping there's a --device-class=desktop option (or similar) I can add to GNOME when it starts.

Comment 8 Hans de Goede 2019-03-21 14:28:16 UTC
Created attachment 1546521 [details]
brcmfmac43455-sdio.txt

Ok, after some digging through their forums I've found the nvram file for the wifi.

Please save the attached file as:

/lib/firmware/brcm/brcmfmac43455-sdio.txt

And then do:

sudo rmmod brcmfmac
sudo modprobe brcmfmac

And after that the wifi will hopefully work (this may require a reboot) not sure if this will fix your shutdown issues, but it will be good to get the wifi working regardless.

As for the auto-suspend issues with GNOME, if you are running the machine headless you do:

systemctl set-default -f multi-user.target

To simply not start the graphical login-manager at all.

Comment 9 Jeffrey Walton 2019-03-21 14:58:48 UTC
(In reply to Hans de Goede from comment #8)
> Created attachment 1546521 [details]
> brcmfmac43455-sdio.txt
> 
> Ok, after some digging through their forums I've found the nvram file for
> the wifi.
> 
> Please save the attached file as:
> 
> /lib/firmware/brcm/brcmfmac43455-sdio.txt
> 
> And then do:
> 
> sudo rmmod brcmfmac
> sudo modprobe brcmfmac
> 
> And after that the wifi will hopefully work (this may require a reboot) not
> sure if this will fix your shutdown issues, but it will be good to get the
> wifi working regardless.
> 
> As for the auto-suspend issues with GNOME, if you are running the machine
> headless you do:
> 
> systemctl set-default -f multi-user.target
> 
> To simply not start the graphical login-manager at all.

Thanks man.

Do you want remote access to the box? I'm happy to provide root access. I need authorized_keys; send to noloader, gmail.  (I've got about a dozen machines shared for testing. It is no big deal to me).

Comment 10 Jeffrey Walton 2019-03-21 15:04:07 UTC
(In reply to Hans de Goede from comment #8)
> Created attachment 1546521 [details]
> brcmfmac43455-sdio.txt
> 
> Ok, after some digging through their forums I've found the nvram file for
> the wifi.
> 
> Please save the attached file as:
> 
> /lib/firmware/brcm/brcmfmac43455-sdio.txt
> 
> And then do:
> 
> sudo rmmod brcmfmac
> sudo modprobe brcmfmac

Modprobe result:

[101759.740895] usbcore: deregistering interface driver brcmfmac
[101767.393958] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[101767.394159] usbcore: registered new interface driver brcmfmac
[101767.401956] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.Default string-Default string.txt failed with error -2
[101767.495145] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[101767.495240] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.clm_blob failed with error -2
[101767.495245] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available
[101767.497213] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar  1 2015 07:29:38 version 7.45.18 (r538002) FWID 01-6a2c8ad4
[101767.665467] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready


I think you fixed the reboot problem. I 'shutdown -r now'. The machine was already POST'ing before I turned around to look at the monitor.

Comment 11 Jeffrey Walton 2019-03-21 15:24:27 UTC
(In reply to Jeffrey Walton from comment #10)
> (In reply to Hans de Goede from comment #8)
> > Created attachment 1546521 [details]
> > brcmfmac43455-sdio.txt
> > 
> > Ok, after some digging through their forums I've found the nvram file for
> > the wifi.
> > 
> > Please save the attached file as:
> > 
> > /lib/firmware/brcm/brcmfmac43455-sdio.txt
> > 
> > And then do:
> > 
> > sudo rmmod brcmfmac
> > sudo modprobe brcmfmac
> 
> Modprobe result:
> 
> [101759.740895] usbcore: deregistering interface driver brcmfmac
> [101767.393958] brcmfmac: brcmf_fw_alloc_request: using
> brcm/brcmfmac43455-sdio for chip BCM4345/6
> [101767.394159] usbcore: registered new interface driver brcmfmac
> [101767.401956] brcmfmac mmc1:0001:1: Direct firmware load for
> brcm/brcmfmac43455-sdio.Default string-Default string.txt failed with error
> -2
> [101767.495145] brcmfmac: brcmf_fw_alloc_request: using
> brcm/brcmfmac43455-sdio for chip BCM4345/6
> [101767.495240] brcmfmac mmc1:0001:1: Direct firmware load for
> brcm/brcmfmac43455-sdio.clm_blob failed with error -2
> [101767.495245] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available
> (err=-2), device may have limited channels available
> [101767.497213] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0:
> Mar  1 2015 07:29:38 version 7.45.18 (r538002) FWID 01-6a2c8ad4
> [101767.665467] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
> 
> 
> I think you fixed the reboot problem. I 'shutdown -r now'. The machine was
> already POST'ing before I turned around to look at the monitor.

Yeah, that fixed it. The machine rebooted in about 3 seconds on a second try. Thanks.

Don't worry about IPv6 failure. I don't run IPv6 so the device can't get a DHCP address.

You are still welcomed to remote access if you would like it.

Comment 12 Jeffrey Walton 2019-03-22 05:07:22 UTC
The folks at ACEPC (In reply to Jeffrey Walton from comment #7)
> (In reply to Hans de Goede from comment #4)
> > (In reply to Jeffrey Walton from comment #2)
> > > There are some other unusual things going on with this machine. As a first
> > > example, GNOME reports batter status like a cell phone or laptop even though
> > > there is no battery. And more amazingly, it decrements the charge over time.
> > > It will start at 97% or so and slowly decrease to a minimum like 5%.
> > 
> > Yes, well the vendor took a tablet design and then put it in a different
> > enclosure
> > and they did not bother to disable the charger/fuel-gauge part of the PMIC
> > in there BIOS, from your dmidecode:
> > 
> > Chassis Information
> >         Manufacturer: To be filled by O.E.M.
> >         Type: Tablet
> > 
> > The axp288 fuel-gauge driver reads the charger/fuel-gauge enable register, as
> > some vendors actually bother to set the charger / fg enable bits to 0 there.
> > 
> > Normally we blacklist the fuel-gauge on systems where there is no battery but
> > the firmware does not bother to disable things, but this is done based on
> > DMI strings and your systems DMI strings are not usable for blacklisting:
> > 
> > System Information
> >         Manufacturer: Default string
> >         Product Name: Default string
> >         Version: Default string
> > 
> > Base Board Information
> >         Manufacturer: To be filled by O.E.M.
> >         Product Name: Cherry Trail CR
> >         Version: 1.0
> > 
> > Chassis Information
> >         Manufacturer: To be filled by O.E.M.
> > 
> > All way to generic, so nothing we can do here.
> ...
> 
> I contacted the company and requested an updated BIOS/UEFI with the Chassis
> Information and System Information filled-in as expected. The email cites
> this issue report. I can post the request here, if it would be helpful.

The folks at ACEPC replied and provided a link to a BIOS. I presume the download is an updated BIOS. The link is https://www.dropbox.com/s/uz3gknrvz0c6apo/T5_T8_2G_32G_8723.zip?dl=0

The name "T5_T8_2G_32G_8723" is the same as the one on their download page at https://www.iacepc.com/download/. I'm guessing the name does not include version information.

I'm getting ready to re-install Windows, update the BIOS update, then re-install Fedora.

By the way, I saw someone else was having problems with the battery indicator on Ubuntu. See https://www.amazon.com/gp/customer-reviews/R2E9C94U8YDGEF .

Comment 13 Hans de Goede 2019-03-22 07:33:48 UTC
I would not install that BIOS update if I were you, I do not believe it is an update (just the factory image) and the 8723 in the name in the name indicate it is for a version with the RTL8723BS wifi chip, while your version has a broadcom wifi chip!

I need to also do a dmi match to be able to push the nvram firmware file to linux-firmware upstream (it is model specific so it needs a model specific name). I plan to use the BIOS version + BIOS date as additional checks, that is the usual work around for too generic strings.

Since I'm going to do the BIOS date hack anyway I also plan to fix the battery charging problem the same way.

Comment 14 Jeffrey Walton 2019-03-22 09:04:04 UTC
(In reply to Hans de Goede from comment #13)
> I would not install that BIOS update if I were you, I do not believe it is
> an update (just the factory image) and the 8723 in the name in the name
> indicate it is for a version with the RTL8723BS wifi chip, while your
> version has a broadcom wifi chip!

Yeah, you're right.

It was a BIOS, but it was version 1.0 from July 2017. The thing about this version is, it uses 'Type: 30'. I'm guessing that sidesteps the 'Type: Tablet' issue. (The other fields look the same - a lot of "To be filled by O.E.M."

> I need to also do a dmi match to be able to push the nvram firmware file to
> linux-firmware upstream (it is model specific so it needs a model specific
> name). I plan to use the BIOS version + BIOS date as additional checks, that
> is the usual work around for too generic strings.
> 
> Since I'm going to do the BIOS date hack anyway I also plan to fix the
> battery charging problem the same way.

I can load Linux again with the v1.0 BIOS if you'd like a dmidecode snapshot. Currently the machine is ruunning Windows so I can patch the firmware.

Otherwise, I'm going back to v1.7 of the BIOS and Fedora 29.

Comment 15 Hans de Goede 2019-03-22 10:06:39 UTC
Since you have the new (old) BIOS now, it would be good to gather some info before going back to the 1.7 BIOS.
You should be able to do this from a livecd image dd-ed to a USB stick, no need to reinstall.

I wonder did you do a BIOS upgrade before, or did the machine ship with the 1.7 BIOS ?

Please collect the following info running the new (old) BIOS from the livecd:

1. dmesg output
2. dmidecode output
3. run "sudo dnf install -y acpicatools" (this will install to the ramdisk overlay the live-env has), and then:
   run "sudo acpidump -o acpidump.acepc-t8-bios1.0" and save the acpidump.acepc-t8-bios1.0 file somewhere.

After that feel free to go back to the newer BIOS, it would also be good to get an
acpidump of the new BIOS:  "sudo acpidump -o acpidump.acepc-t8-bios1.7"

I'm going to need those acpidumps to see if I can eventually also fix the bluetooth not working.

Comment 16 Hans de Goede 2019-03-22 10:07:30 UTC
Correction, "acpicatools" above should be "acpica-tools".

Comment 17 Jeffrey Walton 2019-03-22 11:11:11 UTC
(In reply to Hans de Goede from comment #15)
> ...
> I wonder did you do a BIOS upgrade before, or did the machine ship with the
> 1.7 BIOS ?

The machine shipped with v1.7.

The other info will follow shortly.

Comment 18 Jeffrey Walton 2019-03-22 11:12:07 UTC
Created attachment 1546821 [details]
UEFI v1.0 dumps

Comment 19 Hans de Goede 2019-03-28 15:29:12 UTC
Hi Jeffrey,

I've written 3 patches to make Linux work better on the ACEPC T8:

1) Use DMI identification (including BIOS date because of generic strings) to make brcmfmac look for: brcm/brcmfmac43455-sdio.acepc-t8.txt as nvram file. Once your testing confirms that this works this will allow me to add the nvram file to the upstream linux-firmware and then when distros get both the new kernel with the DMI based patch and a new linux-firmware the wifi will just work for users (and the shutdown/reboot problem will go away)

2) Use DMI identification to blacklist the axp288_fuel_gauge driver so that you will not get a bogus battery device on the T8 and so that e.g. GNOME will not shutdown the machine due to the battery being critically low. After this you should have just the axp288_charger show up in "ls /sys/class/power_supply" and
the axp288_fuel_gauge entry there should be gone (and upower / GNOME should no longer see a battery)

3) Add some debugging to the bluetooth hci-bcm code to figure out why it is giving a -16 code and bluetooth is not working.

I've started a test kernel-build with these patches added here:

https://koji.fedoraproject.org/koji/taskinfo?taskID=33804240

This will take a couple of hours to finish, once it is done, please download, install and reboot into this kernel and then:
a) check the fuel_gauge indeed no longer shows in "ls /sys/class/power_supply" 
b) collect the output of "dmesg" and attach that here, then I can check 1. from above and also see what extra info patch 3 gives me.

For generic instructions for installing a kernel from koji see:
https://fedorapeople.org/~jwrdegoede/kernel-test-instructions.txt

Note koji keeps test builds around only for about 3-4 days, to free up the disk space afterwards. If you do not have time right now to do the testing, please  at least download the necessary rpms for later, see the instructions for which files you need.

Regards,

Hans

Comment 20 Hans de Goede 2019-04-03 10:34:54 UTC
Have you had a chance to test the test kernel-build with the patches for this yet?

Comment 21 Hans de Goede 2019-04-15 18:37:07 UTC
Ping? It would be nice to get some feedback on this so that I can submit the patches for this upstream.

Comment 22 Jeffrey Walton 2019-04-15 18:43:58 UTC
(In reply to Hans de Goede from comment #21)
> Ping? It would be nice to get some feedback on this so that I can submit the
> patches for this upstream.

My bad Hans. I got tied up on another project.

Give me a day or two to test things.

An FYI... The device is at BIOS version 1.0 from July 2017. Since I downgraded the company has not provided a link to the v1.7 BIOS for download. They stopped communicating with me when I made the request fr the 1.7 BIOS.

I can buy another device that should ship with the v1.7 BIOS if you want it tested (that was this device's config before I downloaded). Do you want me to purchase another device for testing?

Comment 23 Hans de Goede 2019-04-15 19:58:35 UTC
Bummer that they cannot provide you the 1.7 BIOS, but there is no need to buy another device.

I do need to respin the patches so that will work with both the DMI strings from the v1.7 BIOS and those from the v1.0 BIOS. Also koji has garbage collected the old test build I did, so I need to do a new build anyways.

I'll start preparing a new build.

Comment 24 Hans de Goede 2019-04-16 15:43:45 UTC
Ok I've updated the 2 patches so that they should also work with your new (old) BIOS.

A new kernel scratch/test build with these patches added is building here now:
https://koji.fedoraproject.org/koji/taskinfo?taskID=34215831

Note this is still building atm, this takes a couple of hours. For testing instructions see comment 19.

Looking at the dmesg you attached from the 1.0 BIOS, it seems that that BIOS does not include info on the bluetooth part of the wifi/bt card in your T8, at least the following message is missing from dmesg with the 1.0 BIOS:

[   12.759815] hci_uart_bcm: probe of serial0-0 failed with error -16

Can you check if you can select which model bluetooth module you have somewhere in your BIOS and if there is an option for this, change it to a BRCM or BCM model?

I also noticed that the 1.7 BIOS has a "System SKU" of "T11" in its DMI info and to get the nvram file I also needed to download T11 drivers, so I think that the version of the T8 with the broadcom wifi like yours is using the T11 BIOS as the T11 also has the broadcom wifi and otherwise there is not much of a difference. So perhaps you can ask acepc support if they have a 1.7 T11 BIOS for you?

Comment 25 Jeffrey Walton 2019-04-19 22:25:05 UTC
(In reply to Hans de Goede from comment #24)
> Ok I've updated the 2 patches so that they should also work with your new
> (old) BIOS.
> 
> A new kernel scratch/test build with these patches added is building here
> now:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=34215831
> 
> Note this is still building atm, this takes a couple of hours. For testing
> instructions see comment 19.

Ack, I'm getting ready to start testing the v1.0 BIOS machine.

> Can you check if you can select which model bluetooth module you have
> somewhere in your BIOS and if there is an option for this, change it to a
> BRCM or BCM model?

Yes, will do. I'll report back with options.

> I also noticed that the 1.7 BIOS has a "System SKU" of "T11" in its DMI info
> and to get the nvram file I also needed to download T11 drivers, so I think
> that the version of the T8 with the broadcom wifi like yours is using the
> T11 BIOS as the T11 also has the broadcom wifi and otherwise there is not
> much of a difference. So perhaps you can ask acepc support if they have a
> 1.7 T11 BIOS for you?

OK, will do.

I also purchased another new T8 which should come with the v1.7 BIOS. Since we are managing to sync up we may as well attempt to knock out all the problems with this family of machines. The new machine should arrive this weekend or early next week.

Comment 26 Jeffrey Walton 2019-04-20 06:34:46 UTC
Hi Hans. I've got the kernel installed. These three RPM's were downloaded and installed with 'sudo rpm -ivh --oldpackage kernel*.rpm'.

* https://kojipkgs.fedoraproject.org//work/tasks/5838/34215838/kernel-core-5.0.7-300.hdg20190416.fc30.x86_64.rpm
* https://kojipkgs.fedoraproject.org//work/tasks/5838/34215838/kernel-modules-5.0.7-300.hdg20190416.fc30.x86_64.rpm
* https://kojipkgs.fedoraproject.org//work/tasks/5838/34215838/kernel-modules-extra-5.0.7-300.hdg20190416.fc30.x86_64.rpm

The machine shutdown and rebooted OK.

After reboot I did not log back into the machine. Instead, I used SSH to log in remotely. Unfortunately, the SSH session was killed when the screen shutoff.

I enabled GNOME settings (1) Blank Screen: Never, and (2) Power Suspend: Off.

This is/was annoying. It may not be present since I changed the GNOME settings. When I press a key or shake the mouse the computer tries to wake up from the low power state. It starts to wake the screen, but the screen never turns on (completely?). It stays a chalky grey, and no login screen appears. Then I have to unplug the machine and power it back on.

*************************

Now the requests from comments 19-24 (I hope I got them all).

19> and the shutdown/reboot problem will go away

    Confirmed, no reboot problem.

19> a) check the fuel_gauge indeed no longer shows in "ls /sys/class/power_supply"

    $ ls /sys/class/power_supply
    axp288_charger

19> (and upower / GNOME should no longer see a battery)

    Confirmed, no longer present in the upper-right of the desktop

19> b) collect the output of "dmesg" and attach that here, then I can check 1. from
19>    above and also see what extra info patch 3 gives me.

    Attached as dmesg-hdg.txt.zip

24> Can you check if you can select which model bluetooth module you have somewhere in
24> your BIOS and if there is an option for this, change it to a BRCM or BCM model?

    The 1.0 BIOS is lame. It does not have the rich settings like 1.7.
    I did not see a similar setting. In fact, I could not find an area
    to setup buses likes USB and PCI.

24> I also noticed that the 1.7 BIOS has a "System SKU" of "T11" in its DMI info and to
24> get the nvram file I also needed to download T11 drivers

    dmidecode-hdg.txt.zip attached. I suspect it is the same as the other one.

24> So perhaps you can ask acepc support if they have a 1.7 T11 BIOS for you?

    I wrote to ACEPC on Wednesday evening. No reply as of Saturday evening.

Comment 27 Jeffrey Walton 2019-04-20 06:36:06 UTC
Created attachment 1556642 [details]
Output of dmesg using HDG kernel

Comment 28 Jeffrey Walton 2019-04-20 06:36:46 UTC
Created attachment 1556643 [details]
Output from dmidecode using HDG kernel

Comment 29 Jeffrey Walton 2019-04-20 07:18:31 UTC
(In reply to Jeffrey Walton from comment #26)
> ...
> 
> 24> So perhaps you can ask acepc support if they have a 1.7 T11 BIOS for you?
> 
>     I wrote to ACEPC on Wednesday evening. No reply as of Saturday evening.

Email sent to support at info@iacepc.com.

On Wed, Apr 17, 2019 at 8:15 PM ... wrote:
>
> Hi Everyone,
>
> I found the T8 Wifi drivers at
> https://mega.nz/#F!hlQAEbaC!fhySXJ7rNqv72HIIJJFIgw . I am looking for
> the T11 Wifi drivers.
>
> Could you please provide a download for the T11 Wifi drivers.
>
> Thanks in advance,

Comment 30 Hans de Goede 2019-04-20 09:52:52 UTC
Thank you for the testing.

WRT teaching the kernel to load the wifi nvram using a proper filename, that seems to have succeeded:
[   14.364706] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.acepc-t8.txt failed with error -2

But it seems you did not add the nvram file like you did last time:
[   14.364740] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.txt failed with error -2

So it seems that the reboot problem is gone without needing the nvram, this might have something to do with you using the 1.0 BIOS now.


The problem with a battery falsely being registered is also gone now, so with that confirmed I can submit the 2 kernel patches for this upstream as well as submit the nvram file upstream under the brcm/brcmfmac43455-sdio.acepc-t8.txt name.


To further debug the -EBUSY error with the bluetooth, we will need to wait till you receive the other T8 with the 1.7 BIOS. Please save the kernel rpms from the testbuild somewhere for this. Since this is a testbuild koji will garbage collect them. and throw them away.


As for the auto-suspend problem, this is a gdm "feature" other then booting into multi-user mode instead of graphical mode, so that gdm does not run, I do not think there is much we can do about this.

Comment 31 Jeffrey Walton 2019-04-20 17:33:08 UTC
(In reply to Hans de Goede from comment #30)
> 
> WRT teaching the kernel to load the wifi nvram using a proper filename, that
> seems to have succeeded:
> [   14.364706] brcmfmac mmc1:0001:1: Direct firmware load for
> brcm/brcmfmac43455-sdio.acepc-t8.txt failed with error -2
> 
> But it seems you did not add the nvram file like you did last time:
> [   14.364740] brcmfmac mmc1:0001:1: Direct firmware load for
> brcm/brcmfmac43455-sdio.txt failed with error -2

Yeah, I was not sure of the new kernels had some magic to handle it; or if I should add the file to /lib/firmware/brcm/brcmfmac43455-sdio.txt again.

Since adding it again with the HDG kernel:

$ dmesg | grep -i brcm
[   14.666608] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   14.667170] usbcore: registered new interface driver brcmfmac
[   14.672189] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.acepc-t8.txt failed with error -2
[   14.765279] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   14.765354] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available
[   14.766288] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar  1 2015 07:29:38 version 7.45.18 (r538002) FWID 01-6a2c8ad4



> To further debug the -EBUSY error with the bluetooth, we will need to wait
> till you receive the other T8 with the 1.7 BIOS. Please save the kernel rpms
> from the testbuild somewhere for this. Since this is a testbuild koji will
> garbage collect them. and throw them away.

Ack. With a little luck I will have the results for you Saturday night.

Comment 32 Justin M. Forbes 2019-08-20 17:39:59 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.

Fedora 29 has now been rebased to 5.2.9-100.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30.

If you experience different issues, please open a new bug report for those.

Comment 33 Justin M. Forbes 2019-09-17 20:03:18 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.


Note You need to log in before you can comment on or make changes to this bug.