Bug 2015174 - PInebook Pro fails to fully poweroff when booting from NVMe
Summary: PInebook Pro fails to fully poweroff when booting from NVMe
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: arm-trusted-firmware
Version: 35
Hardware: aarch64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2012354 (view as bug list)
Depends On:
Blocks: ARMTracker
TreeView+ depends on / blocked
 
Reported: 2021-10-18 15:03 UTC by Jordan Williams
Modified: 2022-06-24 16:55 UTC (History)
12 users (show)

Fixed In Version: arm-trusted-firmware-2.5-3.fc35
Clone Of: 2012354
Environment:
Last Closed: 2021-11-24 23:41:13 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jordan Williams 2021-10-18 15:03:52 UTC
+++ This bug was initially created as a clone of Bug #2012354 +++
Re-creating bug to fix incorrect "Groups" selection.

Description of problem:

When booting Fedora 35 on an NVMe SSD on the Pinebook Pro via U-Boot on the SPI flash, the system does not completely power-down but remains running.
The bottom left of the laptop remains slightly warm, as if the CPU is still running.
The power drains completely unless hard-powering down the system by holding the power button for about 10 seconds.
The screen turns off and the shutdown process appears to complete.

Version-Release number of selected component (if applicable):

2021.04-1.fc34
2021.10-1.fc35


How reproducible:

100%

Steps to Reproduce:
1. Install U-Boot on SPI according to https://nullr0ute.com/2021/05/fedora-on-the-pinebook-pro/
2. Use the arm-image-installer tool to flash Fedora 35 Workstation to an NVMe disk.
Note this installed to an NVMe drive over a USB connection, which is why the disk is `/dev/sda` here, but the NVMe is connected directly to the PCIe interface using PINE64's offical NVMe adapter for the Pinebook Pro.

```
sudo arm-image-installer \
  --image="$HOME/Downloads/Fedora-Workstation-35.aarch64.raw.xz" \
  --target=none \
  --media=/dev/sda \
  --resizefs \
  --relabel \
  --addconsole \
  --showboot \
  --addkey="$HOME/.ssh/id_rsa.pub" \
  --norootpass \
  -y
```

3. Install the NVMe drive in the Pinebook Pro.
4. Boot.
5. Complete the installation.
6. Log in and update / do whatever you like.
7. Power down the machine in software.

Actual results:

The CPU remains running, keeping the bottom of the Pinebook Pro warm and draining the battery, though the screen is powered off.

Expected results:

A complete shutdown.

Additional info:

NVMe disk: SK hynix SuperCore Series - Gold P31 (Model #: SHGP31-1000GM-2)

--- Additional comment from Jordan Williams on 2021-10-08 23:43:49 UTC ---

Sorry, meant for this bug to be public but I can't unselect the `fedora_contrib_private` group from the list.

Comment 1 Jordan Williams 2021-10-18 15:05:06 UTC
*** Bug 2012354 has been marked as a duplicate of this bug. ***

Comment 2 nocar 2021-11-06 01:41:24 UTC
I have been encountering this issue as well, it is not just on my pinebook pro but also on my rockpro64 using the same method. The only difference is I have been encountering this when installed on the eMMC module.

Comment 3 Fedora Update System 2021-11-15 21:04:23 UTC
FEDORA-2021-15ead4e942 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-15ead4e942

Comment 4 Fedora Update System 2021-11-16 15:52:28 UTC
FEDORA-2021-15ead4e942 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-15ead4e942`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-15ead4e942

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2021-11-22 01:20:25 UTC
FEDORA-2021-15ead4e942 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 6 Peter Robinson 2021-11-24 09:26:46 UTC
> When booting Fedora 35 on an NVMe SSD on the Pinebook Pro via U-Boot on the
> SPI flash, the system does not completely power-down but remains running.
> The bottom left of the laptop remains slightly warm, as if the CPU is still
> running.

So running the 5.15.x kernel with OS on NVME and firmware on SPI I can't recreate this. Running poweroff and monitoring from the serial console the device turns off AFAICT, the power LED goes off and the serial console goes dead.

Comment 7 nocar 2021-11-24 14:28:39 UTC
While I'm not using the NVMe drive, the first time I figured out what was going on was when I tried to power off my pbp in a very dark (basically pitch black) room and noticed the backlight was still on at a very low level, not sure if this will help or not

Comment 8 Jordan Williams 2021-11-24 18:09:56 UTC
> > When booting Fedora 35 on an NVMe SSD on the Pinebook Pro via U-Boot on the
> > SPI flash, the system does not completely power-down but remains running.
> > The bottom left of the laptop remains slightly warm, as if the CPU is still
> > running.
> 
> So running the 5.15.x kernel with OS on NVME and firmware on SPI I can't
> recreate this. Running poweroff and monitoring from the serial console the
> device turns off AFAICT, the power LED goes off and the serial console goes
> dead.
(In reply to Peter Robinson from comment #6)

I was able to notice the issue due to the fact that the battery would drain after powering off.
The easiest way to test this was for me to charge the Pinebook Pro to pretty full charge, power it down via GNOME's dropdown, and then try to start it up the next day.
It doesn't start the next day due to the drained battery.
When I do a soft power down followed by a hard shutdown, holding the power button down for 10 seconds, the next day there is plenty of battery.
Therefore, the battery doesn't appear to be the issue.
From all appearances, the device is completely shut down after a soft power down.
No LED's blink and the computer screen's backlight appears to be completely off.
I'm wondering if somehow the NVMe is remaining in a low-power state somehow while the rest of the machine has powered off.

This weekend I'll re-flash the SPI, ensure Fedora Workstation 35 is completely up-to-date, and re-test.
I have PINE64's serial adapter, so I can also hook it up to serial to see if there's any indication that it isn't completely powering down.

Comment 9 Dan Horák 2021-11-24 19:10:12 UTC
I have same experience as Jordan describes, but with F-34 PBP without NVMe, u-boot on SPI. I will upgrade and recheck.

Comment 10 Peter Robinson 2021-11-24 23:41:13 UTC
Sorry, I'm closing this as can't fix.

I shut down my pinebook pro and I don't see any light from the backlight.

Ultimately this is a $200 device with free software, I do the support and enablement in Fedora within my time constraints and my ability. I don't have access to any of the low level upstream data sheets and I don't have the low level debug skill sets or the time to devote to fixing this problem. If it gets fixed upstream will will consume the upstream fix (I engaged upstream for a suspend fix) but I don't have the time or resources to deal with this directly so I'm closing it to set expectations appropriately.

Do feel free to engage with upstream (probably needs to be ATF - https://github.com/ARM-software/arm-trusted-firmware/) and if you get/find a fix to reference it here and I will look at it for a Fedora specific fix.

Comment 11 Dan Horák 2021-11-25 08:59:29 UTC
one more data point to capture here - IIRC the preinstalled Manjaro KDE(?) haven't suffered from such issue.

Comment 12 Peter Robinson 2021-11-25 09:25:54 UTC
(In reply to Dan Horák from comment #11)
> one more data point to capture here - IIRC the preinstalled Manjaro KDE(?)
> haven't suffered from such issue.

Well if you can point me to the patch that deals with that (could be arm-trusted-firmware/u-boot/kernel) I will review it but I just don't have the time.

Comment 13 Dan Horák 2021-11-25 12:16:32 UTC
- https://koji.fedoraproject.org/koji/taskinfo?taskID=79255229 is a scratch build of uboot 2022.01-rc2 with ATF 2.6-rc1
- I have updated SPI with this build, let's see what will happen ...

Comment 14 Dan Horák 2021-11-26 09:06:26 UTC
(In reply to Dan Horák from comment #13)
> - https://koji.fedoraproject.org/koji/taskinfo?taskID=79255229 is a scratch
> build of uboot 2022.01-rc2 with ATF 2.6-rc1
> - I have updated SPI with this build, let's see what will happen ...

not sure if it's a consequence of the uboot build above, but I can't power on my PBP any more after (soft) powering it off and keeping it unplugged for ~8 hours.

Comment 15 Dan Horák 2021-11-26 10:58:59 UTC
(In reply to Dan Horák from comment #14)
> (In reply to Dan Horák from comment #13)
> > - https://koji.fedoraproject.org/koji/taskinfo?taskID=79255229 is a scratch
> > build of uboot 2022.01-rc2 with ATF 2.6-rc1
> > - I have updated SPI with this build, let's see what will happen ...
> 
> not sure if it's a consequence of the uboot build above, but I can't power
> on my PBP any more after (soft) powering it off and keeping it unplugged for
> ~8 hours.

uff, it's alive again, seems 20+ sec long pressed power button "fixed" it

Comment 16 Stephan W. 2022-06-24 11:37:28 UTC
Seems to be a RK3399 UEFI poweroff issues
This is the Tow-Boot workaround:
https://github.com/Tow-Boot/Tow-Boot/commit/818cae1b84a7702f2a509927f2819900c2881979#diff-56266a451b63cbd4f6c641a20c89f32867b9c18c5bc3c65ac6af1cc6831ffe16

Comment 17 Peter Robinson 2022-06-24 14:41:51 UTC
(In reply to Stephan W. from comment #16)
> Seems to be a RK3399 UEFI poweroff issues
> This is the Tow-Boot workaround:
> https://github.com/Tow-Boot/Tow-Boot/commit/
> 818cae1b84a7702f2a509927f2819900c2881979#diff-
> 56266a451b63cbd4f6c641a20c89f32867b9c18c5bc3c65ac6af1cc6831ffe16

That work around is categorically wrong, we won't be using that.

Comment 18 Dan Horák 2022-06-24 15:38:29 UTC
Stephen, I think pointers to what is actually missing in u-boot would be great.

Comment 19 Peter Robinson 2022-06-24 16:55:27 UTC
(In reply to Dan Horák from comment #18)
> Stephen, I think pointers to what is actually missing in u-boot would be
> great.

I don't believe it's missing from U-Boot, power is usually dealt by PSCI (Power State Coordination Interface - https://developer.arm.com/Architectures/Power%20State%20Coordination%20Interface) which is handled by TF (Arm Trusted Firmware), so it's more than likely a bug in ATF.


Note You need to log in before you can comment on or make changes to this bug.