Bug 2179624 - rock64 unbootable with uboot 2023.04
Summary: rock64 unbootable with uboot 2023.04
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: uboot-tools
Version: rawhide
Hardware: aarch64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-19 09:35 UTC by bonin' o'brien
Modified: 2023-04-11 22:33 UTC (History)
9 users (show)

Fixed In Version: uboot-tools-2023.04-0.4.rc5.fc38 uboot-tools-2023.04-1.fc38
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-11 22:33:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description bonin' o'brien 2023-03-19 09:35:40 UTC
fedora rawhide up to date
rock64 rk3328
sandisk 64g sd card
partitions all offset 30720 sectors to allow rockchip first 16 megabytes of the card
flash spi blank
dtb manually copied to efi system partition

U-Boot TPL 2023.04-rc4 (Mar 14 2023 - 00:00:00)                                 
LPDDR3, 800MHz                                                                  
BW=32 Col=10 Bk=8 CS0 Row=15 CS=1 Die BW=16 Size=1024MB                         
Trying to boot from BOOTROM                                                     
Returning to boot ROM...                                                        
                                                                                
U-Boot SPL 2023.04-rc4 (Mar 14 2023 - 00:00:00 +0000)                           
Trying to boot from MMC1                                                        
cannot find image node '': -1                                                   
                                                                                
                                                                                
U-Boot 2023.04-rc4 (Mar 14 2023 - 00:00:00 +0000)                               
                                                                                
Model: Pine64 Rock64                                                            
DRAM:  1 GiB (effective 1022 MiB)                                               
PMIC:  RK8050 (on=0x10, off=0x04)                                               
Core:  228 devices, 24 uclasses, devicetree: separate                           
MMC:   mmc@ff500000: 1, mmc@ff520000: 0                                         
Loading Environment from MMC... OK                                              
In:    serial@ff130000                                                          
Out:   serial@ff130000                                                          
Err:   serial@ff130000                                                          
Model: Pine64 Rock64                                                            
Net:   eth0: ethernet@ff540000                                                  
Hit any key to stop autoboot:  0                                                
switch to partitions #0, OK                                                     
mmc1 is current device                                                          
Scanning mmc 1:1...                                                             
Found EFI removable media binary efi/boot/bootaa64.efi                          
Found DTB mmc 1:1 /rockchip/rk3328-rock64.dtb                                   
36275 bytes read in 8 ms (4.3 MiB/s)                                            
966711 bytes read in 49 ms (18.8 MiB/s)                                         
Working FDT set to 1f00000                                                      
Card did not respond to voltage select! : -110                                  
No EFI system partition                                                         
No EFI system partition                                                         
Failed to persist EFI variables                                                 
Booting /efi\boot\bootaa64.efi    

it hangs there, rolled back to 2023.01 and it booted (but not with Kernel 6.3)
also noticed there is no bl31 information in the uboot dialog

Comment 1 David Sebek 2023-03-30 00:02:59 UTC
I am experiencing a similar problem. I had U-Boot version 2022-10 installed in the SPI of my RockPro64 board. When I updated U-Boot in the SPI to version 2023-04-rc4 (from the package uboot-images-armv8-2023.04-0.3.rc4.fc38.noarch), the boot stopped right after this message:

Booting /efi\boot\bootaa64.efi

I tried resetting environment variables:

env default -a
saveenv

which resulted in this error message on the next boot instead:

Hit any key to stop autoboot:  0
## Error: "distro_bootcmd" not defined
=> 

I discovered that there have been commits in the upstream U-Boot repo that address the issue of the undefined distro_bootcmd variable. The fixes are in U-Boot version 2023.04-rc5.

But when I built a custom version of the Fedora's uboot-images-armv8 package (which is a part of the uboot-tools package) with the version 2023.04-rc5 of U-Boot source code, the boot got stuck after this message again (with the default environment variables):

Booting /efi\boot\bootaa64.efi

After that, I built the upstream U-Boot version 2023.04-rc5 using instructions from a Gentoo wiki page. I also built BL31 from the upstream arm-trusted-firmware version 2.8.0. This time, the boot was successful.

To summarize, U-Boot version 2023.04-rc4 in Fedora's package doesn't work on RockPro64. Upstream U-Boot version 2023.04-rc5 works. Simply updating the Fedora's package with U-Boot 2023.04-rc5 didn't work for me.

Comment 2 Peter Robinson 2023-03-30 06:27:43 UTC
Yes, it's a known problem with all rockchips devices, it's almost fixed in upstream rc5, I'm just working on a final fix before I push that build to Fedora.

Comment 3 Peter Robinson 2023-03-30 06:28:44 UTC
> Hit any key to stop autoboot:  0
> ## Error: "distro_bootcmd" not defined
> => 

This is the problem, I have been working with upstream to fix it.

Comment 4 Fedora Update System 2023-04-02 22:33:49 UTC
FEDORA-2023-8ba62c0e8c has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-8ba62c0e8c

Comment 5 Fedora Update System 2023-04-03 02:35:08 UTC
FEDORA-2023-8ba62c0e8c has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-8ba62c0e8c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2023-04-04 00:17:50 UTC
FEDORA-2023-8ba62c0e8c has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 7 bonin' o'brien 2023-04-04 09:40:29 UTC
still dead on rk3328

# cd /usr/share/uboot/rock64-rk3328/
# dd if=idbloader.img of=/dev/mmcblk0 seek=64
208+0 records in
208+0 records out
106496 bytes (106 kB, 104 KiB) copied, 0.0237441 s, 4.5 MB/s
# dd if=u-boot.itb of=/dev/mmcblk0 seek=16384
1465+0 records in
1465+0 records out
750080 bytes (750 kB, 732 KiB) copied, 0.208837 s, 3.6 MB/s
# systemcrl reboot
...
[ 2070.425254] reboot: Restarting system                                        
                                                                             
U-Boot TPL 2023.04-rc5 (Mar 28 2023 - 00:00:00)                                 
LPDDR3, 800MHz                                                                  
BW=32 Col=10 Bk=8 CS0 Row=15 CS=1 Die BW=16 Size=1024MB                         
Trying to boot from BOOTROM                                                     
Returning to boot ROM...                                                        
                                                                                
U-Boot SPL 2023.04-rc5 (Mar 28 2023 - 00:00:00 +0000)                           
Trying to boot from MMC1                                                        
cannot find image node '': -1                                                   
                                                                                
                                                                                
U-Boot 2023.04-rc5 (Mar 28 2023 - 00:00:00 +0000)                               
                                                                                
Model: Pine64 Rock64                                                            
DRAM:  1 GiB (effective 1022 MiB)                                               
PMIC:  RK8050 (on=0x10, off=0x04)                                               
Core:  228 devices, 24 uclasses, devicetree: separate                           
MMC:   mmc@ff500000: 1, mmc@ff520000: 0                                         
Loading Environment from MMC... OK                                              
In:    serial@ff130000                                                          
Out:   serial@ff130000                                                          
Err:   serial@ff130000                                                          
Model: Pine64 Rock64                                                            
Net:   eth0: ethernet@ff540000                                                  
Hit any key to stop autoboot:  0                                                
switch to partitions #0, OK                                                     
mmc1 is current device                                                          
Scanning mmc 1:1...                                                             
Found EFI removable media binary efi/boot/bootaa64.efi                          
Found DTB mmc 1:1 /rockchip/rk3328-rock64.dtb                                   
50894 bytes read in 9 ms (5.4 MiB/s)                                            
966711 bytes read in 49 ms (18.8 MiB/s)                                         
Working FDT set to 1f00000                                                      
Card did not respond to voltage select! : -110                                  
No EFI system partition                                                         
No EFI system partition                                                         
Failed to persist EFI variables                                                 
Booting /efi\boot\bootaa64.efi

Comment 8 Peter Robinson 2023-04-04 10:16:36 UTC
That looks like it's working fine to me:

Found EFI removable media binary efi/boot/bootaa64.efi                          
Found DTB mmc 1:1 /rockchip/rk3328-rock64.dtb                                   
Booting /efi\boot\bootaa64.efi


Those three lines say it's found the storage and started to run shim, can you provide more information? Is that serial console, HDMI, what storage, what devices attached etc.

Comment 9 bonin' o'brien 2023-04-04 10:42:02 UTC
as in original report
>flash spi blank
>dtb manually copied to efi system partition
i.e. i mkdir /boot/efi/rockchip and copy the dtb from the /boot/dtb... to it
been doing this find for about a year, until 2023.04

also i just looked at the uboot-tools build log on koji and compared the results to my own locally built uboot

i explicitly set the BL31 path to where to find the binary and it's added to atf-bl31-path=

  ./tools/binman/binman   --toolpath ./tools  build -u -d u-boot.dtb -O . -m  -I . -I . -I ./board/rockchip/evb_rk3328 -I arch/arm/dts -a of-list="rk3328-rock64"  -a atf-bl31-path=ARM-software/arm-trusted-firmware/build/rk3328/release/bl31/bl31.elf -a tee-os-path= -a opensbi-path= -a default-dt="rk3328-rock64" -a scp-path= -a rockchip-tpl-path= -a spl-bss-pad= -a tpl-bss-pad= -a spl-dtb=y -a tpl-dtb= -a pre-load-key-path= 

but on koji builds, atf-bl31-path= is blank and some targets give a warning about it possibly being needed
make[1]: Entering directory '/builddir/build/BUILD/u-boot-2023.04-rc5/builds/rock64-rk3328'
  /builddir/build/BUILD/u-boot-2023.04-rc5/tools/binman/binman   --toolpath ./tools  build -u -d u-boot.dtb -O . -m --allow-missing --ignore-missing -I . -I /builddir/build/BUILD/u-boot-2023.04-rc5 -I /builddir/build/BUILD/u-boot-2023.04-rc5/board/rockchip/evb_rk3328 -I arch/arm/dts -a of-list="rk3328-rock64"  -a atf-bl31-path= -a tee-os-path= -a opensbi-path= -a default-dt="rk3328-rock64" -a scp-path= -a rockchip-tpl-path= -a spl-bss-pad= -a tpl-bss-pad= -a spl-dtb=y -a tpl-dtb= -a pre-load-key-path= 
make[1]: Leaving directory '/builddir/build/BUILD/u-boot-2023.04-rc5/builds/rock64-rk3328'
Image 'simple-bin' is missing external blobs and is non-functional: atf-bl31
/binman/simple-bin/fit/images/@atf-SEQ/atf-bl31:
   See the documentation for your board. You may need to build ARM Trusted
   Firmware and build with BL31=/path/to/bl31.bin
Some images are invalid

so something needs to get plugged on the build somewhere

Comment 10 Peter Robinson 2023-04-05 19:34:04 UTC
Can you test this scratch build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=99563382

Comment 11 bonin' o'brien 2023-04-05 20:23:24 UTC
i have to confess, i've never managed to successfully use rpmbuild once in my life, i've been compiling everything from the original git repos with the uefi-distro-load-FDT-from-any-partition-on-boot-device patch applied to uboot
but looking at the specfiles and tools/binman/binman.rst in uboot, i think all you need to do is wedge something like

export BL31=builds/$(echo $board)/bl31.elf

into the specfile, because otherwise, binman instead of giving a hard error will just produce a invalid binary with a warning

grep -B4 "Some images are invalid" https://kojipkgs.fedoraproject.org//packages/uboot-tools/2023.04/0.4.rc5.fc39/data/logs/aarch64/build.log

BL31 needs to be explictly set

Comment 12 Peter Robinson 2023-04-05 20:35:37 UTC
So are you actually testing the Fedora builds?

You don't have to use rpmbuild, you just need to install the rpms and test the binary builds.

For this one you'd just do: "rpm -Uvh https://kojipkgs.fedoraproject.org//work/tasks/3432/99563432/uboot-images-armv8-2023.04-1.fc39.noarch.rpm" and consume the binary.

Comment 13 David Sebek 2023-04-05 21:21:13 UTC
I tried a RockPro64 image from https://kojipkgs.fedoraproject.org//work/tasks/3432/99563432/uboot-images-armv8-2023.04-1.fc39.noarch.rpm, and it booted without any issues.

I see that the build log contains "Some images are invalid" message at multiple places, although I am not sure if it's an issue or not. This is the warning message:

Image 'u-boot-sunxi-with-spl' is missing external blobs and is non-functional: scp
/binman/u-boot-sunxi-with-spl/fit/images/scp/scp:
   SCP firmware is required for system suspend, but is otherwise optional.
   Please read the section on SCP firmware in board/sunxi/README.sunxi64
Some images are invalid

Comment 14 David Sebek 2023-04-05 21:24:35 UTC
Now I see in the build log that some images are missing BL31 too:

Image 'u-boot-sunxi-with-spl' is missing external blobs and is non-functional: atf-bl31 scp
/binman/u-boot-sunxi-with-spl/fit/images/atf/atf-bl31:
   Please read the section on ARM Trusted Firmware (ATF) in
   board/sunxi/README.sunxi64
/binman/u-boot-sunxi-with-spl/fit/images/scp/scp:
   SCP firmware is required for system suspend, but is otherwise optional.
   Please read the section on SCP firmware in board/sunxi/README.sunxi64
Some images are invalid

Comment 15 Peter Robinson 2023-04-05 21:29:30 UTC
> Image 'u-boot-sunxi-with-spl' is missing external blobs and is

Your bug report is about the Rock64, please confirm you are testing the Fedora builds with the Rock64. If you have other devices please test those. Please confirm actual real world testing.

Comment 16 Peter Robinson 2023-04-05 21:31:44 UTC
> BL31 needs to be explictly set

Most SoCs explictly set it in config files withing the build process.

Comment 17 Peter Robinson 2023-04-05 21:34:19 UTC
> Image 'u-boot-sunxi-with-spl' is missing external blobs and is
> non-functional: scp
> /binman/u-boot-sunxi-with-spl/fit/images/scp/scp:
>    SCP firmware is required for system suspend, but is otherwise optional.
>    Please read the section on SCP firmware in board/sunxi/README.sunxi64
> Some images are invalid

the SCP is optional and we've never shipped it in Fedora to date. That is not a regression. Please actually concentrate on the problem you are seeing.

Comment 18 David Sebek 2023-04-05 21:51:40 UTC
The updated package fixed the issue on my RockPro64 board.

bonin' o'brien can test if the issue is fixed on the Rock64 board too.

bonin' o'brien also mentioned the warning messages in the boot log. The missing SCP looked harmless to me, but I am not sure if the missing atf-bl31 is normal. The message is repeated multiple times in the build log.

Image 'u-boot-sunxi-with-spl' is missing external blobs and is non-functional: atf-bl31 scp
/binman/u-boot-sunxi-with-spl/fit/images/atf/atf-bl31:
   Please read the section on ARM Trusted Firmware (ATF) in
   board/sunxi/README.sunxi64
/binman/u-boot-sunxi-with-spl/fit/images/scp/scp:
   SCP firmware is required for system suspend, but is otherwise optional.
   Please read the section on SCP firmware in board/sunxi/README.sunxi64
Some images are invalid

Comment 19 bonin' o'brien 2023-04-05 23:00:58 UTC
(In reply to Peter Robinson from comment #12)
> So are you actually testing the Fedora builds?
i am!
> 
> You don't have to use rpmbuild, you just need to install the rpms and test
> the binary builds.
> 
> For this one you'd just do: "rpm -Uvh
> https://kojipkgs.fedoraproject.org//work/tasks/3432/99563432/uboot-images-
> armv8-2023.04-1.fc39.noarch.rpm" and consume the binary.

that one works!

Comment 20 bonin' o'brien 2023-04-06 00:54:42 UTC
just to defend myself, a diff between uart outputs of booting and non-booting binaries

-[ 2070.425254] reboot: Restarting system
+[ 7634.397815] reboot: Restarting system
 
-U-Boot TPL 2023.04-rc5 (Mar 28 2023 - 00:00:00)
+U-Boot TPL 2023.04 (Apr 04 2023 - 00:00:00)
 LPDDR3, 800MHz
@@ -148,8 +286,10 @@ Returning to boot ROM...
 
-U-Boot SPL 2023.04-rc5 (Mar 28 2023 - 00:00:00 +0000)
+U-Boot SPL 2023.04 (Apr 04 2023 - 00:00:00 +0000)
 Trying to boot from MMC1
-cannot find image node '': -1
+NOTICE:  BL31: v2.8(release):
+NOTICE:  BL31: Built : 00:00:00, Jan 18 2023
+NOTICE:  BL31:Rockchip release version: v1.2
 
 
-U-Boot 2023.04-rc5 (Mar 28 2023 - 00:00:00 +0000)
+U-Boot 2023.04 (Apr 04 2023 - 00:00:00 +0000)
 
@@ -157,3 +297,3 @@ Model: Pine64 Rock64
 DRAM:  1 GiB (effective 1022 MiB)
-PMIC:  RK8050 (on=0x10, off=0x04)
+PMIC:  RK8050 (on=0x50, off=0x01)
 Core:  228 devices, 24 uclasses, devicetree: separate
@@ -172,3 +312,3 @@ Found EFI removable media binary efi/boo
 Found DTB mmc 1:1 /rockchip/rk3328-rock64.dtb
-50894 bytes read in 9 ms (5.4 MiB/s)
+50886 bytes read in 8 ms (6.1 MiB/s)
 966711 bytes read in 49 ms (18.8 MiB/s)
@@ -180 +320,817 @@ Failed to persist EFI variables
 Booting /efi\boot\bootaa64.efi
+No EFI system partition
+Failed to persist EFI variables
+No EFI system partition
+Failed to persist EFI variables
+No EFI system partition
+Failed to persist EFI variables
+Speed: 100, full duplex
+GRUB version 2.06
+

Comment 21 Fedora Update System 2023-04-06 08:02:41 UTC
FEDORA-2023-dd6efcf8d3 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-dd6efcf8d3

Comment 22 Fedora Update System 2023-04-07 02:00:10 UTC
FEDORA-2023-dd6efcf8d3 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-dd6efcf8d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 23 Fedora Update System 2023-04-11 22:33:40 UTC
FEDORA-2023-dd6efcf8d3 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.