Bug 2241252 - U-Boot fails to load kernel provided DTs from /boot (RPi4 boots to blank screen)
Summary: U-Boot fails to load kernel provided DTs from /boot (RPi4 boots to blank screen)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: uboot-tools
Version: 39
Hardware: aarch64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 2244237 (view as bug list)
Depends On:
Blocks: F39FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2023-09-28 20:21 UTC by Paul Whalen
Modified: 2023-11-01 06:08 UTC (History)
18 users (show)

Fixed In Version: uboot-tools-2023.07-3.fc39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-10-31 21:04:45 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
U-Boot test CoreOS video (4.08 MB, video/mp4)
2023-10-24 10:34 UTC, Hristo Marinov
no flags Details
CoreOS Fail to load from mSD (220.00 KB, image/png)
2023-10-26 03:02 UTC, Dusty Mabe
no flags Details

Description Paul Whalen 2023-09-28 20:21:42 UTC
When attempting to boot Fedora-Workstation-39_Beta-1.1.aarch64.raw image on the Raspberry Pi 4, system boots to a black screen. 

Adding 'nomodeset' to the kernel args, initial-setup is displayed as expected. 

Reproducible: Always

Steps to Reproduce:
Boot the RPi4 with uboot-tools-2023.10-0.4.rc3.fc39
Actual Results:  
Black screen

Expected Results:  
Initial-setup 

Downgrading to uboot-tools-2023.07-2.fc39 works as expected. 

It appears the dtb file might not be loaded when using uboot-tools-2023.10-0.4.rc3.fc39.

Comment 1 Peter Robinson 2023-09-28 21:03:38 UTC
Investigating, will need to bisect the upstream commits.

Comment 2 Fedora Blocker Bugs Application 2023-09-28 21:04:47 UTC
Proposed as a Blocker for 39-final by Fedora user pbrobinson using the blocker tracking app because:

 The RPi4 is a key aarch64 platform for workstation

Comment 3 Paul Whalen 2023-09-28 21:12:02 UTC
Created attachment 1990989 [details]
Fedora-Workstation-39_Beta-1.1.aarch64 journalctl

Comment 4 Adam Williamson 2023-09-28 21:14:38 UTC
Out of interest, how did we miss this for Beta release? Did nobody with an rpi test it?

Comment 5 Peter Robinson 2023-09-28 21:20:42 UTC
(In reply to Adam Williamson from comment #4)
> Out of interest, how did we miss this for Beta release? Did nobody with an
> rpi test it?

TBH I've not seen any QE testing on aarch64, I've had little time so I've been focused on core boot testing on Minimal and IoT and haven't done any desktop testing at all because just not enough cycles. pwhalen is no longer the arm QE person and those testing pieces were supposed to be taken over by others, I'm not sure who those others are. I suggest we add this to the release retrospective.

Comment 6 Adam Williamson 2023-09-28 21:47:45 UTC
sgallagh did some testing for Beta, that's right, I remember now. During the go/no-go meeting. He used a VM. So looks like we didn't do any HW tests. :/

Comment 7 Dusty Mabe 2023-09-29 01:44:04 UTC
I'd like to thank pwhalen for digging in here and opening the bug. For our FCOS test day [1] we were executing our Rpi4 test case [2] and were having trouble and after working around one issue [3] we then hit the VGA issue reported here.

[1] https://testdays.fedoraproject.org/events/159
[2] https://fedoraproject.org/wiki/QA:Testcase_CoreOS_Raspberry_Pi#aarch64
[3] https://github.com/coreos/fedora-coreos-docs/pull/590

Comment 8 Kamil Páral 2023-10-02 12:26:23 UTC
Accepted as F39 Final blocker in https://pagure.io/fedora-qa/blocker-review/issue/1352

Comment 9 Adam Williamson 2023-10-10 22:20:17 UTC
Might https://github.com/u-boot/u-boot/commit/833ff23047c50e4053fb1bda21f4e2c9f6a3aa6a help with this?

Comment 10 Adam Williamson 2023-10-10 22:21:59 UTC
specifically, I'm wondering if https://github.com/u-boot/u-boot/commit/6826c432e35115bbdafc17062733f5030b7f8366 might be the thing we need.

Comment 11 Peter Robinson 2023-10-11 09:36:48 UTC
(In reply to Adam Williamson from comment #10)
> specifically, I'm wondering if
> https://github.com/u-boot/u-boot/commit/
> 6826c432e35115bbdafc17062733f5030b7f8366 might be the thing we need.

Err, nope, RPi doesn't have a SPL (secondary program loader) or ATF at all due to it's "special" firmware so absolutely none of that PR relates to RPi at all (hence why it's not in that patch series at all).

Comment 12 Peter Robinson 2023-10-11 09:37:25 UTC
I am working through this, I thought I had the problem and hence solution but it's incomplete.

Comment 13 Peter Robinson 2023-10-18 20:59:09 UTC
*** Bug 2244237 has been marked as a duplicate of this bug. ***

Comment 14 Peter Robinson 2023-10-19 20:09:39 UTC
So we've basically got to the bottom of the problem and worked out the issue, I now just need to come up with a fix.

Comment 15 Adam Williamson 2023-10-19 20:48:35 UTC
🥳🥳

Comment 16 Peter Robinson 2023-10-21 09:27:15 UTC
So this bug also causes an issue with Jetson Nano (probably all older pre Xavier Jetsons). It's a conflict between upstream and Fedora's DTs being in /boot (2nd partition on a UEFI system).

https://pbrobinson.fedorapeople.org/rpi-u-boot/u-boot.bin

The above U-Boot build should boot and work on a RPi4 booting off the mSD card (sudo cp ~/u-boot.bin /boot/efi/rpi-u-boot.bin), it will only work with mSD as it's hard coded as the test for the logic.

I have most of the logic now in place to search for the DT correctly but U-Boot isn't running the script where/when I need it to and I need to ensure the logic is sound across mSD/eMMC/NVME/USB/SATA etc for the devices we support.

The reason I never saw it was that the fallback is that U-Boot falls back to use the U-Boot provided DT which for the devices I test regularly is the same as the Linux kernel so I wasn't seeing the issue.

Comment 17 Geoffrey Marr 2023-10-23 16:22:34 UTC
Thanks for this test uboot build pbrobinson. I have tested this using RPi4 on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz, Fedora-Workstation-39-20231016.n.0.aarch64.raw.xz, and Fedora-Workstation-39-20231023.n.0.aarch64.raw.xz, system now boots to CLI/GUI as it should in all three cases!

Comment 18 Hristo Marinov 2023-10-24 08:01:01 UTC
I can confirm system now boots to CLI on Fedora-Minimal-39-20231023.n.0.aarch64.raw.xz using RPi 4.

On CoreOS this particular build
Failed to load 'dtb/broadcom/bcn2711-rpi-4-b.dtb
ERROR: invalid device tree
U-Boot>

Comment 19 Peter Robinson 2023-10-24 08:33:40 UTC
(In reply to Hristo Marinov from comment #18)
> I can confirm system now boots to CLI on
> Fedora-Minimal-39-20231023.n.0.aarch64.raw.xz using RPi 4.
> 
> On CoreOS this particular build
> Failed to load 'dtb/broadcom/bcn2711-rpi-4-b.dtb
> ERROR: invalid device tree
> U-Boot>

In both cases can you confirm what U-Boot you were using, was it my test one, in the later case did you see get display output right to login?

Comment 20 Hristo Marinov 2023-10-24 10:34:33 UTC
Created attachment 1995273 [details]
U-Boot test CoreOS video

I downloaded and copied the file from the link in comment 16 again to Fedora-Minimal and CoreOS, just in case.
The video shows the CoreOS boot process with the u-boot.bin test build from comment 16.

Comment 21 Fedora Update System 2023-10-26 00:40:49 UTC
FEDORA-2023-7994524575 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

Comment 22 Dusty Mabe 2023-10-26 03:01:54 UTC
With the rpms from https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575 :

/tmp/RPi4boot/bcm283x-firmware-20231017-1.ce3a0b4.fc39.aarch64.rpm
/tmp/RPi4boot/bcm283x-overlays-20231017-1.ce3a0b4.fc39.aarch64.rpm
/tmp/RPi4boot/uboot-images-armv8-2023.07-3.fc39.noarch.rpm

I'm seeing a different issue for my RPi4 after following the CoreOS install steps with the updated copying of /usr/share/uboot/rpi_arm64/u-boot.bin into /boot/efi/rpi-u-boot.bin from https://github.com/coreos/fedora-coreos-docs/pull/590

The issue I'm seeing now is the Pi4 won't boot. Seems like it is trying to boot from USB and not from SD card? See attachment.

Note that these RPMs don't ship in Fedora CoreOS so if this problem ends up somehow being Fedora CoreOS specific I wouldn't consider it a blocker for the rest of F39 GA as it can be fixed later and shipped via an update.

Comment 23 Dusty Mabe 2023-10-26 03:02:58 UTC
Created attachment 1995504 [details]
CoreOS Fail to load from mSD

Comment 24 Adam Williamson 2023-10-26 15:37:16 UTC
can somebody please test with https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231026.0/ and see if this is fixed?

Comment 25 Paul Whalen 2023-10-26 16:15:55 UTC
Fedora-Workstation-39-1.2.aarch64.raw.xz boots with display on the RPi4-4GB

Comment 26 Brandon Nielsen 2023-10-26 16:21:57 UTC
(In reply to Adam Williamson from comment #24)
> can somebody please test with
> https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231026.0/ and see
> if this is fixed?

I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B.

Comment 27 Adam Williamson 2023-10-26 16:27:56 UTC
Brandon: did you use arm-image-installer to write it? Do the other variants work?

Comment 28 Geoffrey Marr 2023-10-26 17:01:35 UTC
Can confirm, Fedora-Workstation-39-1.2.aarch64.raw.xz boots with display on the RPi400-4GB.

Comment 29 Peter Robinson 2023-10-26 17:04:03 UTC
> I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B.

So server is a little special here in that the firmware can't load the kernel DT off /boot (due to XFS) but it should still use the RPi DT, do you have any modification in the config.txt?

Comment 30 Brandon Nielsen 2023-10-26 17:05:27 UTC
(In reply to Adam Williamson from comment #27)
> Brandon: did you use arm-image-installer to write it? Do the other variants
> work?

sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs --addkey ~/.ssh/pi/id_ed25519.pub

Just tried workstation, it boots to gnome-initial-setup, but crashes back to GDM every attempt at signing in. Invocation was:

sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Workstation-39-1.2.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs

Comment 31 Brandon Nielsen 2023-10-26 17:07:54 UTC
(In reply to Peter Robinson from comment #29)
> > I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B.
> 
> So server is a little special here in that the firmware can't load the
> kernel DT off /boot (due to XFS) but it should still use the RPi DT, do you
> have any modification in the config.txt?

There shouldn't be any, especially assuming it gets rewritten every arm-image-installer use.

Comment 32 Brandon Nielsen 2023-10-26 17:39:24 UTC
(In reply to Brandon Nielsen from comment #30)
> (In reply to Adam Williamson from comment #27)
> > Brandon: did you use arm-image-installer to write it? Do the other variants
> > work?
> 
> sudo arm-image-installer
> --image=/home/nielsenb/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz
> --target=rpi4 --media=/dev/sda --resizefs --addkey ~/.ssh/pi/id_ed25519.pub
> 
> Just tried workstation, it boots to gnome-initial-setup, but crashes back to
> GDM every attempt at signing in. Invocation was:
> 
> sudo arm-image-installer
> --image=/home/nielsenb/Desktop/Fedora-Workstation-39-1.2.aarch64.raw.xz
> --target=rpi4 --media=/dev/sda --resizefs

In regards to Workstation, every time it crashes back to GDM, the only interesting thing I see with dmesg is:

[  369.533317] hdmi-audio-codec hdmi-audio-codec.1.auto: ASoC: error at snd_soc_dai_startup on i2s-hifi: -19
[  369.533351]  MAI: ASoC: error at __soc_pcm_open on MAI: -19

Comment 33 Brandon Nielsen 2023-10-26 18:25:26 UTC
(In reply to Brandon Nielsen from comment #32)
> 
> In regards to Workstation, every time it crashes back to GDM, the only
> interesting thing I see with dmesg is:
> 

nomodeset also fixes the crash back to GDM.

Also note I was in error, this is testing with a 4 GB Pi 4B Rev 1.1.

Comment 34 Brandon Nielsen 2023-10-26 20:35:38 UTC
Writing server with dd (sudo xzcat ~/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz | sudo dd of=/dev/sda bs=1M oflag=direct status=progress), results in an image that behaves as expected (successfully boots, but to a black screen without `nomodeset`, successfully boots to initial setup with `nomodeset`.

I have opened another bug for the GDM issue: https://bugzilla.redhat.com/show_bug.cgi?id=2246442

Comment 35 Adam Williamson 2023-10-26 21:06:50 UTC
so in general, I think we can consider the actual originally reported bug here fixed. We can follow up on Brandon's bug in https://bugzilla.redhat.com/show_bug.cgi?id=2246442 and the Server bug discovered by Hristo in https://bugzilla.redhat.com/show_bug.cgi?id=2246428 . Marking this as VERIFIED.

Comment 36 Sally 2023-10-26 22:05:57 UTC
I have tested "Fedora-Minimal-39-1.2.aarch64.raw.xz" on mSD card, used "arm-image-installer" to flash it, the result is that, It's booting as expected (working), and I've tested "Fedora-Workstation-39-1.2.aarch64.raw.xz" as well, but on SSD, used "arm-image-installer, it's booting just fine.
Thank you.

Comment 37 Fedora Update System 2023-10-27 02:14:26 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 38 Dusty Mabe 2023-10-27 21:58:02 UTC
update on https://bugzilla.redhat.com/show_bug.cgi?id=2241252#c22

This was user error. I was trying to test out the packages before they were in any repos and I ended up missing bcm2711-firmware and bcm2835-firmware when grabbing them from koji.

Comment 39 Peter Boy 2023-10-29 18:54:38 UTC
For the sake of completeness:

With RC 1.2, using
arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset"  --media=/dev/mmcblk0

resolves the issue for the Raspi 4. So that's an easy workaround and can be covered in our Server documentation.

I have not been able to test this on any of my other devices (which are more important for Server), because they didn't boot at all.  
So it's not sure whether this will work on other device models (though it probably will).

Comment 40 Peter Robinson 2023-10-29 19:03:47 UTC
(In reply to Peter Boy from comment #39)
> For the sake of completeness:
> 
> With RC 1.2, using
> arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args
> "nomodeset"  --media=/dev/mmcblk0
> 
> resolves the issue for the Raspi 4. So that's an easy workaround and can be
> covered in our Server documentation.

Adding "nomodeset" doesn't "resolve" the issue, it works around the issue.
 
> I have not been able to test this on any of my other devices (which are more
> important for Server), because they didn't boot at all.  
> So it's not sure whether this will work on other device models (though it
> probably will).

Can you file explicit bugs for each of the problems with the "other devices", each of these problems will a independent and different and we can't work to fix them without that, I very much doubt it will work on "other device models" and if it does it's by chance. It's really not that simple.

Comment 41 Peter Boy 2023-10-29 20:40:43 UTC
> > resolves the issue for the Raspi 4. So that's an easy workaround and can be
> > covered in our Server documentation.
>
> Adding "nomodeset" doesn't "resolve" the issue, it works around the issue.

Yes, of course. My internal translator seems to be a bit "rough" at times. What I wanted to express is that I and very likely my server WG colleagues are happy with this 'workaround solution'.  :-)   


> Can you file explicit bugs for each of the problems
https://bugzilla.redhat.com/show_bug.cgi?id=2246871

Comment 42 Fedora Update System 2023-10-30 01:46:52 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 43 Fedora Update System 2023-10-31 21:04:45 UTC
FEDORA-2023-7994524575 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 44 Sally 2023-11-01 00:40:29 UTC
I tested "Fedora-Workstation-39-1.4.aarch64.raw.xz" and "Fedora-Minimal-39-1.4.aarch64.raw.xz" with arm-image-installer-3.9, both of them boot successfully.

Note: At the end of arm-image-installer process, it showed:
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
./arm-image-installer-arm-image-installer-3.9/arm-image-installer: 482: btrfs: not found
= Raspberry Pi 4 Uboot is already in place, no changes needed.
./arm-image-installer-arm-image-installer-3.9/arm-image-installer: 621: getenforce: not found

= Installation Complete! Insert into the rpi4 and boot

* I have used the extracted zip file of arm-image-installer on Debian, I wasn't in the same directory of extracted folder.

Comment 45 Brandon Nielsen 2023-11-01 01:46:01 UTC
I still need --args "nomodeset" passed to arm-image-installer with the FEDORA-2023-7994524575 updates. Without it, system boots to a black screen. Tested Fedora-Server-39-1.4.aarch64.raw.xz on a 4 GB Pi 4B Rev 1.1.

Comment 46 Adam Williamson 2023-11-01 06:08:57 UTC
That's a different issue: https://bugzilla.redhat.com/show_bug.cgi?id=2246428


Note You need to log in before you can comment on or make changes to this bug.