Bug 2269407 - osbuild-built ARM minimal disk image does not boot on Raspberry Pi 4b+
Summary: osbuild-built ARM minimal disk image does not boot on Raspberry Pi 4b+
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: osbuild
Version: 40
Hardware: aarch64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Image Builder team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F40BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2024-03-13 18:05 UTC by Brandon Nielsen
Modified: 2024-03-18 23:13 UTC (History)
21 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-03-18 23:13:40 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Brandon Nielsen 2024-03-13 18:05:09 UTC
Almost certainly the wrong component.

Fedora 40 Minimal Beta 1.3[0] does not boot on my Raspberry Pi 4b+. Green LED flashes 7 times in a repeating pattern. HDMI signal is seen by my monitor, but nothing is displayed.

The corresponding desktop image[1] does boot.

Image written with arm-image installer (`sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Minimal-40_Beta-1.3.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs --addkey ~/Desktop/pi.pub`).

[0] - https://kojipkgs.fedoraproject.org/compose/40/Fedora-40-20240313.0/compose/Spins/aarch64/images/Fedora-Minimal-40_Beta-1.3.aarch64.raw.xz
[1] - https://kojipkgs.fedoraproject.org/compose/40/Fedora-40-20240313.0/compose/Workstation/aarch64/images/Fedora-Workstation-40_Beta-1.3.aarch64.raw.xz

Reproducible: Always

Steps to Reproduce:
1. Write image to uSD card with arm-image-installer (`sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Minimal-40_Beta-1.3.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs --addkey ~/Desktop/pi.pub`)
2. Insert into Raspberry Pi 4b+
3. Power on
Actual Results:  
System does not boot. Green LED flashes 7 times in a repeating pattern. Monitor sees an HDMI signal, but displays nothing. No matter how long I wait, I can never SSH in.

Expected Results:  
The system to boot into a usable state.

Comment 1 Fedora Blocker Bugs Application 2024-03-13 18:08:38 UTC
Proposed as a Blocker for 40-beta by Fedora user nielsenb using the blocker tracking app because:

 Raspberry Pi 4 is a supported platform, minimal is a release blocking image, "release-blocking images must boot".

Comment 2 Adam Williamson 2024-03-13 20:01:59 UTC
Can you try the 1.2 image also, and 1.4 when it shows up? This may be related to building or not building the image with osbuild...

Comment 3 Brandon Nielsen 2024-03-13 20:51:38 UTC
(In reply to Adam Williamson from comment #2)
> Can you try the 1.2 image also, and 1.4 when it shows up? This may be
> related to building or not building the image with osbuild...

Exact same behavior (green LED flashing 7 times) observed with 1.2 and 1.4.

Comment 4 Adam Williamson 2024-03-13 21:45:16 UTC
So on further investigation, all the candidates were built with osbulld. It looks like the last image built with imgfac was the 0312.n.0 nightly... Sorry to keep asking for tests, but if we could find out whether that boots that'd be great.

Comment 5 Brandon Nielsen 2024-03-13 22:51:00 UTC
(In reply to Adam Williamson from comment #4)
> So on further investigation, all the candidates were built with osbulld. It
> looks like the last image built with imgfac was the 0312.n.0 nightly...
> Sorry to keep asking for tests, but if we could find out whether that boots
> that'd be great.

20240312.n.0 boots normally.

Comment 6 Adam Williamson 2024-03-13 23:17:30 UTC
OK, so it sounds like this is an osbuild issue somehow. We might need to revert arm minimal to building with imgfac and do yet another compose 😔

Comment 7 Adam Williamson 2024-03-14 17:23:29 UTC
It'd be interesting to know if this boots on any *other* hardware, I guess, or if there's kinda a fundamental issue booting SBCs on osbuild-built images...

Comment 8 Sherif Nagy 2024-03-14 17:52:16 UTC
I did a quick test and the behaviour discovered in the bug is correct, so I did a bit of more digging and seems like arm-image-installer doesn't install rpi-u-boot.bin actually , so I did fetch the RPM, extracted, copy the u-boot.bin from rpi_4 into the SDCARD boot and the image booted just fine!

To be honest, I can't recall if the u-boot.bin should be packed in the image at build time or it's arm-image-installer , I always thought it's arm-image-installer job.

TL;DR, rpi-u-boot.bin is missing from the image in the bug report, image boots when this file is copied to the Sdcard

Comment 9 Peter Robinson 2024-03-14 17:52:50 UTC
(In reply to Adam Williamson from comment #7)
> It'd be interesting to know if this boots on any *other* hardware, I guess,
> or if there's kinda a fundamental issue booting SBCs on osbuild-built
> images...

I have it booting on Rock5B, two imx8 devices, a couple of rk3399 devices.

Comment 10 Peter Robinson 2024-03-14 17:54:24 UTC
> To be honest, I can't recall if the u-boot.bin should be packed in the image
> at build time or it's arm-image-installer , I always thought it's
> arm-image-installer job.

We pack it in at build time so it just works.

> TL;DR, rpi-u-boot.bin is missing from the image in the bug report, image
> boots when this file is copied to the Sdcard

We added that functionally in osbuild, I wonder why it's not working but it's a great catch!

Comment 11 Peter Robinson 2024-03-14 17:56:27 UTC
Also I have been copying around rpi firmware manually a LOT because of the other issue so that would likely explain why I missed that oversight!

Comment 12 Adam Williamson 2024-03-14 17:58:59 UTC
yeah, awesome job Sherif! Thanks.

Comment 13 Sherif Nagy 2024-03-14 18:06:44 UTC
(In reply to Peter Robinson from comment #11)
> Also I have been copying around rpi firmware manually a LOT because of the
> other issue so that would likely explain why I missed that oversight!

That's fair also I thought now rpi_arm64 in uboot should be like "universal for rpi_[3-4]?

(In reply to Adam Williamson from comment #12)
> yeah, awesome job Sherif! Thanks.

Glad to help :)

Comment 14 Peter Robinson 2024-03-14 18:17:16 UTC
> That's fair also I thought now rpi_arm64 in uboot should be like "universal
> for rpi_[3-4]?

It is, we've shifted to that a few releases ago, it also supports rpi5 when we can eventually can support that.

Comment 15 Simon de Vlieger 2024-03-14 21:39:33 UTC
This is a bug in the image definitions for osbuild; I'll create an upstream PR. We do indeed copy the appropriate firmwares for IoT, but not for Minimal.

Upstream bug: https://github.com/osbuild/images/issues/520

Comment 16 Simon de Vlieger 2024-03-14 21:56:58 UTC
Upstream PR: https://github.com/osbuild/images/pull/521

Comment 17 Adam Williamson 2024-03-14 22:00:42 UTC
Thanks! Will we be able to get the fixed deployed to prod fairly rapidly? We'll need that to get a new F40 Beta candidate done promptly without having to fall back to imagefactory for this image.

Comment 18 Simon de Vlieger 2024-03-14 22:03:08 UTC
I'll try to get it onto production as soon as possible which should be within 24 hours, I'll keep this issue updated.

Comment 19 Peter Robinson 2024-03-15 13:20:24 UTC
This should be now fixes, I am running a scratch build here: https://koji.fedoraproject.org/koji/taskinfo?taskID=114989755

Comment 20 Simon de Vlieger 2024-03-15 13:47:37 UTC
That scratch build doesn't seem to have ran against correct versions (there is no u-boot being copied in the manifest so it's not in the disk image either) so I'm currently verifying if everything has fully deployed.

Comment 21 Simon de Vlieger 2024-03-15 15:40:52 UTC
I've ran a new build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2421346 this does copy the `rpi-u-boot.bin` file:

```
user@muja ~ € unxz Fedora-Minimal-40-20240315.n.3.aarch64.raw.xz
user@muja ~ € sudo losetup -fP Fedora-Minimal-40-20240315.n.3.aarch64.raw
user@muja ~ € sudo mount /dev/loop1p1 p1  # mount /boot
user@muja ~ € fd u-boot p1
p1/rpi-u-boot.bin
```

Comment 22 Adam Williamson 2024-03-15 16:11:45 UTC
Awesome! Thanks. If folks with Pis can verify whether that image and then the next Rawhide/F40 nightlies boot OK, that'd be great.

Comment 23 Brandon Nielsen 2024-03-15 17:17:20 UTC
(In reply to Simon de Vlieger from comment #21)
> I've ran a new build:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2421346 this does copy
> the `rpi-u-boot.bin` file:
> 
> ```
> user@muja ~ € unxz Fedora-Minimal-40-20240315.n.3.aarch64.raw.xz
> user@muja ~ € sudo losetup -fP Fedora-Minimal-40-20240315.n.3.aarch64.raw
> user@muja ~ € sudo mount /dev/loop1p1 p1  # mount /boot
> user@muja ~ € fd u-boot p1
> p1/rpi-u-boot.bin
> ```

I can confirm that build boots on my Pi 4b.

Comment 24 Adam Williamson 2024-03-16 06:56:58 UTC
Discussed at 2024-03-14 Fedora Beta Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/teams/f40-beta-go-no-go-meeting/f40-beta-go-no-go-meeting.2024-03-14-17.01.html . Accepted as a Beta blocker as a clear violation of "Release-blocking ARM disk images must boot to the initial-setup utility."

Comment 25 Adam Williamson 2024-03-18 23:13:40 UTC
OK, since the fix is already in production and I think we already confirmed Beta-1.7 boots on Pi 4, I think we can close this out. Please re-open if there's still anything outstanding here.


Note You need to log in before you can comment on or make changes to this bug.