Bug 2269407

Proposed as a Blocker for 40-beta by Fedora user nielsenb using the blocker tracking app because:

 Raspberry Pi 4 is a supported platform, minimal is a release blocking image, "release-blocking images must boot".

Can you try the 1.2 image also, and 1.4 when it shows up? This may be related to building or not building the image with osbuild...

(In reply to Adam Williamson from comment #2)
> Can you try the 1.2 image also, and 1.4 when it shows up? This may be
> related to building or not building the image with osbuild...

Exact same behavior (green LED flashing 7 times) observed with 1.2 and 1.4.

So on further investigation, all the candidates were built with osbulld. It looks like the last image built with imgfac was the 0312.n.0 nightly... Sorry to keep asking for tests, but if we could find out whether that boots that'd be great.

(In reply to Adam Williamson from comment #4)
> So on further investigation, all the candidates were built with osbulld. It
> looks like the last image built with imgfac was the 0312.n.0 nightly...
> Sorry to keep asking for tests, but if we could find out whether that boots
> that'd be great.

20240312.n.0 boots normally.

OK, so it sounds like this is an osbuild issue somehow. We might need to revert arm minimal to building with imgfac and do yet another compose 😔

It'd be interesting to know if this boots on any *other* hardware, I guess, or if there's kinda a fundamental issue booting SBCs on osbuild-built images...

I did a quick test and the behaviour discovered in the bug is correct, so I did a bit of more digging and seems like arm-image-installer doesn't install rpi-u-boot.bin actually , so I did fetch the RPM, extracted, copy the u-boot.bin from rpi_4 into the SDCARD boot and the image booted just fine!

To be honest, I can't recall if the u-boot.bin should be packed in the image at build time or it's arm-image-installer , I always thought it's arm-image-installer job.

TL;DR, rpi-u-boot.bin is missing from the image in the bug report, image boots when this file is copied to the Sdcard

(In reply to Adam Williamson from comment #7)
> It'd be interesting to know if this boots on any *other* hardware, I guess,
> or if there's kinda a fundamental issue booting SBCs on osbuild-built
> images...

I have it booting on Rock5B, two imx8 devices, a couple of rk3399 devices.

> To be honest, I can't recall if the u-boot.bin should be packed in the image
> at build time or it's arm-image-installer , I always thought it's
> arm-image-installer job.

We pack it in at build time so it just works.

> TL;DR, rpi-u-boot.bin is missing from the image in the bug report, image
> boots when this file is copied to the Sdcard

We added that functionally in osbuild, I wonder why it's not working but it's a great catch!

Also I have been copying around rpi firmware manually a LOT because of the other issue so that would likely explain why I missed that oversight!

yeah, awesome job Sherif! Thanks.

(In reply to Peter Robinson from comment #11)
> Also I have been copying around rpi firmware manually a LOT because of the
> other issue so that would likely explain why I missed that oversight!

That's fair also I thought now rpi_arm64 in uboot should be like "universal for rpi_[3-4]?

(In reply to Adam Williamson from comment #12)
> yeah, awesome job Sherif! Thanks.

Glad to help :)

> That's fair also I thought now rpi_arm64 in uboot should be like "universal
> for rpi_[3-4]?

It is, we've shifted to that a few releases ago, it also supports rpi5 when we can eventually can support that.

This is a bug in the image definitions for osbuild; I'll create an upstream PR. We do indeed copy the appropriate firmwares for IoT, but not for Minimal.

Upstream bug: https://github.com/osbuild/images/issues/520

Upstream PR: https://github.com/osbuild/images/pull/521

Thanks! Will we be able to get the fixed deployed to prod fairly rapidly? We'll need that to get a new F40 Beta candidate done promptly without having to fall back to imagefactory for this image.

I'll try to get it onto production as soon as possible which should be within 24 hours, I'll keep this issue updated.

This should be now fixes, I am running a scratch build here: https://koji.fedoraproject.org/koji/taskinfo?taskID=114989755

That scratch build doesn't seem to have ran against correct versions (there is no u-boot being copied in the manifest so it's not in the disk image either) so I'm currently verifying if everything has fully deployed.

I've ran a new build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2421346 this does copy the `rpi-u-boot.bin` file:

```
user@muja ~ € unxz Fedora-Minimal-40-20240315.n.3.aarch64.raw.xz
user@muja ~ € sudo losetup -fP Fedora-Minimal-40-20240315.n.3.aarch64.raw
user@muja ~ € sudo mount /dev/loop1p1 p1  # mount /boot
user@muja ~ € fd u-boot p1
p1/rpi-u-boot.bin
```

Awesome! Thanks. If folks with Pis can verify whether that image and then the next Rawhide/F40 nightlies boot OK, that'd be great.

(In reply to Simon de Vlieger from comment #21)
> I've ran a new build:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2421346 this does copy
> the `rpi-u-boot.bin` file:
> 
> ```
> user@muja ~ € unxz Fedora-Minimal-40-20240315.n.3.aarch64.raw.xz
> user@muja ~ € sudo losetup -fP Fedora-Minimal-40-20240315.n.3.aarch64.raw
> user@muja ~ € sudo mount /dev/loop1p1 p1  # mount /boot
> user@muja ~ € fd u-boot p1
> p1/rpi-u-boot.bin
> ```

I can confirm that build boots on my Pi 4b.

Discussed at 2024-03-14 Fedora Beta Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/teams/f40-beta-go-no-go-meeting/f40-beta-go-no-go-meeting.2024-03-14-17.01.html . Accepted as a Beta blocker as a clear violation of "Release-blocking ARM disk images must boot to the initial-setup utility."

OK, since the fix is already in production and I think we already confirmed Beta-1.7 boots on Pi 4, I think we can close this out. Please re-open if there's still anything outstanding here.