Bug 2231599

Summary: Upgrade to Fedora 38 failed on a Raspberry Pi 3 Model B
Product: [Fedora] Fedora Reporter: Adrian Torregrosa <adrian.torregrosa>
Component: bcm283x-firmwareAssignee: Peter Robinson <pbrobinson>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: laura, pbrobinson, pwhalen
Target Milestone: ---Keywords: Upgrades
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-21 12:32:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Adrian Torregrosa 2023-08-12 10:06:58 UTC
My Raspberry PI Model B was running Fedora 37. I had installed it with Fedora 31 and upgraded it w/o major problems up to Fedora 37.

Recently I attempted to upgrade to Fedora 38. The dnf command went through with no errors at all. Then I rebooted, but the Raspberry Pi failed to start: nothing was displayed on the screen and the green LED entered a cycle of blinking seven times, pausing a couple of seconds or so, and then blinking again. I tried the upgrade twice.

This morning I retried the upgrade but excluding the bcm283x-firmware related rpms, adding the following statement to the dnf command:
-x bcm2711-firmware,bcm2835-firmware,bcm283x-firmware,bcm283x-overlays
This way the upgrade succeeded. The excluded rpms are:
bcm2711-firmware-20230728-1.e03ae3c.fc38.aarch64.rpm
bcm2835-firmware-20230728-1.e03ae3c.fc38.aarch64.rpm
bcm283x-firmware-20230728-1.e03ae3c.fc38.aarch64.rpm
bcm283x-overlays-20230728-1.e03ae3c.fc38.aarch64.rpm

For some reason one of these products caused the Raspberry Pi not to be able to boot. As the system was completely unresponsive and the screen was black I cannot add much more detail. Below you can see the current situation regarding the kernel and the firmware packages:
[root@rpi3b ~]# rpm -qa | grep -e ^kernel -e ^bcm2 | sort
bcm2711-firmware-20220826-1.ecc243c.fc37.aarch64
bcm2835-firmware-20220826-1.ecc243c.fc37.aarch64
bcm283x-firmware-20220826-1.ecc243c.fc37.aarch64
bcm283x-overlays-20220826-1.ecc243c.fc37.aarch64
kernel-6.4.7-200.fc38.aarch64
kernel-core-6.4.7-200.fc38.aarch64
kernel-devel-6.4.7-200.fc38.aarch64
kernel-headers-6.4.4-200.fc38.aarch64
kernel-modules-6.4.7-200.fc38.aarch64
kernel-modules-core-6.4.7-200.fc38.aarch64
kernel-srpm-macros-1.0-19.fc38.noarch
[root@rpi3b ~]#


Reproducible: Always

Steps to Reproduce:
1. Install Fedora 37 Minimal on a Raspberry Pi Model B
2. run dnf -y --releasever 38 upgrade

Actual Results:  
The Raspberry Pi failed to start.

Expected Results:  
The Raspberry Pi should be able to start.

The Raspberry Pi functions properly with the 20220826 version of the firmware packages.

Comment 1 Adrian Torregrosa 2023-08-12 11:35:59 UTC
I found earlier versions of the bcm283x-firmware here: https://koji.fedoraproject.org/koji/packageinfo?packageID=20547 so I went on trying them one after another. Here's the outcome:

20220826: ok
20230118: ok
20230308: NOK

Comment 2 Peter Robinson 2023-08-17 12:58:08 UTC
Have you updated U-Boot during this upgrade cycle? Can you post the output of "dmesg |grep DMI"?
Also have you modified the /boot/efi/config.txt at all? You may have a /boot/efi/config.txt.rpmnew file where you have to do some manual syncs.

Comment 3 Adrian Torregrosa 2023-08-20 08:18:55 UTC
I did not update U-boot, basically because I did not know how to. dmesg showed the following
[    0.040348] DMI: raspberrypi rpi/rpi, BIOS 2020.04 04/20/2020
which most likely came with Fedora-Minimal-31-1.9.aarch64.raw.xz. I had not modified the config.txt file.

Following your hint I noticed that the config.txt from bcm283x-firmware-20230728 is rather different than that of version 20230118, the new version has no pi02 nor pi3 sections altogether. And the only "kernel" line states
kernel=rpi-u-boot.bin

So I
- upgraded the bcm* rpms to their official 20230728 version
- extracted ./usr/share/uboot/rpi_3/u-boot.bin from uboot-images-armv8-2023.04-2.fc38 to /boot/efi
- renamed it to rpi-u-boot.bin

This way the Raspberry Pi boots correctly and the dmesg states:
[    0.040681] DMI: raspberrypi,3-model-b Raspberry Pi 3 Model B Rev 1.2/Raspberry Pi 3 Model B Rev 1.2, BIOS 2023.04 04/01/2023

I'm not sure whether this is a bug or not, or maybe the change in the config.txt file ought to be documented (or maybe it is already and I couldn't find it).

Comment 4 Peter Robinson 2023-08-21 12:32:20 UTC
(In reply to Adrian Torregrosa from comment #3)
> I did not update U-boot, basically because I did not know how to. dmesg
> showed the following
> [    0.040348] DMI: raspberrypi rpi/rpi, BIOS 2020.04 04/20/2020
> which most likely came with Fedora-Minimal-31-1.9.aarch64.raw.xz. I had not
> modified the config.txt file.

This "BIOS 2020.04 04/20/2020" shows you were running quite an old U-Boot. There were some weird bugs that required newer U-Boots.

> Following your hint I noticed that the config.txt from
> bcm283x-firmware-20230728 is rather different than that of version 20230118,
> the new version has no pi02 nor pi3 sections altogether. And the only
> "kernel" line states
> kernel=rpi-u-boot.bin

Correct, we moved to to a single unified u-boot that works for all aarch64 capable variants of the rpi which simplified things a lot.

> So I
> - upgraded the bcm* rpms to their official 20230728 version
> - extracted ./usr/share/uboot/rpi_3/u-boot.bin from
> uboot-images-armv8-2023.04-2.fc38 to /boot/efi
> - renamed it to rpi-u-boot.bin

If you run rpi-uboot-update it will do all of that for you.

> This way the Raspberry Pi boots correctly and the dmesg states:
> [    0.040681] DMI: raspberrypi,3-model-b Raspberry Pi 3 Model B Rev
> 1.2/Raspberry Pi 3 Model B Rev 1.2, BIOS 2023.04 04/01/2023
> 
> I'm not sure whether this is a bug or not, or maybe the change in the
> config.txt file ought to be documented (or maybe it is already and I
> couldn't find it).

The firmware on rpi is quite complex and there are upgrade paths that are run to try an mitigate the changes, but in some cases if you jump from very old to new things are sometimes missed. We have tools like rpi-uboot-update to mitigate and clean up some of these things but it also doesn't always get everything as well.