Bug 1886187

Summary: bcm283x-firmware-20200928-2.f0eab3a.fc33.aarch64 fails to boot on Raspberry Pi 3 Model B
Product: [Fedora] Fedora Reporter: Joern Wiesner <joern.wiesner>
Component: bcm283x-firmwareAssignee: Peter Robinson <pbrobinson>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 33CC: gmarr, laura, pbrobinson, pwhalen, robatino
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: bcm283x-firmware-20201008-2.63b1922.fc33 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-14 00:42:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1766777    
Attachments:
Description Flags
/boot/efi/config.txt none

Description Joern Wiesner 2020-10-07 20:37:36 UTC
Description of problem:

After upgrading to bcm283x-firmware-20200928-2.f0eab3a.fc33.aarch64 my Raspberry Pi 3 Model B fails to boot. Nothing is displayed via serial console or HDMI.

Version-Release number of selected component (if applicable):

bcm283x-firmware-20200928-2.f0eab3a.fc33.aarch64

How reproducible:

100%

Steps to Reproduce:

1. Upgrade to bcm283x-firmware-20200928-2.f0eab3a.fc33.aarch64
2. Reboot system.
3. Observe boot process

Actual results:

System fails to boot, no display output observed.

Expected results:

Boot the system successfully.

Additional info:

Works without problems on Raspberry Pi 4 Model B with 4GB RAM.

Comment 1 Peter Robinson 2020-10-07 20:42:24 UTC
Which generation of 3B? Which HW rev. I tested it on mine and it booted fine. Please provide more details, what image etc. Please also attach your config.txt, it it default or have you modified it?

Comment 2 Joern Wiesner 2020-10-07 21:06:20 UTC
Created attachment 1719853 [details]
/boot/efi/config.txt

Comment 3 Joern Wiesner 2020-10-07 21:12:29 UTC
Hardware is a Raspberry Pi 3 Model B V1.2. 

Used Fedora-Minimal-33_Beta-1.3.aarch64.raw.xz on a 32GB SD card. After several tries I ended up with applying all updates via 'sudo dnf -y update --exclude=bcm283x-firmware'. After reboot only bcm283x-firmware-20200928-2.f0eab3a.fc33.aarch64 was missing with bcm283x-firmware-20200401-1.c2c6ce8.fc33.aarch64 still installed. After running 'sudo dnf -y update' I got:

Upgrading:
 bcm283x-firmware   aarch64   20200928-2.f0eab3a.fc33   updates-testing    49 k
Installing dependencies:
 bcm2711-firmware   aarch64   20200928-2.f0eab3a.fc33   updates-testing   2.8 M
 bcm2835-firmware   aarch64   20200928-2.f0eab3a.fc33   updates-testing   3.9 M
 bcm283x-overlays   aarch64   20200928-2.f0eab3a.fc33   updates-testing    99 k

The updates applied without a problem. After reboot no output via HDMI or serial console.

Comment 4 Joern Wiesner 2020-10-07 21:14:33 UTC
/boot/efi/config.txt is the stock version without any modification:

-rwx------. 1 root root    1817 Apr  6  2020 config.txt

Comment 5 Peter Robinson 2020-10-10 17:23:16 UTC
OK, I think the Oct 08th firmware fixes this. Booted across ARMv7/aarch64 across most devices. ARMv7: rpi2B/3B/3B+/4-4gb/4-8gb, aarch64: 3B/3B+/4-8gb. I'm not sure where my 3A+ is ATM to test that.

RPi2B ARMv7:
09-29:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 2 Model B
[    8.512447] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-09-28T12:55:29

10-08:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 2 Model B
[    9.940289] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:55:24

RPi3B ARMv7:
09-29: no boot

10-08:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B
[    8.302267] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:55:24


RPi3B+ ARMv7:
09-29:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B+
[    6.873777] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-09-28T12:55:29

10-08:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B+
[    8.302399] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:55:24

RPi4:
09-29: no test

10-08:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B
[    6.115628] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:53:53

RPi4-8Gb:
09-29: no test

10-08:
[    0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B
[    6.110737] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:53:53


RPi3B aarch64:
04-01:

10-08:
[    0.000000] Machine model: Raspberry Pi 3 Model B
[    0.352079] DMI: raspberrypi rpi/rpi, BIOS 2020.10-rc1 08/10/2020
[    7.917871] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:55:24

RPi3B+ aarch64:
04-01:
[    0.000000] Machine model: Raspberry Pi 3 Model B+
[    0.352206] DMI: raspberrypi rpi/rpi, BIOS 2020.10-rc1 08/10/2020
[    7.969320] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-04-01T16:31:58

10-08:
[    0.000000] Machine model: Raspberry Pi 3 Model B+
[    0.352122] DMI: raspberrypi rpi/rpi, BIOS 2020.10-rc1 08/10/2020
[    7.921882] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:55:24

RPi4-8Gb aarch64:
10-08:
[    0.000000] Machine model: Raspberry Pi 4 Model B
[    0.371386] DMI: raspberrypi rpi/rpi, BIOS 2020.10-rc1 08/10/2020
[    6.354025] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-10-08T16:53:53

Comment 6 Fedora Update System 2020-10-10 17:39:25 UTC
FEDORA-2020-884692ec02 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-884692ec02

Comment 7 Peter Robinson 2020-10-10 17:40:53 UTC
Joern: Can you test this new build and let me know how you get on with it.

Comment 8 Fedora Update System 2020-10-11 22:31:24 UTC
FEDORA-2020-884692ec02 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-884692ec02`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-884692ec02

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Blocker Bugs Application 2020-10-12 07:01:14 UTC
Proposed as a Blocker for 33-final by Fedora user pbrobinson using the blocker tracking app because:

 Raspberry Pi is a common platform and there was some issues with the new firmware that stopped it booting on some generations/configurations of the device. The new firmware fixes this. Because it's a boot/start issue it needs to be in the arm images so these devices will work.

Comment 10 Joern Wiesner 2020-10-12 14:23:21 UTC
(In reply to Peter Robinson from comment #7)
> Joern: Can you test this new build and let me know how you get on with it.
Hi Peter,

Gave it another shot today without luck. Did burn a new SD card for each trial and run:

sudo dnf -y update --exclude=bcm283x-firmware
sudo reboot
sudo dnf -y upgrade

Upgraded:
  bcm283x-firmware-20201008-1.63b1922.fc33.aarch64

Installed:
  bcm2711-firmware-20201008-1.63b1922.fc33.aarch64
  bcm2835-firmware-20201008-1.63b1922.fc33.aarch64
  bcm283x-overlays-20201008-1.63b1922.fc33.aarch64

Complete!

sudo reboot

Models in use:

Raspberry Pi 3 Model A+
Raspberry Pi 3 Model B V1.2
Raspberry Pi 3 Model B+
Raspberry Pi 4 Model B, 4GB

Works only on the RPi4, not output via serial console or HDMI on the RPi3 models.

What did I do wrong? How can I further debug the issue?

BR, Joern.

Comment 11 Geoffrey Marr 2020-10-12 16:54:49 UTC
Wrote Fedora-Minimal-33_Beta-1.3.aarch64.raw.xz to a 32gb MicroSD card with arm-image-installer: "sudo arm-image-installer --image=Fedora-Minimal-33_Beta-1.3.aarch64.raw.xz --target=rpi3 --media=/dev/sdd". No changes to config.txt. Ran through initial-setup, then booted into system. Rebooted to make sure the system came back up. Upon reboot (which worked), installed new firmware: "sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-884692ec02". Installed fine, rebooted system. System does not output via HDMI upon reboot, hangs at black screen. Waited for 5 minutes and attempted to access virtual terminals, no luck.

Comment 12 Geoffrey Marr 2020-10-12 17:12:29 UTC
My above testing was performed with a RPi 3B+.

Comment 13 Fedora Update System 2020-10-12 18:59:44 UTC
FEDORA-2020-884692ec02 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-884692ec02

Comment 14 Geoffrey Marr 2020-10-12 19:27:14 UTC
Discussed during the 2020-10-05 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedBlocker (Final)" was made as it violates the following criterion:

"All release-blocking images must boot in their supported configurations"

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-10-12/f33-blocker-review.2020-10-12-16.00.txt

Comment 15 Geoffrey Marr 2020-10-12 19:52:26 UTC
(In reply to Fedora Update System from comment #13)
> FEDORA-2020-884692ec02 has been submitted as an update to Fedora 33.
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-884692ec02

Installed build bcm283x-firmware-20201008-2.63b1922.fc33, tested, works for me! System boots and is usable. (RPi3B+)

Comment 16 Joern Wiesner 2020-10-12 22:05:06 UTC
Installed on RPi3A+, RPi3B and RPi3B+. All systems survived a reboot! Great, many thanks!

Comment 17 Fedora Update System 2020-10-12 22:45:37 UTC
FEDORA-2020-884692ec02 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-884692ec02`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-884692ec02

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2020-10-14 00:42:15 UTC
FEDORA-2020-884692ec02 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.