This was discovered by Hristo Marinov and confirmed by Paul Whalen in testing of https://bugzilla.redhat.com/show_bug.cgi?id=2241252 and https://bugzilla.redhat.com/show_bug.cgi?id=2244305 . Per pwhalen, it seems there's a reproducible and understandable problem specific to the Server images even after the rollback to uboot-tools 2023.07. pwhalen says "Its a bug in the FW provided dtb it seems. But it also seems to only affect some revisions" As I understand it, the Server images (only) use the FW provided dtb because the kernel dtb cannot be read from xfs. Other images use the kernel dtb because they don't use xfs. That's why this bug is specific to Server images. The bug can be worked around by booting with nomodeset. Proposing as a Final blocker as a conditional violation of "Release-blocking ARM disk images must boot to the initial-setup utility" - when booting a Server image on an affected Pi variant and expecting to see i-s on an attached monitor.
Discussed at 2023-10-26 go/no-go meeting acting as a blocker review meeting - https://meetbot-raw.fedoraproject.org/fedora-meeting/2023-10-26/ . Rejected as a blocker on the basis that physical monitor display is less important on Server than e.g. Workstation (many folks use it headless), this only affects some hardware and/or firmware versions, and there is a documentable workaround (using nomodeset).
This issue reproduces here. HDMI device probing fails with "vc4_hdmi fef00700.hdmi: Could not register PCM component: -517" which is caused by the different device tree bindings used for the HDMI audio dma engine: Kernel DTB | dma@7e007000 { | compatible = "brcm,bcm2835-dma"; | reg = <0x7e007000 0xb00>; | interrupts = <0x00 0x50 0x04 0x00 0x51 0x04 0x00 0x52 0x04 0x00 0x53 0x04 0x00 0x54 0x04 0x00 0x55 0x04 0x00 0x56 0x04 0x00 0x57 0x04 0x00 0x57 0x04 0x00 0x58 0x04 0x00 0x58 0x04>; | interrupt-names = "dma0\0dma1\0dma2\0dma3\0dma4\0dma5\0dma6\0dma7\0dma8\0dma9\0dma10"; | #dma-cells = <0x01>; | brcm,dma-channel-mask = <0x7f5>; | phandle = <0x1b>; | }; firmware DTB used by the server image: | dma@7e007b00 { | compatible = "brcm,bcm2711-dma";; | reg = <0x00 0x7e007b00 0x00 0x400>; | interrupts = <0x00 0x59 0x04 0x00 0x5a 0x04 0x00 0x5b 0x04 0x00 0x5c 0x04>; | brcm,dma-channel-mask = <0x3000>; | interrupt-names = "dma11\0dma12\0dma13\0dma14" | #dma-cells = <0x01>; | phandle = <0x2e>; | }; This could be worked around with a couple of line kernel patch adding a check to vc4_hdmi_audio_init() in drivers/gpu/drm/vc4/vc4_hdmi.c This would "break" HDMI audio. Returning 0 after devm_snd_dmaengine_pcm_register() returns an error should work as well. As far as I can tell this would only break HDMI audio.
Proposing as a Final FE, since we've slipped it would be good to fix this if we can.
Created attachment 1995708 [details] rpi4 vc4-hdmi patch to work around broken firmware DT Attached patch allows vc40-hdmi to probe successfully with the broken firmware DT as seen on fedora-server. The trivial patch I had in mind unfortunately doesn't work since the returned error is -EPROBE_DEFER. It's impossible to decide whether this is a temporary error (on a system with a DT matching the kernel's expectations and temporary unavailable module) or a permanent error like in the case with the broken firmware DT. I can send the patch upstream tomorrow (Friday CEST) but I have doubts whether it will be accepted.
+5 FE in https://pagure.io/fedora-qa/blocker-review/issue/1424 , marking accepted.
Piling in here mostly so I get notifications for the right bug. I am also seeing this, Raspberry Pi 4B Rev 1.1, BIOS 2023.07 07/01/2023.
(In reply to Janne Grunau from comment #4) > Created attachment 1995708 [details] > rpi4 vc4-hdmi patch to work around broken firmware DT Do we know what revs of upstream firmware this is broken for? Has it been fixed? Do you have details of an upstream issue/PR?
(In reply to Peter Robinson from comment #7) > (In reply to Janne Grunau from comment #4) > > Created attachment 1995708 [details] > > rpi4 vc4-hdmi patch to work around broken firmware DT > > Do we know what revs of upstream firmware this is broken for? Has it been > fixed? Do you have details of an upstream issue/PR? This looks like a deliberate change in the Raspberry Pi kernel tree https://github.com/raspberrypi/linux/commit/0491a0aecb999b1a013ad4a6ad3816c535ac6e73 and breaks compatibility with the upstream kernel.
(In reply to Adam Williamson from comment #0) > The bug can be worked around by booting with nomodeset. I've tried interupting the bootloader to edit the kernel parameters but there's no cursor and the screen isn't redrawing, so I can't see what I'm doing. I also tried editing /root/anaconda-ks.cfg to add `--append=nomodeset` but that doesn't seem to work. Any pointers on how else to apply the workaround would be welcome. I'm not currently familiar enough with the dracut, grub, etc. interactions to know how to do this for SBC images.
(In reply to Jonathan Watt from comment #9) > (In reply to Adam Williamson from comment #0) > > The bug can be worked around by booting with nomodeset. > > I've tried interupting the bootloader to edit the kernel parameters but > there's no cursor and the screen isn't redrawing, so I can't see what I'm > doing. Yes, that it is a huge annoyance. I just write single character, check where it appeared, delete it... and rinse and repeat until I'm on the right position. > Any pointers on how else to apply the workaround would be welcome. I'm not > currently familiar enough with the dracut, grub, etc. interactions to know > how to do this for SBC images. There is a common issue in Fedora Discussion: https://discussion.fedoraproject.org/t/some-raspberry-pi-boards-have-no-display-output-on-fedora-server/95064 You have three options: - add nomodeset on every boot - use arm-image-installer with argument --args "nomodeset" (note: it is possible that new kernel after update won't have nomodeset on the kernel line) - you can add it permanently to grub config (see the link above)
Hey Lukas, does this also affect Fedora 40 Beta Server? (For the purpose of adjusting the Common Issue documentation). Thanks.
With Fedora-Server-40_Beta-1.10.aarch64 on Raspberry Pi 4, I can see display output from u-boot through grub and systemd (with modesetting to 4K) to initial setup. The system works as expected.
Peter, will we be able to 'fix' this at all for F39, or should we just document that the fix is "use f40 or newer"? Thanks!
FEDORA-2024-6dd5b62267 (bcm283x-firmware-20240229-3.dc94391.fc39) has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-6dd5b62267
(In reply to Adam Williamson from comment #13) > Peter, will we be able to 'fix' this at all for F39, or should we just > document that the fix is "use f40 or newer"? Thanks! Yes, I had intended to, had wanted it to get wider testing on newer releases first.
FEDORA-2024-6dd5b62267 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-6dd5b62267` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-6dd5b62267 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-6dd5b62267 (bcm283x-firmware-20240229-3.dc94391.fc39) has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.