When attempting to boot Fedora-Workstation-39_Beta-1.1.aarch64.raw image on the Raspberry Pi 4, system boots to a black screen. Adding 'nomodeset' to the kernel args, initial-setup is displayed as expected. Reproducible: Always Steps to Reproduce: Boot the RPi4 with uboot-tools-2023.10-0.4.rc3.fc39 Actual Results: Black screen Expected Results: Initial-setup Downgrading to uboot-tools-2023.07-2.fc39 works as expected. It appears the dtb file might not be loaded when using uboot-tools-2023.10-0.4.rc3.fc39.
Investigating, will need to bisect the upstream commits.
Proposed as a Blocker for 39-final by Fedora user pbrobinson using the blocker tracking app because: The RPi4 is a key aarch64 platform for workstation
Created attachment 1990989 [details] Fedora-Workstation-39_Beta-1.1.aarch64 journalctl
Out of interest, how did we miss this for Beta release? Did nobody with an rpi test it?
(In reply to Adam Williamson from comment #4) > Out of interest, how did we miss this for Beta release? Did nobody with an > rpi test it? TBH I've not seen any QE testing on aarch64, I've had little time so I've been focused on core boot testing on Minimal and IoT and haven't done any desktop testing at all because just not enough cycles. pwhalen is no longer the arm QE person and those testing pieces were supposed to be taken over by others, I'm not sure who those others are. I suggest we add this to the release retrospective.
sgallagh did some testing for Beta, that's right, I remember now. During the go/no-go meeting. He used a VM. So looks like we didn't do any HW tests. :/
I'd like to thank pwhalen for digging in here and opening the bug. For our FCOS test day [1] we were executing our Rpi4 test case [2] and were having trouble and after working around one issue [3] we then hit the VGA issue reported here. [1] https://testdays.fedoraproject.org/events/159 [2] https://fedoraproject.org/wiki/QA:Testcase_CoreOS_Raspberry_Pi#aarch64 [3] https://github.com/coreos/fedora-coreos-docs/pull/590
Accepted as F39 Final blocker in https://pagure.io/fedora-qa/blocker-review/issue/1352
Might https://github.com/u-boot/u-boot/commit/833ff23047c50e4053fb1bda21f4e2c9f6a3aa6a help with this?
specifically, I'm wondering if https://github.com/u-boot/u-boot/commit/6826c432e35115bbdafc17062733f5030b7f8366 might be the thing we need.
(In reply to Adam Williamson from comment #10) > specifically, I'm wondering if > https://github.com/u-boot/u-boot/commit/ > 6826c432e35115bbdafc17062733f5030b7f8366 might be the thing we need. Err, nope, RPi doesn't have a SPL (secondary program loader) or ATF at all due to it's "special" firmware so absolutely none of that PR relates to RPi at all (hence why it's not in that patch series at all).
I am working through this, I thought I had the problem and hence solution but it's incomplete.
*** Bug 2244237 has been marked as a duplicate of this bug. ***
So we've basically got to the bottom of the problem and worked out the issue, I now just need to come up with a fix.
🥳🥳
So this bug also causes an issue with Jetson Nano (probably all older pre Xavier Jetsons). It's a conflict between upstream and Fedora's DTs being in /boot (2nd partition on a UEFI system). https://pbrobinson.fedorapeople.org/rpi-u-boot/u-boot.bin The above U-Boot build should boot and work on a RPi4 booting off the mSD card (sudo cp ~/u-boot.bin /boot/efi/rpi-u-boot.bin), it will only work with mSD as it's hard coded as the test for the logic. I have most of the logic now in place to search for the DT correctly but U-Boot isn't running the script where/when I need it to and I need to ensure the logic is sound across mSD/eMMC/NVME/USB/SATA etc for the devices we support. The reason I never saw it was that the fallback is that U-Boot falls back to use the U-Boot provided DT which for the devices I test regularly is the same as the Linux kernel so I wasn't seeing the issue.
Thanks for this test uboot build pbrobinson. I have tested this using RPi4 on Fedora-Minimal-39-20231016.n.0.aarch64.raw.xz, Fedora-Workstation-39-20231016.n.0.aarch64.raw.xz, and Fedora-Workstation-39-20231023.n.0.aarch64.raw.xz, system now boots to CLI/GUI as it should in all three cases!
I can confirm system now boots to CLI on Fedora-Minimal-39-20231023.n.0.aarch64.raw.xz using RPi 4. On CoreOS this particular build Failed to load 'dtb/broadcom/bcn2711-rpi-4-b.dtb ERROR: invalid device tree U-Boot>
(In reply to Hristo Marinov from comment #18) > I can confirm system now boots to CLI on > Fedora-Minimal-39-20231023.n.0.aarch64.raw.xz using RPi 4. > > On CoreOS this particular build > Failed to load 'dtb/broadcom/bcn2711-rpi-4-b.dtb > ERROR: invalid device tree > U-Boot> In both cases can you confirm what U-Boot you were using, was it my test one, in the later case did you see get display output right to login?
Created attachment 1995273 [details] U-Boot test CoreOS video I downloaded and copied the file from the link in comment 16 again to Fedora-Minimal and CoreOS, just in case. The video shows the CoreOS boot process with the u-boot.bin test build from comment 16.
FEDORA-2023-7994524575 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575
With the rpms from https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575 : /tmp/RPi4boot/bcm283x-firmware-20231017-1.ce3a0b4.fc39.aarch64.rpm /tmp/RPi4boot/bcm283x-overlays-20231017-1.ce3a0b4.fc39.aarch64.rpm /tmp/RPi4boot/uboot-images-armv8-2023.07-3.fc39.noarch.rpm I'm seeing a different issue for my RPi4 after following the CoreOS install steps with the updated copying of /usr/share/uboot/rpi_arm64/u-boot.bin into /boot/efi/rpi-u-boot.bin from https://github.com/coreos/fedora-coreos-docs/pull/590 The issue I'm seeing now is the Pi4 won't boot. Seems like it is trying to boot from USB and not from SD card? See attachment. Note that these RPMs don't ship in Fedora CoreOS so if this problem ends up somehow being Fedora CoreOS specific I wouldn't consider it a blocker for the rest of F39 GA as it can be fixed later and shipped via an update.
Created attachment 1995504 [details] CoreOS Fail to load from mSD
can somebody please test with https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231026.0/ and see if this is fixed?
Fedora-Workstation-39-1.2.aarch64.raw.xz boots with display on the RPi4-4GB
(In reply to Adam Williamson from comment #24) > can somebody please test with > https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231026.0/ and see > if this is fixed? I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B.
Brandon: did you use arm-image-installer to write it? Do the other variants work?
Can confirm, Fedora-Workstation-39-1.2.aarch64.raw.xz boots with display on the RPi400-4GB.
> I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B. So server is a little special here in that the firmware can't load the kernel DT off /boot (due to XFS) but it should still use the RPi DT, do you have any modification in the config.txt?
(In reply to Adam Williamson from comment #27) > Brandon: did you use arm-image-installer to write it? Do the other variants > work? sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs --addkey ~/.ssh/pi/id_ed25519.pub Just tried workstation, it boots to gnome-initial-setup, but crashes back to GDM every attempt at signing in. Invocation was: sudo arm-image-installer --image=/home/nielsenb/Desktop/Fedora-Workstation-39-1.2.aarch64.raw.xz --target=rpi4 --media=/dev/sda --resizefs
(In reply to Peter Robinson from comment #29) > > I need `nomodeset` for Fedora-Server-39-1.2.aarch64.raw.xz to boot on my 4B. > > So server is a little special here in that the firmware can't load the > kernel DT off /boot (due to XFS) but it should still use the RPi DT, do you > have any modification in the config.txt? There shouldn't be any, especially assuming it gets rewritten every arm-image-installer use.
(In reply to Brandon Nielsen from comment #30) > (In reply to Adam Williamson from comment #27) > > Brandon: did you use arm-image-installer to write it? Do the other variants > > work? > > sudo arm-image-installer > --image=/home/nielsenb/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz > --target=rpi4 --media=/dev/sda --resizefs --addkey ~/.ssh/pi/id_ed25519.pub > > Just tried workstation, it boots to gnome-initial-setup, but crashes back to > GDM every attempt at signing in. Invocation was: > > sudo arm-image-installer > --image=/home/nielsenb/Desktop/Fedora-Workstation-39-1.2.aarch64.raw.xz > --target=rpi4 --media=/dev/sda --resizefs In regards to Workstation, every time it crashes back to GDM, the only interesting thing I see with dmesg is: [ 369.533317] hdmi-audio-codec hdmi-audio-codec.1.auto: ASoC: error at snd_soc_dai_startup on i2s-hifi: -19 [ 369.533351] MAI: ASoC: error at __soc_pcm_open on MAI: -19
(In reply to Brandon Nielsen from comment #32) > > In regards to Workstation, every time it crashes back to GDM, the only > interesting thing I see with dmesg is: > nomodeset also fixes the crash back to GDM. Also note I was in error, this is testing with a 4 GB Pi 4B Rev 1.1.
Writing server with dd (sudo xzcat ~/Desktop/Fedora-Server-39-1.2.aarch64.raw.xz | sudo dd of=/dev/sda bs=1M oflag=direct status=progress), results in an image that behaves as expected (successfully boots, but to a black screen without `nomodeset`, successfully boots to initial setup with `nomodeset`. I have opened another bug for the GDM issue: https://bugzilla.redhat.com/show_bug.cgi?id=2246442
so in general, I think we can consider the actual originally reported bug here fixed. We can follow up on Brandon's bug in https://bugzilla.redhat.com/show_bug.cgi?id=2246442 and the Server bug discovered by Hristo in https://bugzilla.redhat.com/show_bug.cgi?id=2246428 . Marking this as VERIFIED.
I have tested "Fedora-Minimal-39-1.2.aarch64.raw.xz" on mSD card, used "arm-image-installer" to flash it, the result is that, It's booting as expected (working), and I've tested "Fedora-Workstation-39-1.2.aarch64.raw.xz" as well, but on SSD, used "arm-image-installer, it's booting just fine. Thank you.
FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
update on https://bugzilla.redhat.com/show_bug.cgi?id=2241252#c22 This was user error. I was trying to test out the packages before they were in any repos and I ended up missing bcm2711-firmware and bcm2835-firmware when grabbing them from koji.
For the sake of completeness: With RC 1.2, using arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset" --media=/dev/mmcblk0 resolves the issue for the Raspi 4. So that's an easy workaround and can be covered in our Server documentation. I have not been able to test this on any of my other devices (which are more important for Server), because they didn't boot at all. So it's not sure whether this will work on other device models (though it probably will).
(In reply to Peter Boy from comment #39) > For the sake of completeness: > > With RC 1.2, using > arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args > "nomodeset" --media=/dev/mmcblk0 > > resolves the issue for the Raspi 4. So that's an easy workaround and can be > covered in our Server documentation. Adding "nomodeset" doesn't "resolve" the issue, it works around the issue. > I have not been able to test this on any of my other devices (which are more > important for Server), because they didn't boot at all. > So it's not sure whether this will work on other device models (though it > probably will). Can you file explicit bugs for each of the problems with the "other devices", each of these problems will a independent and different and we can't work to fix them without that, I very much doubt it will work on "other device models" and if it does it's by chance. It's really not that simple.
> > resolves the issue for the Raspi 4. So that's an easy workaround and can be > > covered in our Server documentation. > > Adding "nomodeset" doesn't "resolve" the issue, it works around the issue. Yes, of course. My internal translator seems to be a bit "rough" at times. What I wanted to express is that I and very likely my server WG colleagues are happy with this 'workaround solution'. :-) > Can you file explicit bugs for each of the problems https://bugzilla.redhat.com/show_bug.cgi?id=2246871
FEDORA-2023-7994524575 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
I tested "Fedora-Workstation-39-1.4.aarch64.raw.xz" and "Fedora-Minimal-39-1.4.aarch64.raw.xz" with arm-image-installer-3.9, both of them boot successfully. Note: At the end of arm-image-installer process, it showed: The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. ./arm-image-installer-arm-image-installer-3.9/arm-image-installer: 482: btrfs: not found = Raspberry Pi 4 Uboot is already in place, no changes needed. ./arm-image-installer-arm-image-installer-3.9/arm-image-installer: 621: getenforce: not found = Installation Complete! Insert into the rpi4 and boot * I have used the extracted zip file of arm-image-installer on Debian, I wasn't in the same directory of extracted folder.
I still need --args "nomodeset" passed to arm-image-installer with the FEDORA-2023-7994524575 updates. Without it, system boots to a black screen. Tested Fedora-Server-39-1.4.aarch64.raw.xz on a 4 GB Pi 4B Rev 1.1.
That's a different issue: https://bugzilla.redhat.com/show_bug.cgi?id=2246428
FEDORA-2024-1d0b793bc1 (arm-image-installer-4.1-3.fc40 and uboot-tools-2024.04-0.8.rc5.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2024-1d0b793bc1
Proposed as a Blocker for 40-final by Fedora user pbrobinson using the blocker tracking app because: The final complete fix for this that we decided to document and not respin beta for never made it stable due to lack of karma so we need this in GA with zero doubt.
Adjusting fields.
Third time lucky, this was the f39 bug hence the wrong bits.
Putting the fields back.