Created attachment 1919107 [details] journal from last boot on 5.19.16 where GUI did not show on HDMI output The fedora 37 installation on Raspberry Pi 4 was working with gdm + gnome GUI using kernel 5.19.14 but fails to give a GUI screen with kernel 5.19.15 (or 16). The screen is blank. Boot under 5.19.15 or .16 and there is no display on HDMI output. Boot under 5.19.14 and do get the gdm GUI screen.
Several folks have confirmed this, some from our team, and Sally A. Haj and 'yesno' on #fedora-arm chat. Proposing as a Final blocker as a violation of Basic criterion "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility" on Raspberry Pi 4, which is a supported/blocking ARM platform.
*** Bug 2136581 has been marked as a duplicate of this bug. ***
Created attachment 1919292 [details] dmesg output (from geoff marr on the dupe bug)
For reference, rawhide kernel fedora 6.1.0-0.rc1.20221018gitbb1a1146467a.16.fc38.aarch64 fixes this issue.
jforbes asks if someone can test whether 5.19.15-300 works, which would help narrow down the likely causes of this.
In today's go/no-go meeting[1], we agreed that this violates Basic criterion "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility" on RPi 4 [1] https://ia904705.us.archive.org/6/items/37-final-go-no-go-meeting.-2022-10-20-17.13.log/37-final-go_no_go-meeting.2022-10-20-17.13.log.txt
So, some background here: in 5.19.15-301 we backported this patch series - https://lore.kernel.org/lkml/20220929-rpi-pi3-unplugged-fixes-v1-0-cd22e962296c@cerno.tech/ - which is intended to fix video on raspi 3, which an earlier change intended for raspi 4 had apparently broken. We kinda suspect those patches are the cause of this, which is why we want to see if 5.19.15-300 (which doesn't have those patches) works. That patch series is only in drm-misc-fixes upstream, it has not landed in Linus' tree yet AFAICS. So this may be working on Rawhide kernels simply because the change that breaks it isn't in those kernels yet. Justin didn't backport this patch series to the Rawhide kernel (AFAICS), only the F37 one.
I guess it might also be useful to have dmesg/journal output booting with `drm.debug=14`.
I did actually test this on a number of my setups and it was working on the RPi4 when I was testing it (I have around 8 various different RPi setups of varying configs that I test). Also note that the vc4 driver moves widely around, just starting to look at this.
Created attachment 1919316 [details] 5.19.15-301 with drm.debug=14 Here is a boot log (kernel journal messages) with drm.debug=14.
Thanks! I guess same output from a working kernel - ideally 5.19.15-300 , assuming we're right that that works - would also be useful.
Created attachment 1919318 [details] 5.19.14-300 with drm.debug=14 (working kernel for reference) kernel 5.19.14-300 boot log with drm.debug=14 (working video kernel for reference) (Sorry couldn't find .15-300 to test)
Try 'koji download-build --arch=aarch64 kernel-5.19.15-300.fc37'
Thanks. 5.19.15-300 is here: https://koji.fedoraproject.org/koji/buildinfo?buildID=2074685
Created attachment 1919321 [details] 5.19.15-300 with drm.debug=14 (also working kernel for reference) Ah... you guys make it too easy 8-) your supposition seems correct -300 works, 301 does not
I can confirm that 5.19.15 plus https://lore.kernel.org/lkml/20220929-rpi-pi3-unplugged-fixes-v1-0-cd22e962296c@cerno.tech/ are breaking the output on the Pi4. I'm currently looking into it.
So, I think I found a fix. It brings back the Pi4 but I haven't tested it on a Pi3 not connected to a monitor. I'll let you know the result.
I just posted that patch here: https://lore.kernel.org/dri-devel/20221021130908.2202086-1-maxime@cerno.tech/ It works for me on both the Pi4 and Pi3, with vc4 builtin or a module.
* All tests done in F37 Workstation and XFCE Spin. * Kernel 5.19.15-300 and bellow that versions are working. * kernels 5.19.15-301 and above don't work. * Peter Robinson may has different RPi4 revision, and if so, might be that's the reason behind his devices have no issue.
Created attachment 1919398 [details] boot.log_kernel.5.19.16.txt
Created attachment 1919399 [details] boot.log_kernel.5.19.14.txt
Created attachment 1919400 [details] d
Comment on attachment 1919400 [details] d dmesg_kernel.5.19.16-300
Created attachment 1919402 [details] dmesg kernel.5.19.14
Same issue with kernel-6.0.2-301.fc37.
So scratch build here with the proposed upstream fix [1] for this. With a single boot on my rpi4 8gb this boots to GDM, will test wider now but it's here for others to test now too. https://koji.fedoraproject.org/koji/taskinfo?taskID=93277266 [1] https://patchwork.freedesktop.org/patch/508036/ Thanks to Maxime for the super quick turn around here!
For those that would like a copy/paste: rpm -ivh https://kojipkgs.fedoraproject.org//work/tasks/7266/93277266/kernel-5.19.16-300.rpi1.fc37.aarch64.rpm https://kojipkgs.fedoraproject.org//work/tasks/7266/93277266/kernel-core-5.19.16-300.rpi1.fc37.aarch64.rpm https://kojipkgs.fedoraproject.org//work/tasks/7266/93277266/kernel-modules-5.19.16-300.rpi1.fc37.aarch64.rpm
PR for jforbes assuming this passes muster: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2115
This fix worked for me (RPi 4 with 4GB ram). I now see gdm with the build kernel-5.19.16-300.rpi1.fc37.aarch64
FEDORA-2022-1c6a1ca835 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-1c6a1ca835
The new update has solved the problem, thank you all.
FEDORA-2022-1c6a1ca835 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-1c6a1ca835` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1c6a1ca835 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Tested Fedora-Workstation-37-1.3.aarch64.raw.xz with fixed kernel on RPi400, no longer see the video issue!
That sounds like VERIFIED to me!
FEDORA-2022-1c6a1ca835 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
It seems this effectively regressed when we pulled in the 6.0 kernel series: [Sally A. haj] A new installation with 'Fedora-Xfce-37-1.6.aarch64.raw.xz', has blank screen, seems the same issue as before. Has anyone tested? [Sally A. haj] I think it comes with kernel-6.0.6-300.fc37 [jforbes] Sally A. haj: wouldn't be the same issue, drm-vc4-hdmi-enforce-the-minimum-rate-at-runtime_res.patch was actually included in 6.0.6 upstream [Sally A. haj] jforbes: Same problem with kernel-6.0.7-300.fc37, installed this kernel in another ssd. by the way "5.19.16-301.fc37" is working fine [jforbes] Sally A. haj: RPi 4? [Sally A. haj] yes jforbes says "The old rPi 4 issue came back, we are carrying the patch still in 6.0, but the rPi 3 patches were not in 6.0...when they came in, it messed up the merge. I have a test built already". Anyhow, the upshot is, this is borked again and we need a new kernel (it will be 6.0.7-301) to fix it.
FEDORA-2022-f3e83cad6f has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f3e83cad6f
pbrobinson wrote on the kernel channel this morning: "the 6.0.7-301.fc37.aarch64 kernel looks good for me on the rpi4/workstation". And that's definitely the kernel in RC-1.7. So I think we're good here, though it'd be great if someone can absolutely check that the RC-1.7 images work on a Pi 4.
Fedora 37 RC-1.7/workstation on RPi4 is booting and working just fine. Note: There is no graphical boot splash, it's just text lines moving. (I think it's related to "plymouth")
Thanks for the confirmation!
FEDORA-2022-f3e83cad6f has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.