Bug 1810134 - Raspberry Pi 4 rev 1.2 gets stuck in dracut shell on boot.
Summary: Raspberry Pi 4 rev 1.2 gets stuck in dracut shell on boot.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 32
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Robinson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-04 15:29 UTC by Steven A. Falco
Modified: 2020-03-16 20:35 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-16 20:35:54 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
journal log from workstation boot (26.32 KB, text/plain)
2020-03-04 19:27 UTC, Steven A. Falco
no flags Details
console serial port log for workstation (47.22 KB, text/plain)
2020-03-04 19:28 UTC, Steven A. Falco
no flags Details
console serial port log for minimal (47.62 KB, text/plain)
2020-03-04 19:31 UTC, Steven A. Falco
no flags Details

Description Steven A. Falco 2020-03-04 15:29:48 UTC
1. Please describe the problem:

Tried booting Fedora-Workstation-32-20200225.n.0.aarch64.raw.xz on an RPi4, hardware revision info:

Hardware        : BCM2835
Revision        : c03112
Serial          : 100000004463a5a7
Model           : Raspberry Pi 4 Model B Rev 1.2 

Instead of booting to a normal login prompt, I wind up in the dracut emergency shell.

2. What is the Version-Release number of the kernel:

5.6.0-0.rc3.git0.1.fc32.aarch64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

I'm not sure where it first appeared, but I do have 5.6.0-0.rc3.git3.1.fc33.aarch64 running properly on an older rawhide image that I have been periodically upgrading.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

I dd'd a fresh copy of Fedora-Workstation-32-20200225.n.0.aarch64.raw.xz to an SD card, inserted it into the RPi4, and it failed to boot.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

I have a second SD card which started out as Fedora-Minimal-Rawhide-20200127.n.0.aarch64.raw.xz, and I have periodically upgraded it.  It is currently running 5.6.0-0.rc3.git3.1.fc33.aarch64 with no problems.  So, I think there is some combination of factors here.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

I'll add the requested logs this afternoon.  I'm out of time right now. :-)

Comment 1 Steven A. Falco 2020-03-04 19:27:16 UTC
Created attachment 1667570 [details]
journal log from workstation boot

Here is the result of journalctl, showing the system ending up in the dracut emergency shell.

Comment 2 Steven A. Falco 2020-03-04 19:28:44 UTC
Created attachment 1667572 [details]
console serial port log for workstation

Here is the serial port log for Fedora-Workstation-32-20200225.n.0.aarch64.raw.xz showing the dracut emergency console.

Comment 3 Steven A. Falco 2020-03-04 19:31:37 UTC
Created attachment 1667573 [details]
console serial port log for minimal

Here in contrast is a log from Fedora-Minimal-32-20200225.n.0.aarch64.raw.xz, which works perfectly.

There is something different between the Minimal and Workstation builds, in that Minimal works fine, but Workstation fails to boot.

Comment 4 billiboy 2020-03-07 13:08:37 UTC
I try to bring up an aarch64 image on a different Soc but also get stuck in the emergency shell.
For me udev does not load any kernel modules and 'udevadm trigger' even segfaults.
Out of curiosity, is udev loading the kernel modules properly for you? I.e. what is the output of 'lsmod' in your emergency shell.

Comment 5 Peter Robinson 2020-03-07 13:26:18 UTC
(In reply to billiboy from comment #4)
> I try to bring up an aarch64 image on a different Soc but also get stuck in
> the emergency shell.

This is explicitly a RPi4 rev 1.2 bug. It's a problem that doesn't affect other SoCs, if you're seeing an issue it's a different bug.

Comment 6 Fedora Update System 2020-03-08 13:13:48 UTC
FEDORA-2020-da32b0c604 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-da32b0c604

Comment 7 Fedora Update System 2020-03-08 17:15:42 UTC
bcm283x-firmware-20200306-1.163d84c.fc32 has been pushed to the Fedora 32 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-da32b0c604

Comment 8 Steven A. Falco 2020-03-08 19:37:31 UTC
I tried merging bcm283x-firmware-20200306-1.163d84c.fc32.aarch64.rpm into several images:

Fedora-Workstation-32-20200225.n.0.aarch64.raw.xz
Fedora-Workstation-32-20200307.n.0.aarch64.raw.xz
Fedora-Workstation-32-20200308.n.0.aarch64.raw.xz

In all cases, the dracut shell bug is still present.

Comment 9 Peter Robinson 2020-03-08 23:36:03 UTC
(In reply to Steven A. Falco from comment #8)
> I tried merging bcm283x-firmware-20200306-1.163d84c.fc32.aarch64.rpm into
> several images:
> 
> Fedora-Workstation-32-20200225.n.0.aarch64.raw.xz
> Fedora-Workstation-32-20200307.n.0.aarch64.raw.xz
> Fedora-Workstation-32-20200308.n.0.aarch64.raw.xz
> 
> In all cases, the dracut shell bug is still present.

Did you also add the latest rawhide kernel into that?

Comment 10 Peter Robinson 2020-03-08 23:37:23 UTC
This bug may also be coming into play (it might also be the issue with other SoCs mentioned previously).
https://www.spinics.net/lists/linux-mmc/msg57733.html

Comment 11 Steven A. Falco 2020-03-09 01:13:13 UTC
I did not try the rawhide kernel.  I'm not sure how I would do that, since we are still running from the initial ram disk when the error occurs.  I could try unpacking the initial ram disk, substituting a different kernel, and repacking the initial ram disk.  Is that the right way to run the experiment you are suggesting?  Or is there a way that I can generate a compose locally with the rawhide kernel?  Running a compose is something I'd be interested in learning how to do, anyway.

I'll try Fedora-Workstation-Rawhide-20200307.n.1.aarch64.raw.xz once it finishes downloading, with and without the bcm283x-firmware-20200306-1.163d84c.fc32.aarch64.rpm package to see if that shows anything interesting.

Also, we should not lose sight of the fact that the Minimal image does not get stuck in dracut, but the Workstation image does.  I don't know if that is due to a timing difference, or something else.

Comment 12 Steven A. Falco 2020-03-09 15:43:20 UTC
I tried Fedora-Workstation-Rawhide-20200307.n.1.aarch64.raw.xz both without bcm283x-firmware-20200306-1.163d84c.fc32.aarch64.rpm and with it.  Both experiments landed in "emergency mode".

This build is using kernel 5.6.0-0.rc4.git1.1.fc33.aarch64.  I see that there is a newer kernel, kernel-5.6.0-0.rc5.git0.1.fc33, that was just built.  I'll see if there is a way that I can hack that into the Workstation images.

Comment 13 Peter Robinson 2020-03-09 18:08:46 UTC
> Also, we should not lose sight of the fact that the Minimal image does not
> get stuck in dracut, but the Workstation image does.  I don't know if that
> is due to a timing difference, or something else.

We should not lose sight that accelerated graphics doesn't work yet on the RPi4 and the support for it is still experimental.

Comment 14 Fedora Update System 2020-03-16 20:35:54 UTC
bcm283x-firmware-20200306-1.163d84c.fc32 has been pushed to the Fedora 32 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.