Bug 2011423 - RPi-400 - fails to boot with 2021.10 - mmc0: invalid bus width
Summary: RPi-400 - fails to boot with 2021.10 - mmc0: invalid bus width
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 36
Hardware: aarch64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: ARMTracker
TreeView+ depends on / blocked
 
Reported: 2021-10-06 15:02 UTC by Paul Whalen
Modified: 2023-05-25 19:38 UTC (History)
31 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-25 19:38:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
log of failed boot of Fedora-Minimal-35-1.2.aarch64.raw.xz on RPI4 8GB (10.67 KB, application/gzip)
2021-10-29 10:12 UTC, Frank Danapfel
no flags Details

Description Paul Whalen 2021-10-06 15:02:02 UTC
Description of problem:

Attempting to boot the RPi400 with uboot-tools-2021.10-1.fc35 fails with the following error repeated

[   23.649955] mmc0: error -22 whilst initialising SD card
[   23.912382] mmc0: invalid bus width
[   23.915927] mmc0: error -22 whilst initialising SD card
[   25.217407] mmc0: invalid bus width
[   25.220994] mmc0: error -22 whilst initialising SD card
[   25.472831] mmc0: invalid bus width
[   25.476392] mmc0: error -22 whilst initialising SD card
[   25.739497] mmc0: invalid bus width
[   25.743089] mmc0: error -22 whilst initialising SD card


Version-Release number of selected component (if applicable):
First noticed with uboot-tools-2021.10-0.7.rc5.fc35

How reproducible:
Always

Steps to Reproduce:
1. Boot a recent disk image on the RPi400, update U-Boot, reboot

Working on other Raspberry Pi's.

Comment 1 Paul Whalen 2021-10-06 17:59:33 UTC
Output from uboot-tools-2021.10-0.7.rc5.fc35:

[   12.146693] mmc0: ADMA error: 0x02000000
[   12.152411] hub 1-0:1.0: 1 port detected
[   12.154445] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[   12.154451] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00001002
[   12.159435] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.14
[   12.164956] mmc0: sdhci: Blk size:  0x00007008 | Blk cnt:  0x00000001
[   12.164964] mmc0: sdhci: Argument:  0x00000000 | Trn mode: 0x00000013
[   12.164970] mmc0: sdhci: Present:   0x1fff0206 | Host ctl: 0x00000011
[   12.164977] mmc0: sdhci: Power:     0x0000000f | Blk gap:  0x00000080
[   12.164983] mmc0: sdhci: Wake-up:   0x00000000 | Clock:    0x00007d07
[   12.164990] mmc0: sdhci: Timeout:   0x00000000 | Int stat: 0x00000000
[   12.164995] mmc0: sdhci: Int enab:  0x03ff100b | Sig enab: 0x03ff100b
[   12.165001] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[   12.165007] mmc0: sdhci: Caps:      0x45ee6432 | Caps_1:   0x0000a525
[   12.165013] mmc0: sdhci: Cmd:       0x0000333a | Max curr: 0x00080008
[   12.165019] mmc0: sdhci: Resp[0]:   0x00000920 | Resp[1]:  0x00edc87f
[   12.165026] mmc0: sdhci: Resp[2]:   0x325b5900 | Resp[3]:  0x00400e00
[   12.171575] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.179932] mmc0: sdhci: Host ctl2: 0x00000008
[   12.186483] usb usb2: Product: xHCI Host Controller
[   12.192987] mmc0: sdhci: ADMA Err:  0x00000001 | ADMA Ptr: 0xe9000200
[   12.192993] mmc0: sdhci: ============================================
[   12.192998] mmc0: sdhci: e9000200: DMA 0xf4000000, LEN 0x0008, Attr=0x21
[   12.199532] usb usb2: Manufacturer: Linux 5.14.9-300.fc35.aarch64 xhci-hcd
[   12.206055] mmc0: sdhci: e9000208: DMA 0x00000000, LEN 0x0000, Attr=0x03
[   12.308844] usb usb2: SerialNumber: 0000:01:00.0
[   12.313701] mmc0: error -5 whilst initialising SD card
[   12.315011] hub 2-0:1.0: USB hub found
[   12.323086] hub 2-0:1.0: 4 ports detected
[   12.328024] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[   12.340204] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[   12.347625] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[   12.356459] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[   12.363903] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[   12.439475] mmc1: new high speed SDIO card at address 0001

Comment 2 Paul Whalen 2021-10-06 18:02:04 UTC
Created attachment 1829988 [details]
boot with uboot-2021.10 rc5

Comment 3 Geoffrey Marr 2021-10-07 00:49:05 UTC
Tested with updated Fedora-Workstation-35-20210920.n.0.aarch64.raw.xz and uboot-2021.10-1.fc35 on RPi400. Could not reproduce. Will try with a newer image that hasn't been updated from.

Comment 4 Geoffrey Marr 2021-10-07 16:53:55 UTC
Tested with Fedora-Minimal-35_Beta-1.2.aarch64.raw.xz and uboot-2021.10-1.fc35 installed on the command line on RPi400. Ran "rpi-uboot-update" after and rebooted (unlike on install from comment 3). System would not fully boot and would eventually escape to a dracut emergency terminal.

Comment 5 Peter Robinson 2021-10-08 06:47:57 UTC
Can you please provide more information than "System would not fully boot and would eventually escape to a dracut emergency terminal." please?

Comment 6 Paul Whalen 2021-10-08 16:44:29 UTC
Removing the dtb symlink to force it to use the firmware provided dtb works.

Comment 7 Frank Danapfel 2021-10-28 13:39:15 UTC
Hello,

I'm having the same issue trying to get both Fedora 34 and 35 aarch64 to boot from SD Card (Samsung Evo 32GB) on a Raspberry Pi 4 8GB. I've tried various different images, the latest being Fedora-Workstation-34-1.2.aarch64.raw.xz and Fedora-Workstation-35-20211027.n.0.aarch64.raw.xz, but all end up hanging with the same error message mentioned in comment #0.

Booting the latest version of Raspberry OS using exactly the sam SD card works without issues.

Tried to use a different SD card as well (SanDisk 64GB), but this din't make a difference.

I've also tried updating the firmware of the Raspberry Pi 4, but that didn't help either

(In reply to Paul Whalen from comment #6)
> Removing the dtb symlink to force it to use the firmware provided dtb works.

Could you provide some instructions on how to do this?

Regards,
Frank

Comment 8 Paul Whalen 2021-10-28 15:36:30 UTC
(In reply to Frank Danapfel from comment #7)
> Hello,
> 
> I'm having the same issue trying to get both Fedora 34 and 35 aarch64 to
> boot from SD Card (Samsung Evo 32GB) on a Raspberry Pi 4 8GB. I've tried
> various different images, the latest being
> Fedora-Workstation-34-1.2.aarch64.raw.xz and
> Fedora-Workstation-35-20211027.n.0.aarch64.raw.xz, but all end up hanging
> with the same error message mentioned in comment #0.
> 

The expected Fedora 35 GA images are here - https://kojipkgs.fedoraproject.org/compose/35/Fedora-35-20211026.0/compose/Spins/aarch64/images/

Could you try the Minimal image and give us the output from a serial console? It should work as expected

> 
> (In reply to Paul Whalen from comment #6)
> > Removing the dtb symlink to force it to use the firmware provided dtb works.
> 
> Could you provide some instructions on how to do this?

Delete the symlink '/boot/dtb' on the image, this will force the Pi4 to use the dtb files in /boot/efi

Comment 9 Frank Danapfel 2021-10-28 17:53:29 UTC
Hello Paul,

thanks for the quick response!

(In reply to Paul Whalen from comment #8)
> (In reply to Frank Danapfel from comment #7)
> > Hello,
> > 
> > I'm having the same issue trying to get both Fedora 34 and 35 aarch64 to
> > boot from SD Card (Samsung Evo 32GB) on a Raspberry Pi 4 8GB. I've tried
> > various different images, the latest being
> > Fedora-Workstation-34-1.2.aarch64.raw.xz and
> > Fedora-Workstation-35-20211027.n.0.aarch64.raw.xz, but all end up hanging
> > with the same error message mentioned in comment #0.
> > 
> 
> The expected Fedora 35 GA images are here -
> https://kojipkgs.fedoraproject.org/compose/35/Fedora-35-20211026.0/compose/
> Spins/aarch64/images/
> 
> Could you try the Minimal image and give us the output from a serial
> console? It should work as expected

Unfortunately I got the same issue using the Fedora-Minimal-35-1.2.aarch64.raw.xz from the link you provided:
---
$ sudo fedora-arm-image-installer --image=./Fedora-Minimal-35-1.2.aarch64.raw.xz --media=/dev/mmcblk0 --showboot --target=rpi4
[sudo] password for frank: 

=====================================================
= Selected Image:                                 
= ./Fedora-Minimal-35-1.2.aarch64.raw.xz
= Selected Media : /dev/mmcblk0
= U-Boot Target : rpi4
= Boot messages will be shown onscreen.
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
 
 Type 'YES' to proceed, anything else to exit now 
 
= Proceed? YES
= Writing: 
= ./Fedora-Minimal-35-1.2.aarch64.raw.xz 
= To: /dev/mmcblk0 ....
6414139392 bytes (6.4 GB, 6.0 GiB) copied, 126 s, 50.9 MB/s
0+714503 records in
0+714503 records out
6442450944 bytes (6.4 GB, 6.0 GiB) copied, 348.914 s, 18.5 MB/s
= Writing image complete!
= Raspberry Pi 4 Uboot is already in place, no changes needed.

= Installation Complete! Insert into the rpi4 and boot.
---

> > 
> > (In reply to Paul Whalen from comment #6)
> > > Removing the dtb symlink to force it to use the firmware provided dtb works.
> > 
> > Could you provide some instructions on how to do this?
> 
> Delete the symlink '/boot/dtb' on the image, this will force the Pi4 to use
> the dtb files in /boot/efi

OK, thanks. After figuring out which of the 3 partitions that were written on the SD card using the image was actually 'boot' i was able to find the 'dtb' symlink and remove it, and afterwards boot Fedora on the RPI4 without the error.

Unfortunately for some reason I'm not able to capture the entire output from the boot process via serial console, all I get is the output of the bootloader upto the following lines:
---
U-Boot 2021.10 (Oct 14 2021 - 00:00:00 +0000)

DRAM:  7.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In:    serial
Out:   vidconsole
Err:   vidconsole
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 5000420 NbrPorts 5
Starting the controller
USB XHCI 1.00
scanning bus xhci_pci for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
827392 bytes read in 57 ms (13.8 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
ESC7ESC[rESC[999;999HESC[6nESC8Card did not respond to voltage select! : -110
Scanning disk mmcnr...
Disk mmcnr not ready
Scanning disk mmc...
Found 4 disks
No EFI system partition
Booting /efi\boot\bootaa64.efi
No EFI system partition
Failed to persist EFI variables
No EFI system partition
Failed to persist EFI variables
No EFI system partition
Failed to persist EFI variables
ESC[0;37;40mESC[0;37;40methernet@7d580000 Waiting for PHY auto negotiation to complete......... TIMEOUT !
bcmgenet: PHY startup failed: -110
<some garbled output omitted>
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
---

Any kernel and other output after that is only shown on the HDMI console. Is there a trick to also get the rest of the output thats shown during boot to also be logged on the serial console?

Comment 10 Paul Whalen 2021-10-28 20:37:11 UTC
(In reply to Frank Danapfel from comment #9)

> OK, thanks. After figuring out which of the 3 partitions that were written
> on the SD card using the image was actually 'boot' i was able to find the

Apologies, I should have been more specific. 

> 'dtb' symlink and remove it, and afterwards boot Fedora on the RPI4 without
> the error.

Could you check the version of the eeprom bootloader by plugging in the RPi4-8GB without any media. On HDMI it will show a Raspberry image and list some information including the version. 

On my working RPi4-8GB: bootloader - c2f8c388 - Apr 29 2021

> 
> Any kernel and other output after that is only shown on the HDMI console. Is
> there a trick to also get the rest of the output thats shown during boot to
> also be logged on the serial console?

If you removed the dtb symlink you would add 'console=ttyS0,115200' to the kernel parameters. With the symlink the RPi4 uses the kernel provided dtb and the serial would be 'console=ttyS1,115200'. 

Ideally, the output from the kernel provided dtb would be most helpful.

Comment 11 Frank Danapfel 2021-10-29 10:12:40 UTC
Created attachment 1838267 [details]
log of failed boot of Fedora-Minimal-35-1.2.aarch64.raw.xz on RPI4 8GB

(In reply to Paul Whalen from comment #10)
> (In reply to Frank Danapfel from comment #9)
> 
> > OK, thanks. After figuring out which of the 3 partitions that were written
> > on the SD card using the image was actually 'boot' i was able to find the
> 
> Apologies, I should have been more specific. 

No worries. It wasn't too hard to figure it out.
 
> Could you check the version of the eeprom bootloader by plugging in the
> RPi4-8GB without any media. On HDMI it will show a Raspberry image and list
> some information including the version. 
> 
> On my working RPi4-8GB: bootloader - c2f8c388 - Apr 29 2021

The version on my RPI4-8GB is identical:
---
Raspberry Pi 4 Model B . 8GB
bootloader: c2f8c388  Apr 29 2021
update-ts: 1633676244
---

> > Any kernel and other output after that is only shown on the HDMI console. Is
> > there a trick to also get the rest of the output thats shown during boot to
> > also be logged on the serial console?
> 
> If you removed the dtb symlink you would add 'console=ttyS0,115200' to the
> kernel parameters. With the symlink the RPi4 uses the kernel provided dtb
> and the serial would be 'console=ttyS1,115200'. 

Thanks. I ended up specifying '--addconsole' as an additional parameter when creating the SD card using fedora-arm-image-installer, afterwards the kernel and other output is now logged both on HDMI and on the serial console. 
 
> Ideally, the output from the kernel provided dtb would be most helpful.

See the attached screenlog.0.gz. Hope tha this contains all the info you need.

Comment 12 David W. Legg 2022-01-12 20:31:20 UTC
@pwhalen

Paul,
Did you ever get a proper fix for this problem?

I only ask because I am getting the same symptoms with a 2018 4GB rpi4 board and:

Fedora-Minimal-34-1.2.aarch64.raw.xz

and

Fedora-Minimal-35-1.2.aarch64.raw.xz

dwlegg

Comment 13 David W. Legg 2022-01-13 15:57:00 UTC
@fdanapfe and @pwhalen

Comment 14 David W. Legg 2022-01-13 16:09:01 UTC
So my 4GB rpi4 refused to boot with the symptoms described about, i.e. mmc0 errors.

I also found that Fedora-Minimal-34-1.2.aarch64.raw.xz and Fedora-Minimal-35-1.2.aarch64.raw.xz also failed to boot.
So, I downloaded the latest PIOS image, cut an SD card, and (hey presto) it booted (so no problems with my pi, psu or other hardware).

However, there is a hint in the Fedora aarch64 wiki that one needs to remove /boot/dtb to get Fedora to boot.
The only problem was that my Fedoras 34 and 35 both fell over without ever creating /boot :(
So, here is what I did from my root history on my x86_64 Fedora desktop box:-

 1002  # Put an SD card in USB card writer/reader.
 1003  arm-image-installer --image=Fedora-Minimal-35-1.2.aarch64.raw.xz --target=rpi4 --media=/dev/sdX # Your X will vary. Use journalctl -f

 1005  mkdir /mnt/usb
 1006  mount /dev/sdX2 /mnt/usb/

 1007  # You have remove the dtb soft link and the directory it point to:-
 1008  rm /mnt/usb/dtb
 1009  rm /mnt/usb/dtb-5.14.10-300.fc35.aarch64/
 1010  rm -rf /mnt/usb/dtb-5.14.10-300.fc35.aarch64/

 1012  umount /mnt/usb 
 1017  gparted /dev/sdX&   # Usegparted to enlarge /dev/sdf3 partition so that the whole image is < 10MB.
 1018  parted /dev/sdX

 1019  # Take a copy of the SD Card image and check that all partitions are actually there:-
 1020  dd if=/dev/sdX of=Fedora-Minimal-35-1.2.aarch64-rm_dtb-size-10000MB.img status=progress bs=1M count=10000
 1021  
 1023  fdisk -l Fedora-Minimal-35-1.2.aarch64-rm_dtb-size-10000MB.img
 1024  # Put card into rpi4 and boot.

This works for F34 and F35 on a 4GB rpi4. I never had to do all this previously, but maybe these instructions will help some-one :)

Meanwhile, the Fedora wiki needs to be updated, and/or the Fedora .xz file images need to be fixed by some-one with the superpowers.

Comment 15 David W. Legg 2022-02-03 09:39:45 UTC
Apparently, this step (above) is not required:

 1009  rm /mnt/usb/dtb-5.14.10-300.fc35.aarch64/

Comment 16 Ben Cotton 2022-02-08 20:18:01 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 17 Peter Robinson 2022-08-21 10:15:36 UTC
This is now fixed in Fedora 37 and later releases. You need the latest U-Boot 2022.10 RC build.

It is unlikely this will be back ported to earlier releases because it really needs to be part of the release media and to back port it will take a lot of my time for wide spread testing and I don't really have enough spare time ATM.

Comment 18 Peter Robinson 2022-08-21 10:25:49 UTC
Basically the summary of this problem is that RPi introduced a new rev of the SoC in the RPi4 series of devices and adjusted some pieces of how bits of the SoC is wired up. The RPi400, and rev 1.4 and later of the RPi4b have this new rev of the SoC. We need to handle it at the firmware level so Linux doesn't have to deal with it.

Comment 19 Frank Danapfel 2022-09-16 09:10:47 UTC
(In reply to Peter Robinson from comment #17)
> This is now fixed in Fedora 37 and later releases. You need the latest
> U-Boot 2022.10 RC build.

Thanks a lot for fixing this.

I just tried booting my RPi4B using https://kojipkgs.fedoraproject.org/compose/37/latest-Fedora-37/compose/Spins/aarch64/images/Fedora-Minimal-37_Beta-1.5.aarch64.raw.xz and it now boots without this issue.

Comment 20 Ben Cotton 2023-04-25 18:28:50 UTC
This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 21 Ludek Smid 2023-05-25 19:38:55 UTC
Fedora Linux 36 entered end-of-life (EOL) status on 2023-05-16.

Fedora Linux 36 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.