Another person confirmed this problem here:
https://discussion.fedoraproject.org/t/fedora-39-server-does-not-boot-on-raspberry-pi-4-from-microsd-card-slot/92755

Proposing for a blocker discussion.

Could this just be https://bugzilla.redhat.com/show_bug.cgi?id=2241252 , but with an extra error message that happens to be present on your device? Does it work if you boot with 'nomodeset' ?

> 2. Boot from the Raspberry Pi 4's microSD card slot
> Actual Results:  
> The following error appears on the U-Boot screen:
> 
> scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping...
> (3db57620 00000004 01000000 01008401)

That error is a USB error, nothing to do with the mSD, so I'd really need more details.

> sudo arm-image-installer --target=rpi4
> --image=Fedora-Server-39-20231010.n.0.aarch64.raw.xz --media=/dev/sdc
> --resizefs --showboot --norootpass --args "nomodeset" --addkey

Adam: I think he's already answered the nomodeset question.

oh, whoops, I didn't see that part. so this is a different bug? that would be unfortunate...

(In reply to Adam Williamson from comment #4)
> oh, whoops, I didn't see that part. so this is a different bug? that would
> be unfortunate...

I think I've already got a fix for that one, it's been discussed upstream, but I'm not sure around the USB crash.

The asahi downstream u-boot repo (https://github.com/AsahiLinux/u-boot/commits/asahi-releng) has several usb related fixes which might fix this issue. The at least some of the issues were triggered by specific connected usb devices.

Does the issue reproduce without any usb devices connected?

> scanning bus xhci_pci for devices... Unexpected XHCI event TRB, skipping...
> (3db57620 00000004 01000000 01008401)

I'm seeing this problem on occasion, it's causing me issues bisecting the screen problem, I *think* it's fixed in the upstream 2023.10 GA release.

> Does the issue reproduce without any usb devices connected?

Yes, I can, what USB controller do you have downstream on asahi?

Not sure if all issues were controller specific. SoC based usb4/thunderbolt ports use dwc3 for usb3, also present on desktop devices are PCIe xhci controllers from Fresco Logic or Asmedia.

Created attachment 1994206 [details]
Shows U-Boot Screen

Not sure if this would be helpful, but here is the short video of what happens after I apply power to the board.

I haven't been able to reproduce with Fedora-Server-39-20231014.n.0.aarch64.raw.xz on my Pi 4B 8 GB. Both uSD and USB boot seem to work consistently.

Discussed at 2023-10-16 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2023-10-16/f39-blocker-review.2023-10-16-16.00.html . Accepted as a blocker as a clear violation of the requirements for release-blocking images to boot and deploy on Pi 4, which is a blocking platform for aarch64. We decided to take this one on its face for now, as we can adjust the decision later if it turns out this is rare or something.

Created attachment 1994294 [details]
Shows smbiosview output

Here's more info about the board provided by the smbiosview command from the UEFI Firmware Shell.

So upstream has a fix for this, from reports (OpenSUSE) it's not perfect there's also disagreement upstream on the fix (partially from me) but I think the fix will be good enough for most of the Fedora usecases for the short term.

https://koji.fedoraproject.org/koji/taskinfo?taskID=107868656

This build (it's rawhide but will be fine on f39) should have the MMC, and maybe fix the USB. It would be good to get feedback to see if it improves the situation (note this absolutely does NOT fix the screen bug).

> So upstream has a fix for this, from reports (OpenSUSE) it's not perfect

And others.

Just wanted to add my two-cents here: over the last week, I have booted an RPi4 with SD cards using several different days (20231014, 20231016, and 20231023) F39 images (minimal, server, workstation) and have not been able to trigger this bug. Since last Monday, I've written and booted around 20 microSD cards and haven't had any issues. This is just my experience, but wanted to make this known, should our blocker process get hung up on this bug and we need justification to waive.

Unfortunately at least from my side until Fedora-Server-39-20231024.n.0.aarch64.raw.xz the issue is still there.

However, if it comes to a situation where this is the only blocker, I personally think that this could be revisited and possibly overturned.

I was only able to reproduce a crash in u-boot during xhci init once in ~25 tries. Using a display and not serial console so I can't say with certainty that's the same crash. Using pieeprom-2023-05-11 with USB -> SD boot.

FEDORA-2023-7994524575 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

Fedora-Server-39-1.2.aarch64.raw.xz image boots to login prompt and I can also ssh to the system.

I tested two different brands/sizes of mSD cards several times.

I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.

That xHCI error sounds exactly like what I fixed here: https://github.com/AsahiLinux/u-boot/commit/8079fe5d87267868381dd1928c70e3d95042f7da

Have the people hitting the bug mentioned what USB devices they have connected? This happens (sometimes, randomly, it's a timing thing) when u-boot tries to stop an endpoint, which usually means something went wrong already, which can be triggered by various other u-boot bugs with other devices.

Honestly though, you'll probably want to try pulling in everything USB-related from our branch and see if that fixes all this pain, because USB is and always has been very broken in u-boot and I fixed like 15 or so bugs here: https://github.com/AsahiLinux/u-boot/commits/asahi-releng

> Have the people hitting the bug mentioned what USB devices they have
> connected? This happens (sometimes, randomly, it's a timing thing) when
> u-boot tries to stop an endpoint, which usually means something went wrong
> already, which can be triggered by various other u-boot bugs with other
> devices.

No reports, I've seen it with a couple of various upstream points when doing a bisect both with (a Logitech USB dongle) and without devices but it seems to be gone with 23.10 GA.

> Honestly though, you'll probably want to try pulling in everything
> USB-related from our branch and see if that fixes all this pain, because USB
> is and always has been very broken in u-boot and I fixed like 15 or so bugs
> here: https://github.com/AsahiLinux/u-boot/commits/asahi-releng

For the moment we've rolled back to 2023.07, it was more straight forward at this point rather than the game of whack-a-mole that .10 has proven to be.

Do you have a plan around upstreaming them, I don't particularly have an urge to carry a lot of non upstreamed patches for an indeterminate amount of time.

I tried booting about 50 times but couldn't reproduce the bug with uboot-images-armv8-2023.10-0.4.rc3.fc39.noarch. I also tried about 10 times with uboot-images-armv8-2023.10-0.8.fc40.noarch and still no repro.

If I'm right, the bug is a rare race and likely depends on a bunch of random things including possibly clock drift, which is why we're having so much trouble reproing. The best I can offer is this stripped down patch series on top of 2023.10-rc3. If I'm right about the root cause here, this should fix it: https://github.com/AsahiLinux/u-boot/tree/fedora/usb-fixes

I do intend to upstream them but it's on my backburner since right now my hands are full with Fedora Asahi stuff :)

FWIW 2023.07 is equally broken, it probably just doesn't hit the race or only hits it for other people since it is, after all, a race and anything can change the timing.

> FWIW 2023.07 is equally broken, it probably just doesn't hit the race or
> only hits it for other people since it is, after all, a race and anything
> can change the timing.

Yes, I'm aware of a number of issues with 2023.07 even outside of USB, ATM a bunch of stuff upstream is going on and causing problems, as it stands at the moment I think 2023.07 is the most straight forward solution to unblock the Fedora release. I'm actively working with upstream on a bunch of stuff to hopefully solve some of the core issues we've seen this cycle.

> I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.

Hristo, this rather concerns me. Are you saying https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main reason we did the revert was to fix that...this one seems less important.

(In reply to Adam Williamson from comment #26)
> > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> 
> Hristo, this rather concerns me. Are you saying
> https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> reason we did the revert was to fix that...this one seems less important.

Yes, unfortunately https://bugzilla.redhat.com/show_bug.cgi?id=2241252 is there, I just tested it again.

(In reply to Adam Williamson from comment #26)
> > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> 
> Hristo, this rather concerns me. Are you saying
> https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> reason we did the revert was to fix that...this one seems less important.

It is working across my RPi4 devices. Can you post the single line output you get from "dmesg|grep DMI"

(In reply to Peter Robinson from comment #28)
> (In reply to Adam Williamson from comment #26)
> > > I still need to add a nomodeset parameter to arm-image-installer to show boot messages on the screen, but I prefer it anyway.
> > 
> > Hristo, this rather concerns me. Are you saying
> > https://bugzilla.redhat.com/show_bug.cgi?id=2241252 isn't fixed? The main
> > reason we did the revert was to fix that...this one seems less important.
> 
> It is working across my RPi4 devices. Can you post the single line output
> you get from "dmesg|grep DMI"

Barging in here, since I have a 4B that apparently still `needs nomodeset`.

[    0.038253] DMI: raspberrypi,4-model-b Raspberry Pi 4 Model B Rev 1.1/Raspberry Pi 4 Model B Rev 1.1, BIOS 2023.07 07/01/2023

Sorry, let's keep discussion of that bug in https://bugzilla.redhat.com/show_bug.cgi?id=2241252 . That's the correct bug. This one is something else. I just mentioned it because Hristo mentioned in passing that he needed nomodeset and it made me worry. We should do the follow-up in the other bug.

Given https://bugzilla.redhat.com/show_bug.cgi?id=2244305#c20 , I'm gonna mark *this* bug as VERIFIED. Other issues can be followed up in other reports.

FYI I just sent the xHCI part of the USB fixes upstream.

FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-7994524575 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-7994524575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

As follow up of our release-blocker-meeting discussion I post my test environment here as well.
Otherwise see https://bugzilla.redhat.com/show_bug.cgi?id=2246871

(1)
My test environment

# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Server Edition)"
ID=fedora
VERSION_ID=39

# xz  -t Fedora-Server-39-1.2.aarch64.raw.xz 

(no probs detected)

# rpm -q arm-image-installer 
arm-image-installer-3.9-2.fc39.noarch



(2) Write image to disk

arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset"  --media=/dev/mmcblk0  --debug 

[root@rbox assets]# arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset"  --media=/dev/mmcblk0  --debug  
+ shift
+ '[' 0 -gt 0 ']'
++ whoami
+ '[' root '!=' root ']'
+ '[' /dev/mmcblk0 = '' ']'
++ mount
++ grep 'on / '
++ awk '{printf $1"\n"}'
+ ROOTDISK=/dev/mapper/sysvg-root
+ case "$ROOTDISK" in
+ '[' /dev/mmcblk0 = /dev/mapper/sysvg-root ']'
+ '[' '' = cubietruck ']'
+ '[' '' '!=' '' ']'
+ '[' '!' -f Fedora-Server-39-1.2.aarch64.raw.xz ']'
+ '[' '!' -e /dev/mmcblk0 ']'
+ echo ''

+ echo =====================================================
=====================================================
+ '[' -b /dev/fedora/root ']'
+ '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']'
+ echo '= Selected Image:                                 '
= Selected Image:                                 
+ echo '= Fedora-Server-39-1.2.aarch64.raw.xz'
= Fedora-Server-39-1.2.aarch64.raw.xz
+ echo '= Selected Media : /dev/mmcblk0'
= Selected Media : /dev/mmcblk0
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ echo =====================================================
=====================================================
+ echo ' '
 
+ echo '*****************************************************'
*****************************************************
+ echo '*****************************************************'
*****************************************************
+ echo '******** WARNING! ALL DATA WILL BE DESTROYED ********'
******** WARNING! ALL DATA WILL BE DESTROYED ********
+ echo '*****************************************************'
*****************************************************
+ echo '*****************************************************'
*****************************************************
+ '[' '' '!=' 1 ']'
+ echo ' '
 
+ echo ' Type '\''YES'\'' to proceed, anything else to exit now '
 Type 'YES' to proceed, anything else to exit now 
+ echo ' '
 
+ printf '= Proceed? '
= Proceed? + read PROCEED
YES
++ echo YES
++ tr '[:lower:]' '[:upper:]'
+ '[' YES '!=' YES ']'
+ umount /dev/mmcblk0 /dev/mmcblk0p1 /dev/mmcblk0p2 /dev/mmcblk0p3
++ fdisk -l /dev/mmcblk0
++ grep LVM
+ '[' '/dev/mmcblk0p3      3328000 124735487 121407488 57.9G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ vgchange -a n ''
+ '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']'
+ echo '= Writing: '
= Writing: 
+ echo '= Fedora-Server-39-1.2.aarch64.raw.xz '
= Fedora-Server-39-1.2.aarch64.raw.xz 
+ echo '= To: /dev/mmcblk0 ....'
= To: /dev/mmcblk0 ....
+ xzcat Fedora-Server-39-1.2.aarch64.raw.xz
+ dd of=/dev/mmcblk0 oflag=direct bs=4M status=progress iflag=fullblock7507804160 bytes (7.5 GB, 7.0 GiB) copied, 364 s, 20.6 MB/s
1792+0 records in
1792+0 records out
7516192768 bytes (7.5 GB, 7.0 GiB) copied, 364.517 s, 20.6 MB/s
+ sync
+ sleep 3
+ echo '= Writing image complete!'
= Writing image complete!
+ partprobe /dev/mmcblk0
+ '[' -e /dev/mmcblk0p5 ']'
+ '[' -e /dev/mmcblk0p4 ']'
+ '[' -e /dev/mmcblk0p3 ']'
+ export FIRMPART=/dev/mmcblk0p1
+ FIRMPART=/dev/mmcblk0p1
+ BOOTPART=/dev/mmcblk0p2
+ ROOTPART=/dev/mmcblk0p3
+ PARTNUM=3
++ blkid /dev/mmcblk0p3 -s TYPE -o value
+ FS_TYPE=LVM2_member
+ '[' LVM2_member = btrfs ']'
+ '[' '' '!=' '' ']'
+ sleep 5
+ get_lvm_name
++ fdisk -l /dev/mmcblk0
++ grep LVM
+ '[' '/dev/mmcblk0p3      3328000 14680063 11352064  5.4G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ LVM_NAME=
+ '[' '' = 1 ']'
+ mkdir /tmp/boot /tmp/root /tmp/fw
+ mount /dev/mmcblk0p2 /tmp/boot
+ '[' 0 -ne 0 ']'
+ mount /dev/mmcblk0p1 /tmp/fw
+ '[' 0 -ne 0 ']'
+ '[' '' = '' ']'
+ get_lvm_name
++ grep LVM
++ fdisk -l /dev/mmcblk0
+ '[' '/dev/mmcblk0p3      3328000 14680063 11352064  5.4G 8e Linux LVM' '!=' '' ']'
++ pvs
++ grep /dev/mmcblk0
++ awk '{print $2}'
+ LVM_NAME=
+ vgchange -a y
++ grep /tmp/root /proc/mounts
+ '[' '' = '' ']'
+ '[' '' '!=' '' ']'
+ mount /dev/mmcblk0p3 /tmp/root
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
++ echo Fedora-Server-39-1.2.aarch64.raw.xz
++ grep IoT
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' = 1 ']'
+ PREFIX=/tmp/root
+ '[' '' '!=' '' ']'
+ echo '= No U-boot will be written.'
= No U-boot will be written.
+ TARGET=board
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ '[' '' = 1 ']'
+ '[' '' '!=' '' ']'
++ getenforce
+ '[' Enforcing = Disabled ']'
+ '[' '' '!=' '' ']'
+ '[' nomodeset '!=' '' ']'
+ echo '= Adding optional kernel parameters for board : '
= Adding optional kernel parameters for board : 
+ echo '= Parameter: nomodeset'
= Parameter: nomodeset
+ add_kernel_parameter nomodeset
+ '[' -f /tmp/boot/extlinux/extlinux.conf ']'
+ '[' -f /tmp/fw/EFI/fedora/grub.cfg ']'
+ sed -i 's|GRUB_CMDLINE_LINUX="|& nomodeset |' /tmp/root/etc/default/grub
sed: can't read /tmp/root/etc/default/grub: No such file or directory
+ '[' -f /tmp/fw/EFI/fedora/grubenv ']'
+ add_bls_parameter nomodeset
+ for bls in /tmp/boot/loader/entries/*.conf
+ sed -i 's|options|& nomodeset|' /tmp/boot/loader/entries/1df910c164244dc883cf45f9880ac3b9-6.5.6-300.fc39.aarch64.conf
+ '[' board = rpi4 ']'
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ sync
+ umount /tmp/root /dev/mmcblk0p2 /dev/mmcblk0p1
+ '[' '' '!=' '' ']'
+ rmdir /tmp/root /tmp/boot /tmp/fw
+ '[' '' '!=' '' ']'
+ echo ''

+ echo '= Installation Complete! Insert into the board and boot.'
= Installation Complete! Insert into the board and boot.
+ exit 0


(3)
*  Inseert into Rpi2 / 4gb and boot
*  Execute Firstboot
*  Set locale

(4) Basic system works
% ssh pb.158.125
The authenticity of host '192.168.158.125 (192.168.158.125)' can't be established.
ED25519 key fingerprint is SHA256:U4/iqK91pXk/3Q0GcfUVbguYxcWUklE7Bh9YunbDwNo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.158.125' (ED25519) to the list of known hosts.
pb.158.125's password: 
Web console: https://localhost:9090/ or https://192.168.158.125:9090/

Last login: Mon Oct 30 20:30:11 2023


$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Server Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-05-14
VARIANT="Server Edition"
VARIANT_ID=server


$ uname -a 
Linux localhost.localdomain 6.5.6-300.fc39.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct  6 19:36:57 UTC 2023 aarch64 GNU/Linux


# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0         179:0    0 59.5G  0 disk 
├─mmcblk0p1     179:1    0  600M  0 part /boot/efi
├─mmcblk0p2     179:2    0    1G  0 part /boot
└─mmcblk0p3     179:3    0  5.4G  0 part 
  └─fedora-root 253:0    0  5.4G  0 lvm  /
zram0           252:0    0  3.7G  0 disk [SWAP]

expand partition using cfdisk works

lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk0         179:0    0 59.5G  0 disk 
├─mmcblk0p1     179:1    0  600M  0 part /boot/efi
├─mmcblk0p2     179:2    0    1G  0 part /boot
└─mmcblk0p3     179:3    0 57.9G  0 part 
  └─fedora-root 253:0    0  5.4G  0 lvm  /
zram0           252:0    0  3.7G  0 disk [SWAP]



(4) LVM administration fails

# pvresize  /dev/mmcblk0p3 
    Cannot use /dev/mmcblk0p3: device is not in devices file
    0 physical volume(s) resized or updated / 0 physical volume(s) not resized

# vgs
    Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found.


In Cockpit -> storage the device section is empty.


The output of arm-image-installer is the same for my RockPro64, Roc-rk3399-pc, rock-pi-4a and rock-pi 4b+ But these models don't boot at all, of course (without u-boot)

FEDORA-2023-7994524575 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

I'm puzzled. That updated in announced in the title as 

arm-image-installer-3.9-2.fc39, bcm283x-firmware-20231017-1.ce3a0b4.fc39, & 1 more

But ii installs 3.8-2 and says inside 3.8.

And the webside says it has been pushed to stable 30 mins ago. And it is the same rpm version as distributed with rc 1.4


Please, can someone shed some light in this?

I'm puzzled by your comment. What do you mean "But ii installs 3.8-2 and says inside 3.8."?

The build in the update is https://koji.fedoraproject.org/koji/buildinfo?buildID=2313137 . That is 3.9-2. RC-1.4 has 3.9-2 in it: https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231031.0/compose/Everything/aarch64/os/Packages/a/arm-image-installer-3.9-2.fc39.noarch.rpm . Where are you saying you're finding 3.8?

I followed the Fedora Update system message in comment 36 from today 1 hour ago (this is highly up to date, I think) and got to this page:

https://bodhi.fedoraproject.org/updates/FEDORA-2023-7994524575

And if I follow the instruction

dnf upgrade --refresh --advisory=FEDORA-2023-7994524575

it gives me (on rc 1.4)

dnf upgrade --refresh --advisory=FEDORA-2023-7994524575 
Fedora 39 - x86_64                                                                        14 kB/s | 7.2 kB     00:00    
Fedora 39 openh264 (From Cisco) - x86_64                                                  11 kB/s | 989  B     00:00    
Fedora 39 - x86_64 - Updates                                                             168 kB/s |  20 kB     00:00    
Dependencies resolved.
Nothing to do.
Complete!


[root@rbox ~]# rpm -qi arm-image-installer 
Name        : arm-image-installer
Version     : 3.8
Release     : 2.fc39
...

and

[root@rbox tmp]# cat /usr/bin/arm-image-installer  | more 
#!/usr/bin/sh

# Copyright (C) 2014-2015 Red Hat Inc.
# SPDX-License-Identifier:  GPL-2.0+

# Automate Media Creation for Fedora ARM
# Current version
VERSION=3.8

# usage message
usage() {
    echo "
Usage: $(basename ${0}) <options>

	--image=IMAGE   - xz compressed image file name
	--media=DEVICE  - media device file (/dev/[sdX|mmcblkX])

you don't have updates-testing enabled. the update was only pushed stable an hour ago. it won't be in the stable repositories until tomorrow at least.

> RC-1.4 has 3.9-2 in it: https://kojipkgs.fedoraproject.org/compose/39/Fedora-39-20231031.0/compose/Everything/aarch64/os/Packages/a/arm-image-installer-3.9-2.fc39.noarch.rpm . Where are you saying you're finding 3.8?


Now I'm even more puzzled. I downloaded today in the morning Fedora-Server-dvd-x86_64-39-1.4.iso, and tested this including raid installation, VM installation etc.

And then I installed arm-image-installer and got 3.8-2

That's perfectly normal. Things you install after initial system install come from the mirror repo tree, not from wherever you installed from, so the fact you installed from RC-1.4 doesn't matter.

> you don't have updates-testing enabled. the update was only pushed stable an hour ago. it won't be in the stable repositories until tomorrow at least.

Ah, OK. I see, I have still a lot to learn about our packaging system. So I now install the version from your link and use that for testing.

for a not-yet-released release, when an update is "pushed stable" all that really means is that it gets tagged 'f39' or 'f40' or whatever, which means it will then be included in the next nightly compose.

the main repo ("fedora" for branched, "rawhide" for Rawhide) on a system running that release points to a mirror system location which is populated by those nightly composes. whenever a compose succeeds, it is "synced out" to mirrors, meaning the contents of that location are updated with the contents of the successful compose.

so, basically, when a pending release update is "pushed stable", systems will only actually see it in the main repo a few hours after the next successful nightly compose happens.

Thanks for the info. 

I just finished testing RPi 4. 

We still have the broken LVM issue, so Server Edition isn't usable on ARM SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk image issue. So I better continue with the bug I opened, I think.

> We still have the broken LVM issue, so Server Edition isn't usable on ARM
> SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk
> image issue. So I better continue with the bug I opened, I think.

I've not seen an issue with LVM on the server images I've been testing on a Rock960. Can we have a dedicated bug for this to try and get to the bottom of the issue you're seeing.

(In reply to Peter Robinson from comment #46)
> > We still have the broken LVM issue, so Server Edition isn't usable on ARM
> > SBCs at all with rc 1.4. It's not an arm-image-installer issue but and disk
> > image issue. So I better continue with the bug I opened, I think.
> 
> I've not seen an issue with LVM on the server images I've been testing on a
> Rock960. 

How did you get that device working? With my RockPro64 I get:

> # arm-image-installer --image=Fedora-Server-39-1.5.aarch64.raw.xz --target=rockpro64-rk3399  --media=/dev/mmcblk0
>
> =====================================================
> = Selected Image:                                 
> = Fedora-Server-39-1.5.aarch64.raw.xz
> = Selected Media : /dev/mmcblk0
> = U-Boot Target : rockpro64-rk3399
> =====================================================
> .....
> = Writing image complete!
> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> = No U-Boot files found for rockpro64-rk3399.
> 
> = Installation Complete! Insert into the rockpro64-rk3399 and boot.


And without U-Boot files, it doesn't boot at all.


Well, I saw the issue, Paul Whalen found the cause (https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c9) and a possible solution https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871#c11), I tested the fix (deleting the content of etc/lvm/device) and it worked. 

As it looks to me at the moment, we just need someone with commit privilege to the kickstart repo to grep the arm image kickstart file and add a 'rm -f /etc/lvm/devices' and 'rm -f /etc/lvm/cache' to the clean up section at the end and it should work.

 And in the aftermath, we can try to figure out why this happened in F39 and not in F38 and before.



> Can we have a dedicated bug for this to try and get to the bottom
> of the issue you're seeing.

At the moment we have https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 The title is fine, but the component is wrong.

I can open a new bug, but I don't know which component to select. I found nothing suitable on the list.

Please don't open any more bugs. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2246871 is fine. We can discuss this issue there.