Bug 2391231 - Fails to create boot medium for Fedora Server due to LVM handling failure
Summary: Fails to create boot medium for Fedora Server due to LVM handling failure
Keywords:
Status: ON_QA
Alias: None
Product: Fedora
Classification: Fedora
Component: arm-image-installer
Version: 43
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Paul Whalen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeException
Depends On:
Blocks: F43FinalFreezeException, FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2025-08-27 06:41 UTC by Peter Boy
Modified: 2025-10-20 17:29 UTC (History)
11 users (show)

Fixed In Version:
Clone Of:
: 2396309 (view as bug list)
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Peter Boy 2025-08-27 06:41:27 UTC
Here is the complete output:

> - - - - - - - - - - - - - - <
 # arm-image-installer --image=Fedora-Server-Host-Generic-43-20250821.n.0.aarch64.raw.xz --target=rock-pi-4-rk3399  --media=/dev/mmcblk0 

=====================================================
= Selected Image:                                 
= Fedora-Server-Host-Generic-43-20250821.n.0.aarch64.raw.xz
= Selected Media : /dev/mmcblk0
= U-Boot Target : rock-pi-4-rk3399
= Version: 5.0
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
 
 Type 'YES' to proceed, anything else to exit now 
 
= Proceed? YES

 
= Proceed? YES
= Writing: 
= Fedora-Server-Host-Generic-43-20250821.n.0.aarch64.raw.xz 
= To: /dev/mmcblk0 ....
10729029632 Byte (11 GB, 10 GiB) kopiert, 437 s, 24,5 MB/s 
2560+0 Datensätze ein
2560+0 Datensätze aus
10737418240 Byte (11 GB, 10 GiB) kopiert, 437,523 s, 24,5 MB/s
= Writing image complete!
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
= No U-Boot files found for rock-pi-4-rk3399.

= Installation Complete! Insert into the rock-pi-4-rk3399 and boot.
> - - - - - - - - - - - - - - <

The package uboot-images-armv8-1:2025.10-0.2.rc2.fc43.noarch is installed and all files are in the correct places. If I replace Server by Minimal, the arm-image-installer works without error messages (however the resulting medium is not bootable).  

Host to generate was Fedora Workstation 43 branched 08/21.

Reproducible: Always

Steps to Reproduce:
1. Install arm-image-installer and uboot-images-armv8-1:2025.10-0.2.rc2.fc43.noarch on a Fedora Workstation 43 branched
2. run the program as shown above
Actual Results:
Error messages appear, resulting medium is not bootable

Expected Results:
A bootable medium 

Additional Information:
The hard is obviously OK. I could boot an Armbian without issues.

Comment 1 Peter Robinson 2025-08-27 08:03:34 UTC
I suspect it's because the firmware is too large and we need to increase the offset of the image. Not sure where to move the bug too for that, kiwi probably.

> mount: /tmp/root: unknown filesystem type 'LVM2_member'.
>        dmesg(1) may have more information after failed mount system call.
> = No U-Boot files found rock-pi-4-rk3399.

Has this ever worked on server? It has issues with mounting up the LVM to find the find the firmware.

Comment 2 Peter Boy 2025-08-27 13:19:18 UTC
> I suspect it's because the firmware is too large and we need to increase the offset 
> of the image. Not sure where to move the bug too for that, kiwi probably.

So, would all SBC models affected or just RockChip based models?

And what would be a suitable offset? I thought that the offset was fixed in the primary boot loader? And isn't there an offset in the dd command in arm-image-installer (sorry for the questions, but I'm not at all familiar with these technical details, but would try to find a solution)?


> Has this ever worked on server? 

Yes, it had worked flawlessly with F40 / F41. I could use Fedora Workstation and Fedora Server as the host to generate the boot medium. Paul fixed the issues with LVM caused by the change in LVM device defaults. I don't know about F42, unfortunately.

Comment 3 Adam Williamson 2025-09-11 18:39:10 UTC
I feel like we ought to at least consider this as a Final blocker.

CCing Neal for the Kiwi angle.

Comment 4 Adam Williamson 2025-09-15 16:19:02 UTC
We're going to separate out the "dealing with LVM for the Server image fails" issue from the "whatever's going on with rockchips firmware size" issue. This one is going to be for the Server LVM problem. Peter (Boy) will file a separate bug for the firmware size issue.

Comment 5 Lukas Ruzicka 2025-09-15 18:57:35 UTC
Discussed at the 2025-09-15 (blocker / freeze exception) review meeting:

This is being rejected as it is specific to the Server aarch64 disk image, and that image is not release-blocking, as per https://docs.fedoraproject.org/en-US/releases/f43/blocking/. It is being accepted as a FE as it is evidently a significant bug that cannot be fully resolved with a post-release update.

https://meetbot-raw.fedoraproject.org//blocker-review_matrix_fedoraproject-org/2025-09-15/f43-blocker-review.2025-09-15-16.00.txt

Comment 6 Chris Murphy 2025-09-16 00:10:39 UTC
>       dmesg(1) may have more information after failed mount system call.

Can a dmesg be attached to the bug report? Or possibly better, use `journalctl -f -o short-monotonic --no-hostname` and then reproduce the problem, and post everything that appeared in the journal. There's both user space and kernel space components to LVM. Something's confused.

And also: pvs, vgs, lvs after imaging.

Comment 7 Peter Boy 2025-09-18 06:15:01 UTC
(In reply to Adam Williamson from comment #4)
> We're going to separate out the "dealing with LVM for the Server image
> fails" issue from the "whatever's going on with rockchips firmware size"
> issue. This one is going to be for the Server LVM problem. Peter (Boy) will
> file a separate bug for the firmware size issue.


see:
Bug #2396309 - Radxa Rock Pi 4 and Pine64 RockPro64 boards fail to boot, probably all RockChip models (edit)

Comment 8 Chris Murphy 2025-09-21 17:37:27 UTC
Was this a bug in Fedora 42? Maybe see https://github.com/fedora-arm/arm-image-installer/issues/8

Someone needs to do some basic troubleshooting to see if this is really an arm-image-installer bug or an image bug.

Pretty sure the script made it to here:
https://github.com/fedora-arm/arm-image-installer/blob/c99414d5da93ab95a7afe28e609b9db6fd3c80d7/arm-image-installer#L440

And fails here:
https://github.com/fedora-arm/arm-image-installer/blob/c99414d5da93ab95a7afe28e609b9db6fd3c80d7/arm-image-installer#L576

Since resize wasn't requested, I think somewhere between 543 and 570 there's a problem either with the script, or the image, or both.

Comment 9 Chris Murphy 2025-09-21 17:38:59 UTC
It might help to use bash -x or -v to walk through the script and see where the problem is happening, and what the confusion is.

Comment 10 Peter Boy 2025-09-21 18:09:29 UTC
I did some testing over the weekend.

Ultimately, this appears to be a follow-up error caused by a bug in the template file (the Fedora*.xz file) and/or the u-boot files. 

Depending on the Fedora release combination, a wide variety of problems can occur on the generated SD card. Sometimes the EFI partition is empty, sometimes files are present but the EFI and overload directories are missing, and other issues can arise. 

However, with the template file from releases 40 and 41, for example, F43 arm-image-installer runs, but depending on the version of the uboot files, different problems occur on the SD card.

However, with the template file from releases 40 and 41, for example, arm-image-installer works - even with the server LVM image -, but depending on the version of the uboot files, different problems occur on the SD card - with all the different templates, server, workstation, XFCE, minimal, KDE and others.

I think we should postpone this until https://bugzilla.redhat.com/show_bug.cgi?id=2396309 is resolved.

Comment 11 Ajay Ramaswamy 2025-09-22 01:06:21 UTC
(In reply to Peter Boy from comment #10)
> I did some testing over the weekend.
> 
> Ultimately, this appears to be a follow-up error caused by a bug in the
> template file (the Fedora*.xz file) and/or the u-boot files. 
> 
> Depending on the Fedora release combination, a wide variety of problems can
> occur on the generated SD card. Sometimes the EFI partition is empty,
> sometimes files are present but the EFI and overload directories are
> missing, and other issues can arise. 
> 
> However, with the template file from releases 40 and 41, for example, F43
> arm-image-installer runs, but depending on the version of the uboot files,
> different problems occur on the SD card.
> 
> However, with the template file from releases 40 and 41, for example,
> arm-image-installer works - even with the server LVM image -, but depending
> on the version of the uboot files, different problems occur on the SD card -
> with all the different templates, server, workstation, XFCE, minimal, KDE
> and others.
> 
> I think we should postpone this until
> https://bugzilla.redhat.com/show_bug.cgi?id=2396309 is resolved.

since fedora 38 on my FriendlyElec Nano Pi R5S anny combo of arm-image-installer 
has failed for me so what I have done is install u-boot on the internal emmc 
and fedora on the nvme so the uboot will not overwrite the beginning of the efi partition

I also made a hacky script to make a rescue SD card

#!/bin/bash

cd /tmp || exit 99

xzcat /srv/f42/Fedora-Minimal-42-1.1.aarch64.raw.xz > src.raw
cp ~/srv/uboot/v2025.07-4-ga7548a3ab26/u-boot-rockchip-v2025.07-4-ga7548a3ab26.bin /tmp/
dd if=/dev/zero of=/tmp/dst.raw bs=1M count=4305

sfdisk -d src.raw | sed 's/18432/32768/; s/409600/395264/;' | sfdisk /tmp/dst.raw

kpartx -a /tmp/src.raw
blkid | grep loop0 | sort > /tmp/src_blkid.txt

efs_uuid=$(blkid -s UUID -o value /dev/mapper/loop0p1)
boot_uuid=$(blkid -s UUID -o value /dev/mapper/loop0p2)
root_uuid=$(blkid -s UUID -o value /dev/mapper/loop0p3)

kpartx -a /tmp/dst.raw

mkfs.vfat /dev/mapper/loop1p1
mkfs.ext4 /dev/mapper/loop1p2
mkfs.ext4 /dev/mapper/loop1p3

mkdir -p /tmp/p/{src,dst}/{esp,boot,root}
# tree /tmp/p

mount -t vfat /dev/mapper/loop0p1 /tmp/p/src/esp
mount -t ext4 /dev/mapper/loop0p2 /tmp/p/src/boot
mount -t ext4 /dev/mapper/loop0p3 /tmp/p/src/root

mount -t vfat /dev/mapper/loop1p1 /tmp/p/dst/esp
mount -t ext4 /dev/mapper/loop1p2 /tmp/p/dst/boot
mount -t ext4 /dev/mapper/loop1p3 /tmp/p/dst/root

#rsync -pogAXtlHrDx --no-inc-recursive /tmp/p/src/esp/ /tmp/p/dst/esp/
rsync -pogtlHrDx --no-inc-recursive /tmp/p/src/esp/ /tmp/p/dst/esp/
rsync -pogAXtlHrDx --no-inc-recursive /tmp/p/src/boot/ /tmp/p/dst/boot/
rsync -pogAXtlHrDx --no-inc-recursive /tmp/p/src/root/ /tmp/p/dst/root/

sync
sync
umount /tmp/p/src/esp
umount /tmp/p/src/boot
umount /tmp/p/src/root
kpartx -d /tmp/src.raw

umount /tmp/p/dst/esp
umount /tmp/p/dst/boot
umount /tmp/p/dst/root

dosfslabel /dev/mapper/loop1p1 -i 0x7B7795E7
dosfslabel /dev/mapper/loop1p1 ESP
tune2fs -L boot -U "${boot_uuid}" /dev/mapper/loop1p2
tune2fs -L root -U "${root_uuid}" /dev/mapper/loop1p3


blkid | grep loop1 | sort > /tmp/dst_blkid.txt
kpartx -d /tmp/dst.raw

dd bs=32K seek=1 if=u-boot-rockchip-v2025.07-4-ga7548a3ab26.bin of=dst.raw conv=notrunc,fsync

exit 0

but it will only work with minimal not with the server image since that is lvm, basically
change the start sector of the esp 

sfdisk -d src.raw | sed 's/18432/32768/; s/409600/395264/;' | sfdisk /tmp/dst.raw 

and thus reduce the size of the ESP does the trick for me.

Comment 12 Peter Boy 2025-09-22 12:24:19 UTC
(In reply to Chris Murphy from comment #6)
> >       dmesg(1) may have more information after failed mount system call.
> 
> Can a dmesg be attached to the bug report? Or possibly better, use
> `journalctl -f -o short-monotonic --no-hostname` and then reproduce the
> problem, and post everything that appeared in the journal. There's both user
> space and kernel space components to LVM. Something's confused.
> 
> And also: pvs, vgs, lvs after imaging.

I tried:

< - - - - - - - - -  - - >
root@tbox:/home/pb# journalctl -f -o short-monotonic --no-hostname 
[  886.812460] kernel: audit: type=1334 audit(1758541485.160:233): prog-id=78 op=LOAD
[  886.815935] systemd[1]: Starting logrotate.service - Rotate log files...
[  886.988955] systemd[1]: logrotate.service: Deactivated successfully.
[  886.989826] systemd[1]: Finished logrotate.service - Rotate log files.
[  886.990172] audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  886.990171] audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  886.992064] kernel: audit: type=1130 audit(1758541485.340:234): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  886.992210] kernel: audit: type=1131 audit(1758541485.340:235): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  886.991172] audit: BPF prog-id=78 op=UNLOAD
[  886.993104] kernel: audit: type=1334 audit(1758541485.341:236): prog-id=78 op=UNLOAD


[ 1473.017042] kernel:  mmcblk0: p1 p2 p3
[ 1473.138338] systemd[1]: Starting fstrim.service - Discard unused blocks on filesystems from /etc/fstab...
[ 1473.141172] audit: BPF prog-id=82 op=LOAD
[ 1473.141172] audit: BPF prog-id=83 op=LOAD
[ 1473.141172] audit: BPF prog-id=84 op=LOAD
[ 1473.145033] kernel: audit: type=1334 audit(1758542071.491:268): prog-id=82 op=LOAD
[ 1473.145178] kernel: audit: type=1334 audit(1758542071.491:269): prog-id=83 op=LOAD
[ 1473.145242] kernel: audit: type=1334 audit(1758542071.491:270): prog-id=84 op=LOAD
[ 1473.147171] systemd[1]: Starting plocate-updatedb.service - Update the plocate database...

[ 1473.281583] lvm[2298]: /dev/mmcblk0p3 excluded: device is not in devices file.      <=======================

[ 1474.458131] fstrim[2296]: /var/log: 5 GiB (5368709120 bytes) trimmed on /dev/mapper/fedoraVG-var_log
[ 1474.458131] fstrim[2296]: /boot/efi: 605.7 MiB (635076608 bytes) trimmed on /dev/md125
[ 1474.458131] fstrim[2296]: /boot: 538.7 MiB (564912128 bytes) trimmed on /dev/md126
[ 1474.458131] fstrim[2296]: /: 15 GiB (16106127360 bytes) trimmed on /dev/mapper/fedoraVG-root
[ 1474.462172] audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1474.464534] kernel: audit: type=1130 audit(1758542072.812:271): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1474.462285] systemd[1]: fstrim.service: Deactivated successfully.
[ 1474.464171] audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1474.466706] kernel: audit: type=1131 audit(1758542072.814:272): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fstrim comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1474.462978] systemd[1]: Finished fstrim.service - Discard unused blocks on filesystems from /etc/fstab.
[ 1475.293968] systemd[1]: plocate-updatedb.service: Deactivated successfully.
[ 1475.294634] systemd[1]: Finished plocate-updatedb.service - Update the plocate database.
[ 1475.294172] audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plocate-updatedb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1475.294172] audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plocate-updatedb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1475.295579] systemd[1]: plocate-updatedb.service: Consumed 1.380s CPU time, 164.9M memory peak.
[ 1475.296537] kernel: audit: type=1130 audit(1758542073.644:273): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plocate-updatedb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1475.296633] kernel: audit: type=1131 audit(1758542073.644:274): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plocate-updatedb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 1475.310172] audit: BPF prog-id=82 op=UNLOAD
[ 1475.312065] kernel: audit: type=1334 audit(1758542073.660:275): prog-id=82 op=UNLOAD
[ 1481.240362] kernel: EXT4-fs (mmcblk0p2): mounted filesystem af047d3b-4a3b-458d-81e6-e8d66e9931b8 r/w with ordered data mode. Quota mode: none.
[ 1481.668484] kernel: EXT4-fs (mmcblk0p2): unmounting filesystem af047d3b-4a3b-458d-81e6-e8d66e9931b8.
[ 1481.670231] systemd[1]: tmp-boot.mount: Deactivated successfully.
[ 1481.679979] systemd[1]: tmp-fw.mount: Deactivated successfully.
< - - - - - - - - -  - - >

There seems to be a regression in handling the LVM change in a late f3x, where only partitions are considered which are included in /etc/lvm/devices/system.devices. Alternatively you have to add this on the command line. 

The arm-image-installer says:

arm-image-installer  --image=Fedora-Server-Host-Generic-43-20250920.n.0.aarch64.raw.xz  --target=rockpro64-rk3399  --media=/dev/mmcblk0 

=====================================================
= Selected Image:                                 
= Fedora-Server-Host-Generic-43-20250920.n.0.aarch64.raw.xz
= Selected Media : /dev/mmcblk0
= U-Boot Target : rockpro64-rk3399
= Version: 5.0
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
 
 Type 'YES' to proceed, anything else to exit now 
 
= Proceed? YES
= Writing: 
= Fedora-Server-Host-Generic-43-20250920.n.0.aarch64.raw.xz 
= To: /dev/mmcblk0 ....
10729029632 bytes (11 GB, 10 GiB) copied, 450 s, 23.8 MB/s 
2560+0 records in
2560+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 450.445 s, 23.8 MB/s
= Writing image complete!
mount: /tmp/root: unknown filesystem type 'LVM2_member'.
       dmesg(1) may have more information after failed mount system call.
= No U-Boot files found for rockpro64-rk3399.

= Installation Complete! Insert into the rockpro64-rk3399 and boot.


The LVM part is otherwise OK:
mmcblk0                179:0    0  29.7G  0 disk  
├─mmcblk0p1            179:1    0   500M  0 part  
├─mmcblk0p2            179:2    0  1000M  0 part  
└─mmcblk0p3            179:3    0   8.5G  0 part  
  └─systemVG-LVRoot    252:2    0   5.3G  0 lvm

Comment 13 Peter Boy 2025-09-22 12:30:04 UTC
And the same with Minimal image to eliminate the LVM issue: 


< - - - - - - - - - - - - >
root@tbox:/home/pb# arm-image-installer  --image=Fedora-Minimal-43-20250920.n.0.aarch64.raw.xz  --target=rockpro64-rk3399  --media=/dev/mmcblk0    

=====================================================
= Selected Image:                                 
= Fedora-Minimal-43-20250920.n.0.aarch64.raw.xz
= Selected Media : /dev/mmcblk0
= U-Boot Target : rockpro64-rk3399
= Version: 5.0
=====================================================
 
*****************************************************
*****************************************************
******** WARNING! ALL DATA WILL BE DESTROYED ********
*****************************************************
*****************************************************
 
 Type 'YES' to proceed, anything else to exit now 
 
= Proceed? YES
= Writing: 
= Fedora-Minimal-43-20250920.n.0.aarch64.raw.xz 
= To: /dev/mmcblk0 ....
4504682496 bytes (4.5 GB, 4.2 GiB) copied, 216 s, 20.9 MB/s4514119680 bytes (4.5 GB, 4.2 GiB) copied, 216.417 s, 20.9 MB/s

1076+1 records in
1076+1 records out
4514119680 bytes (4.5 GB, 4.2 GiB) copied, 216.457 s, 20.9 MB/s
= Writing image complete!
= Writing u-boot-rockchip.bin for rockpro64-rk3399 .... on media /dev/mmcblk0
19034+0 records in
19034+0 records out
9745408 bytes (9.7 MB, 9.3 MiB) copied, 2.50357 s, 3.9 MB/s

= Installation Complete! Insert into the rockpro64-rk3399 and boot.
< - - - - - - - - - - - - >

Mounting the EFI partition you get:

< - - - - - - - - - - - - >
root@tbox:/home/pb# mount /dev/mmcblk0p1   /mnt 
root@tbox:/home/pb# ls -al /mnt/
ls: cannot access '/mnt/'$'\002\002\002\002''??'$'\002\002''.'$'\002''?'$'\002': Input/output error
ls: cannot access '/mnt/ports AA.rch': Input/output error
ls: cannot access '/mnt/uild fla.g c': Input/output error
ls: cannot access '/mnt/GS = 0'$'\n''.'$'\024''bl': Input/output error
ls: cannot access '/mnt/tate'$'\n''.nex': Input/output error
ls: cannot access '/mnt/state.'$'\n''.'$'\n''Co': Input/output error
ls: cannot access '/mnt/eration'$'\n': Input/output error
ls: cannot access '/mnt/eded the. su': Input/output error
ls: cannot access '/mnt/ial.chi': Input/output error
ls: cannot access '/mnt/ys_pwr_d.m_s': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?O.?_': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?O.?_': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?{.??'$'\003': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\v''.'$'\001': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
ls: cannot access '/mnt/?7'$'\003''???'$'\003''?.??'$'\003': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
ls: cannot access '/mnt/?7'$'\003''???'$'\003''?.??'$'\003': Input/output error
ls: cannot access '/mnt/?'$'\a''.!?'$'\177': Input/output error
.. .
...
and so on
< - - - - - - - - - - - - >

The journal output:
< - - - - - - - - - - - - >
root@tbox:/home/pb# journalctl -f -o short-monotonic --no-hostname
[ 2574.443514] kernel: usb 1-3.2: Product: Logitech Illuminated Keyboard
[ 2574.444205] kernel: usb 1-3.2: Manufacturer: Logitech
[ 2574.450073] kernel: input: Logitech Logitech Illuminated Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.2/1-3.2:1.0/0003:046D:C318.0005/input/input15
[ 2574.530484] kernel: hid-generic 0003:046D:C318.0005: input,hidraw1: USB HID v1.11 Keyboard [Logitech Logitech Illuminated Keyboard] on usb-0000:00:14.0-3.2/input0
[ 2574.534068] kernel: input: Logitech Logitech Illuminated Keyboard Consumer Control as /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.2/1-3.2:1.1/0003:046D:C318.0006/input/input16
[ 2574.587082] kernel: hid-generic 0003:046D:C318.0006: input,hiddev96,hidraw2: USB HID v1.11 Device [Logitech Logitech Illuminated Keyboard] on usb-0000:00:14.0-3.2/input1
[ 2574.731665] systemd-logind[1276]: Watching system buttons on /dev/input/event5 (Logitech Logitech Illuminated Keyboard Consumer Control)
[ 2574.744934] systemd-logind[1276]: Watching system buttons on /dev/input/event4 (Logitech Logitech Illuminated Keyboard)
[ 2632.030107] kernel:  mmcblk0:
[ 2632.038063] kernel:  mmcblk0:


[ 3013.315053] kernel:  mmcblk0: p1 p2 p3
[ 3021.573202] kernel: EXT4-fs (mmcblk0p2): mounted filesystem 3470c5c3-9586-46e9-bb32-80bb3ceed68b r/w with ordered data mode. Quota mode: none.
[ 3021.915049] kernel: EXT4-fs (mmcblk0p3): mounted filesystem f5d08e53-1560-4bd6-aac9-380d0197519a r/w with ordered data mode. Quota mode: none.
[ 3025.047066] kernel: EXT4-fs (mmcblk0p3): unmounting filesystem f5d08e53-1560-4bd6-aac9-380d0197519a.
[ 3025.055866] systemd[1]: tmp-root.mount: Deactivated successfully.
[ 3025.057472] kernel: EXT4-fs (mmcblk0p2): unmounting filesystem 3470c5c3-9586-46e9-bb32-80bb3ceed68b.
[ 3025.066156] systemd[1]: tmp-fw.mount: Deactivated successfully.
[ 3025.066676] systemd[1]: tmp-boot.mount: Deactivated successfully.




[ 3228.803086] kernel: FAT-fs (mmcblk0p1): error, corrupted directory (invalid entries)
[ 3228.803365] kernel: FAT-fs (mmcblk0p1): Filesystem has been set read-only
[ 3228.805031] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.806035] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.807045] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.808097] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.808230] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.809055] kernel: FAT-fs (mmcblk0p1): error, invalid access to FAT (entry 0x0000d848)
[ 3228.809205] kernel: FAT-fs (mmcblk0p1): error, invalid access to FAT (entry 0x0000e9c8)
[ 3228.811025] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.812033] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid cluster chain (i_pos 0)
[ 3228.813030] kernel: FAT-fs (mmcblk0p1): error, fat_get_cluster: invalid start cluster (i_pos 0, start 0000d53e)
[ 3228.816025] kernel: FAT-fs (mmcblk0p1): error, corrupted directory (invalid entries)
[ 3228.816126] kernel: FAT-fs (mmcblk0p1): error, corrupted directory (invalid entries)
[ 3228.817019] kernel: FAT-fs (mmcblk0p1): error, corrupted directory (invalid entries)
< - - - - - - - - - - - - >

I didn't try to boot the SD card



I hope it helps.

Comment 14 Peter Robinson 2025-09-22 17:47:37 UTC
> since fedora 38 on my FriendlyElec Nano Pi R5S anny combo of
> arm-image-installer 
> has failed for me so what I have done is install u-boot on the internal emmc 
> and fedora on the nvme so the uboot will not overwrite the beginning of the
> efi partition

The firmware doesn't work on NVME, on all rockchip SoCs you can boot off 1) SPI 2) eMMC/uSD 3) a special USB recovery mode. We've only ever had enough space on the minimal images for the rockchips firmware, although they have now expanded even more. I'm not sure what would have ever worked on F-38 in this context.

Comment 15 Peter Boy 2025-10-16 18:45:08 UTC
Some minor additional info: 

arm-image-installer 4.2 which worked smoothly with Server / LVM (and the u-boot version of F40), now produces the same error with uboot 2025.10  and the Server image RC1.

The Server Image RC1 works perfectly on Pine RockPro64 if you do not equip the image with uboot (i.e. w/o parameter --target), but instead boot the device with the current Tow-Boot version (2023-7) in SPI. 

This suggests to me that the problem resides in the uboot-images-armv8 component.

This bug maybe a notabug,  but a followup of an issue with uboot-images-armv8

Comment 16 Chris Murphy 2025-10-16 19:44:12 UTC
(In reply to Peter Boy from comment #12)
[ 1474.458131] fstrim[2296]: /boot/efi: 605.7 MiB (635076608 bytes) trimmed on /dev/md125
[ 1474.458131] fstrim[2296]: /boot: 538.7 MiB (564912128 bytes) trimmed on /dev/md126

These might be trees, and therefore I'm missing the forest, but why is mdadm being used for the EFI system and boot partitions? ESP on software raid is not valid, but also surely is a misconfiguration for an image that's going on a single block device?

Comment 17 Chris Murphy 2025-10-16 19:52:38 UTC
FWIW one of the most wonderful failure modes of sdcards I've seen is when they internally go read-only. It will accept all writes without any error, no writes make it to persistent media. You can write a bunch of stuff, and then the read back is all the old stuff. Zero errors reported. Hilariously, fsck will report fixing everything. But the writes to fix it aren't persistent, so run fsck a 2nd time and it fixes the exact same things.

tl;dr - a) try a different sdcard, does the same exact problem happen? OR b) qualify the sdcard with f3.

Comment 18 Peter Boy 2025-10-17 00:27:06 UTC
> These might be trees, and therefore I'm missing the forest, but why is mdadm being used for the EFI system and boot partitions? 
> ESP on software raid is not valid, but also surely is a misconfiguration for an image that's going on a single block device?
 
Well, I don't know. But what I know, we don't use any kind of Software Raid on SBC's. On all the SBC's in production here, there is no md device at all.

We use Software Raid for boot and EFI, if we use SW raid for a system. I know, that's kind of off-label usage, but currently the only way to keep the EFI partition on various drives in sync, as far as I know. And it works for years, as long as there is not another system on the disk. And with Server there isn't.

Maybe, it's part of a standard step, that probes all possible devices as a precaution. Fedora has this kind of "probes" in several places, especially during installation.

 

> FWIW one of the most wonderful failure modes of sdcards I've seen is when they internally go read-only. It will accept all writes without any error, no writes make it to persistent media.

At first, I suspected a fault in my equipment and tried not only different SD cards, but also different devices, in case an SD card reader was defective.  I can actually rule out such an error, unfortunately.

Comment 19 Chris Murphy 2025-10-17 01:56:46 UTC
>[ 1473.138338] systemd[1]: Starting fstrim.service - Discard unused blocks on filesystems from /etc/fstab...

This service looks at fstab to know what to trim. All I'm able to go on is the log excerpt provided, and it shows fstrim running right after this service starts, on two LVM LVs and two mdadm arrays.

blkid doesn't see mdadm signatures on these files:
Fedora-Server-Host-Generic-43-1.4.aarch64.raw
Fedora-Minimal-43-1.4.aarch64.raw

And after running arm-image-installer on their xz counterparts, to a /dev/loop0 device with --target=rockpro64-rk3399, likewise no mdadm signatures. *shrug* If the kernel is confused, it's an oddly specific way to be confused, to hallucinate two md devices.

Comment 20 Chris Murphy 2025-10-17 02:02:03 UTC
OK so comment 12 maybe aren't boot logs from the SBC booting the sdcard? And are kernel messages from the system running arm-image-installer to the sdcard? In which case ignore everything I've written.

Comment 21 Fedora Update System 2025-10-18 00:07:50 UTC
FEDORA-2025-fa923d12eb (arm-image-installer-5.3-1.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-fa923d12eb

Comment 22 Fedora Update System 2025-10-18 00:10:08 UTC
FEDORA-2025-caa4ed51aa (arm-image-installer-5.3-1.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-caa4ed51aa

Comment 23 Fedora Update System 2025-10-18 01:41:51 UTC
FEDORA-2025-caa4ed51aa has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-caa4ed51aa`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-caa4ed51aa

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Fedora Update System 2025-10-18 02:15:27 UTC
FEDORA-2025-fa923d12eb has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-fa923d12eb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-fa923d12eb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 25 Peter Boy 2025-10-20 16:10:05 UTC
arm-image-installer 5.3-1.fc43 in combination with uboot-images-armv8  2025.10-1.fc43 and Fedora-Server-Host-Generic-43-1.4.aarch64.raw.xz  works flawlessly. The resulting sdcard is bootable and Server works as expected.

arm-image-installer 5.3-1.fc42 in combination with uboot-images-armv8 2025.04-2.fc42 and Fedora-Server-Host-Generic-42-1.1.aarch64.raw.xz performs without error messages, but the resulting sdcard is non bootable. Instead, the first partition (boot) is empty. This is due to writing the boot-images-armv8 2025.04-2.fc42 
If the —target parameter is omitted, a correct SD card with a boot partition is created as expected. This SD card it bootable and Server works fine using another u-boot version, e.g. tow-boot 2023-007.

From my POV the bug can be closed.

Comment 26 Peter Boy 2025-10-20 17:29:59 UTC
With the latest rebuilds:

arm-image-installer 5.3-1.fc43 in combination with uboot-images-armv8  2025.10-1.fc43 and Fedora-Server-Host-Generic-43-1.4.aarch64.raw.xz  works flawlessly. The resulting sdcard is bootable and Server works as expected.

I could not test minimal and workstation so far.

Nevertheless, I guess the bug can be closed.


Note You need to log in before you can comment on or make changes to this bug.