+++ This bug was initially created as a clone of Bug #2391231 +++ Various Fedora images fail to boot, but in a different way: Minimal image: obviously starts to boot, but no sign visible on the device Workstation image: displays the uboot Logo, but then it doesn't go any further XFCe image: boots into the uboot menue and complains Boot0000 and Boot0001 failed and cannot load any image. Relevant comments so far: --- Additional comment from Peter Robinson on 2025-08-27 08:03:34 UTC --- I suspect it's because the firmware is too large and we need to increase the offset of the image. Not sure where to move the bug too for that, kiwi probably.
clearing blocker decision metadata and re-proposing as a Final blocker, since the decision in 2391231 was specific to the Server-only LVM problem and based on the fact that Server disk image is not release-blocking. We should consider this separately, the question here is "is an issue on potentially all RockChip-based SBCs a blocker" based on our new x86_64-like, case-by-case subjective evaluation.
If fedora infrastructure is using image-builder then I had filed a bug there https://github.com/osbuild/images/issues/1578 in the current code at https://github.com/osbuild/images/blob/main/data/distrodefs/fedora/imagetypes.yaml around line 666 and 677 the 8 MiB needs to become 16 MiB
(In reply to Ajay Ramaswamy from comment #2) > If fedora infrastructure is using image-builder then I had filed a bug there > > https://github.com/osbuild/images/issues/1578 > > in the current code at > > https://github.com/osbuild/images/blob/main/data/distrodefs/fedora/ > imagetypes.yaml > > around line 666 and 677 the 8 MiB needs to become 16 MiB Thanks Ajay, I'll take a look at the bug. `image-builder` only builds the Fedora ARM Minimal and Fedora IoT images. I'll discuss with Peter if this bump is correct and if so make it happen. For the other images things would need to go through Kiwi so I'll tag @ngompa13 for those.
Some additional findings: In fact, the transfer of the RockChip u-boot files in the second part of arm-image-installer overwrites parts of the EFI partition on the SD card. If you mount the partition afterwards, the partition is either empty or filled with random characters (or binary data) and issues an input/output error. For various Allwinner boards arm-image-installer worked without complaining with Minimal image and a subsequent mounting does look OK. Due to lack of hardware I was unable to test the de facto bootability. It is at least suspicious that the server image throws the same error as the RockChip boards. This suggests that something is not OK with the generated SD card. A comparison of the current arm-image-installer file version 5 with version 4.2 from version F40, which works, revealed no changes affecting LVM. Only functions relating to ignition and WIFI were added. However, these are only executed if explicitly requested via parameters - as far as I can see. Unfortunately, the RockChip boards are the SBC reference models for IoT, CoreOS, and Server (besides Raspberry).
> In fact, the transfer of the RockChip u-boot files in the second part of > arm-image-installer overwrites parts of the EFI partition on the SD card. If > you mount the partition afterwards, the partition is either empty or filled > with random characters (or binary data) and issues an input/output error. Yes, for rockchips devices where the FW is on the same storage as the OS this has always been the case, this is not new, it has always been the case. > For various Allwinner boards arm-image-installer worked without complaining > with Minimal image and a subsequent mounting does look OK. Due to lack of > hardware I was unable to test the de facto bootability. That is because the space the AW firmware takes up is around 1Mb. > It is at least suspicious that the server image throws the same error as the > RockChip boards. This suggests that something is not OK with the generated > SD card. Or that the server image is doing something weird, what filesystem layout is /boot on the server image? > A comparison of the current arm-image-installer file version 5 with version > 4.2 from version F40, which works, revealed no changes affecting LVM. Only > functions relating to ignition and WIFI were added. However, these are only > executed if explicitly requested via parameters - as far as I can see. Correct, I suspect changes to LVM itself are the issue here. > Unfortunately, the RockChip boards are the SBC reference models for IoT, > CoreOS, and Server (besides Raspberry). Well where they support SPI flash, which is the case for all the IoT supported devices, this is a moot point because you don't need to write firmware to the OS disk (IE you use --target=none).
The description of this bug is wrong, all the Rockchips boards boot just fine with U-Boot. It's the disk offset which is wrong for devices that put the firmware on the OS disk.
So, I see the following disk offsets (space before the first partition starts) in the beta aarch64 disk images: Fedora Server starts at 0 MiB. Fedora Workstation starts at 0 MiB. Fedora IoT starts at 8 MiB. Fedora ARM Minimal starts at 8 MiB. What is the correct offset that should be in these disk images and should it be 8 MiB everywhere, or does it need to be increased to 16 MiB as reported in an upstream issue?
I think if we move to 12 or 16Mb we should be fine, it certainly won't hurt. Probably should do it across arm images.
(In reply to Simon de Vlieger from comment #7) > So, I see the following disk offsets (space before the first partition > starts) in the beta aarch64 disk images: > > Fedora Server starts at 0 MiB. > Fedora Workstation starts at 0 MiB. ImageFactory used to have a 2Mb offset, I think this needs to be fixed in kiwi
To make sure I understand this, we need the GPT table to contain an empty gap between the start of the GPT and the start of the ESP that is 16 MiB? And that's empty, unpartitioned space?
(In reply to Neal Gompa from comment #10) > To make sure I understand this, we need the GPT table to contain an empty > gap between the start of the GPT and the start of the ESP that is 16 MiB? > And that's empty, unpartitioned space? The ARM disk images contain MBR partition tables otherwise, yes there should be 16 MiB of offset before the first partition begins.
Tracked upstream: https://github.com/OSInside/kiwi/issues/2889
Pull request to add the offset: https://pagure.io/fedora-kiwi-descriptions/pull-request/219
Committed to F43 and pending a compose to take effect: https://pagure.io/fedora-kiwi-descriptions/c/c3af7ee267c8e0a0748923fc313893e45cb29615?branch=f43
AGREED AcceptedFinalBlocker Discussed at the 2025-09-22 (blocker / freeze exception) review meeting: This bug is accepted as a violation of "All release-blocking images must boot in their supported configurations" on all Rockchip SBCs for the Workstation and KDE aarch64 disk images, which are release-blocking and produced by Kiwi. https://meetbot-raw.fedoraproject.org//blocker-review_matrix_fedoraproject-org/2025-09-22/f43-blocker-review.2025-09-22-16.01.txt
(In reply to Neal Gompa from comment #10) > To make sure I understand this, we need the GPT table to contain an empty > gap between the start of the GPT and the start of the ESP that is 16 MiB? > And that's empty, unpartitioned space? MBR not GPT, and yes.
We've had composes since the change landed, now, so Peter, can you check with a recent nightly - e.g. https://kojipkgs.fedoraproject.org/compose/branched/Fedora-43-20250926.n.0/ - and see if this is fixed? Thanks.
This appears to be fixed with the Fedora KDE AArch64 image: ngompa@fedora ~/Downloads> fdisk -l Fedora-KDE-Desktop-Disk-43-20250926.n.0.aarch64.raw Disk Fedora-KDE-Desktop-Disk-43-20250926.n.0.aarch64.raw: 13.44 GiB, 14430502912 bytes, 28184576 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xfcc11eb6 Device Boot Start End Sectors Size Id Type Fedora-KDE-Desktop-Disk-43-20250926.n.0.aarch64.raw1 34816 1058815 1024000 500M 6 FAT16 Fedora-KDE-Desktop-Disk-43-20250926.n.0.aarch64.raw2 1058816 3155967 2097152 1G ea Linux extended boot Fedora-KDE-Desktop-Disk-43-20250926.n.0.aarch64.raw3 3155968 28184542 25028575 11.9G 83 Linux
This has also been applied to the ServerDisk profile: https://pagure.io/fedora-kiwi-descriptions/pull-request/221 It has been cherry-picked to F43 too and is now pending a compose to take effect: https://pagure.io/fedora-kiwi-descriptions/c/176a9d56aa6b00ec66ad1c4680ae110c07c00b88?branch=f43
I tested: arm-image-installer 5.2 uboot-images-armv8-2025.10-0.7.rc5.fc43.noarch Fedora-Minimal-43-20250927.n.0.aarch64.raw.xz Fedora-Server-Host-Generic-43-20250927.n.0.aarch64.raw.xz Fedora-Workstation-Disk-43-20250927.n.0.aarch64.raw.xz With arm-image-installer host Workstation as well as Server I got the same error message: mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. = No U-Boot files found for rockpro64-rk3399. That's a regression to versions some releases ago, where a workstation host always worked, even if this error occurred on a server host. Both the other images bootet into the Grub menu and continued with the message: *Fedora Linus (6.17.0-0.rc7.56.fc43 ... Prerelease) and then broke off. The monitor was no longer receiving a signal and the power consumption went down to a state when you power up the device without any boot medium. With all images, when mounting the partitions after flashing on the host, everything looked fine, no obvious defect.
(In reply to Peter Boy from comment #20) > I tested: > arm-image-installer 5.2 > uboot-images-armv8-2025.10-0.7.rc5.fc43.noarch Can you provide the full command line you used. > With arm-image-installer host Workstation as well as Server I got > the same error message: > mount: /tmp/root: unknown filesystem type 'LVM2_member'. > dmesg(1) may have more information after failed mount system call. Are you sure? The Workstation image doesn't use LWM2 at all so I would not expect that error. > = No U-Boot files found for rockpro64-rk3399. They're in the package, can you check the package is in the image? > That's a regression to versions some releases ago, where a workstation host > always worked, even if this error occurred on a server host. The LVM2 issue is long known for Server. LVM2 isn't used on Workstation. > Both the other images bootet into the Grub menu and continued with the > message: If there was no u-boot found for the rockpro where is it getting firmware from? I am presuming it's on the SPI flash? > *Fedora Linus (6.17.0-0.rc7.56.fc43 ... Prerelease) > > and then broke off. The monitor was no longer receiving a signal and the > power consumption went down to a state when you power up the device without > any boot medium. > > With all images, when mounting the partitions after flashing on the host, > everything looked fine, no obvious defect. You seem to be conflating a lot of unrelated issues here. The issue we need to know if it's fixed or not is whether the space at the beginning is correct.
(In reply to Peter Robinson from comment #21) > (In reply to Peter Boy from comment #20) > > I tested: > > arm-image-installer 5.2 > > uboot-images-armv8-2025.10-0.7.rc5.fc43.noarch > > Can you provide the full command line you used. I used on a Server host machine to execute arm-image-installer 1) arm-image-installer --image=Fedora-Minimal-43-20250927.n.0.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/sda 2) arm-image-installer --image=Fedora-Server-Host-Generic-43-20250927.n.0.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/sda 3) arm-image-installer --image=Fedora-Workstation-Disk-43-20250927.n.0.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/sda 4) arm-image-installer --image=Fedora-Minimal-43-20250927.n.0.aarch64.raw.xz --target=rock-pi-4-rk3399 --media=/dev/sda 5) arm-image-installer --image=Fedora-Server-Host-Generic-43-20250927.n.0.aarch64.raw.xz --target=rock-pi-4-rk3399 --media=/dev/sda 6) arm-image-installer --image=Fedora-Workstation-Disk-43-20250927.n.0.aarch64.raw.xz --target=rock-pi-4-rk3399 --media=/dev/sda 1) 3) 4) 6) resulted in a) no issues with overwriting partition 1, as it was with the previous version b) In no case was a complete boot successful; instead, the system hung after the grub kernel menu. 2 and 5 failed with a arm-image-installer error message as described. This is not an issue regarding this bug but #2391231. It is the reason why I could not test the Kiwi made images of Server. But these images had also no issues with overwriting partition 1 In previous releases, the LVM issue didn't occur if you used a machine that doesn't use LVM themselves for flashing the CD card. Therefor I tried on Workstation 7) on a Workstation host to execute arm-image-installer arm-image-installer --image=Fedora-Server-Host-Generic-43-20250927.n.0.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/mmcblk0 8) arm-image-installer --image=Fedora-Server-Host-Generic-43-20250927.n.0.aarch64.raw.xz --target=rock-pi-4-rk3399 --media=/dev/mmcblk0 But got the same error messages (of arm-image-installer). Then, in order to exclude that something Server / LVM specific causes the error in case on Minimal and Workstation I tried: 9) arm-image-installer --image=Fedora-Minimal-43-20250927.n.0.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/mmcblk0 But got the same result. > > > With arm-image-installer host Workstation as well as Server I got > > the same error message: > > mount: /tmp/root: unknown filesystem type 'LVM2_member'. > > dmesg(1) may have more information after failed mount system call. > > Are you sure? The Workstation image doesn't use LWM2 at all so I would not > expect that error. That's a missunderstanding, see above. > > > = No U-Boot files found for rockpro64-rk3399. > > They're in the package, can you check the package is in the image? root@test-f43:/home/pb# rpm -q uboot-images-armv8 uboot-images-armv8-2025.10-0.7.rc5.fc43.noarch root@test-f43:/usr/share/uboot# ls -l total 0 0 drwxr-xr-x. 2 root root 39 Sep 28 16:39 a64-olinuxino 0 drwxr-xr-x. 2 root root 39 Sep 28 16:39 a64-olinuxino-emmc .... 0 drwxr-xr-x. 2 root root 122 Sep 28 16:39 rock-4c-plus-rk3399 0 drwxr-xr-x. 2 root root 122 Sep 28 16:39 rock-4se-rk3399 0 drwxr-xr-x. 2 root root 122 Sep 28 16:39 rock-pi-4-rk3399 <======== 0 drwxr-xr-x. 2 root root 122 Sep 28 16:39 rock-pi-4c-rk3399 ... 0 drwxr-xr-x. 2 root root 91 Sep 28 16:39 rock960-rk3399 0 drwxr-xr-x. 2 root root 122 Sep 28 16:39 rockpro64-rk3399 <======== > > > That's a regression to versions some releases ago, where a workstation host > > always worked, even if this error occurred on a server host. > > The LVM2 issue is long known for Server. LVM2 isn't used on Workstation. arm-image-installer worked perfectly with LVM for several releases. > > > Both the other images bootet into the Grub menu and continued with the > > message: > > If there was no u-boot found for the rockpro where is it getting firmware > from? I am presuming it's on the SPI flash? The "other images" referred to Minimal and Workstation arm-image-installer wrote the u-boot items for those. > > > *Fedora Linus (6.17.0-0.rc7.56.fc43 ... Prerelease) > > > > and then broke off. The monitor was no longer receiving a signal and the > > power consumption went down to a state when you power up the device without > > any boot medium. > > > > With all images, when mounting the partitions after flashing on the host, > > everything looked fine, no obvious defect. > > You seem to be conflating a lot of unrelated issues here. The issue we need > to know if it's fixed or not is whether the space at the beginning is > correct. The space at the beginning is fine now. But that did not completely fixed the issue, which is "Radxa Rock Pi 4 and Pine64 RockPro64 boards fail to boot with Kiwi-produced images, probably all RockChip models" The issue in question here is not "We are missing a nice space at the beginning " At the end, the bug is not fixed yet.
For now I'm setting this back to ASSIGNED based on pboy's feedback. If pbrobinson would prefer new bug reports for the remaining issues, we can do that.
Why was the component moved as part of that comment? Adam please provide all the information.
I can provide some additional info: Today I used the rc1 version of * Fedora-Minimal-43-1.4.aarch64.raw.xz * arm-image-installer version 5.2 * uboot-images-armv8-2025.10-0.7.rc5.fc43.noarch on Pine64 RockPro64 1. arm-image-installer worked 2. the created images booted, showed the uboot selection screen, showed the grub menu and went on 3. Showed the line "Booting 'Fedora Linux .... " which is usually the first line of the Boot process messages 4. After some seconds ii crashed without any message of making any progress (power consumption reduced to idle level) after installing tow-boot onto the SPI 1. the very same SD card without any modification boots without any problems. 2. I could execute the first boot configuration and the standard post-install tasks without any problems. Unfortunately I couldn't manage to get any info from an attached serial terminal (just digital garbage).
Some minor additions: arm-image-installer 4.2 which worked smoothly with Server / LVM (and the u-boot version of F40), now produces the same error with uboot 2025.10 and the Server image RC1. The Server Image RC1 works perfectly on Pine RockPro64 if you do not equip the image with uboot (i.e. w/o parameter --target), but instead boot the device with the current Tow-Boot version (2023-7) in SPI. This suggests to me that the problem resides in the uboot-images-armv8 component.
Peter: I was guessing. Since we resolved the known kiwi issue, it didn't seem to make sense to leave it assigned to kiwi, so I just guessed at something else to assign it to.
> arm-image-installer 4.2 which worked smoothly with Server / LVM (and the > u-boot version of F40), now produces the same error with uboot 2025.10 and > the Server image RC1. You're assuming here that nothing has changed in the LVM2 sofware/configuration, which I believe is not the case, which means you can't extrapolate that. Also U-Boot knows nothing about LVM2 and doesn't care. > The Server Image RC1 works perfectly on Pine RockPro64 if you do not equip > the image with uboot (i.e. w/o parameter --target), but instead boot the > device with the current Tow-Boot version (2023-7) in SPI. Why don't you write the U-Boot to the SPI flash. > This suggests to me that the problem resides in the uboot-images-armv8 > component. Can you attach the full output of the boot including all the firmware output from power on.
> 1. arm-image-installer worked > 2. the created images booted, showed the uboot selection screen, showed the > grub menu and went on > 3. Showed the line "Booting 'Fedora Linux .... " which is usually the > first line of the Boot process messages > 4. After some seconds ii crashed without any message of making any progress > (power consumption reduced to idle level) How do you know it's 'crashed', I am presuming this is on a HDMI display? > Unfortunately I couldn't manage to get any info from an attached serial > terminal (just digital garbage). What console tool are you using, what baud rate and config?
(In reply to Adam Williamson from comment #27) > Peter: I was guessing. Since we resolved the known kiwi issue, it didn't > seem to make sense to leave it assigned to kiwi, so I just guessed at > something else to assign it to. I believe you got #2391231 created to deal with the other issues. This bug was fixed and should have been closed and left as such. If there was then other bugs *another* bug should have been opened! Now we just have a mess of cross bug conversations!
well, it's the other way around. *This* bug was forked from 2391231. 2391321 originally reported both "I have problems writing images with arm-image-installer in various scenarios" and "even if I get an image written, it won't boot on the system". We asked pboy to create this report to cover the second problem. So 2391231 should be for discussion of problems writing the images, this bug should be for discussion of problems actually booting a system, given that you have somehow managed to write an image. What pboy is now saying (AFAICT) is that, ignoring the 'writing an image' problem, the kiwi fixes have made the system get *further* when he tries to boot it, but it's still not actually booting to any usable state. For him, that's still covered by this bug, because the user experience is still "I try to boot it and it doesn't work".
(In reply to Adam Williamson from comment #31) > well, it's the other way around. *This* bug was forked from 2391231. 2391321 > originally reported both "I have problems writing images with > arm-image-installer in various scenarios" and "even if I get an image > written, it won't boot on the system". We asked pboy to create this report > to cover the second problem. So 2391231 should be for discussion of problems > writing the images, this bug should be for discussion of problems actually > booting a system, given that you have somehow managed to write an image. This bug was the one assigned to kiwi to fix the imaging offsets at the beginning. That problem is fixed as a result this bug should have been closed as part of that fix and any other bug should be a different bug.
(In reply to Adam Williamson from comment #27) > Peter: I was guessing. Since we resolved the known kiwi issue, it didn't > seem to make sense to leave it assigned to kiwi, so I just guessed at > something else to assign it to. Based on this I am assigning this back to kiwi and closing it as fixed. I have opened this bug to track the new issue: https://bugzilla.redhat.com/show_bug.cgi?id=2404742 Peter: please update that bug.
And FYI arm-image-installer has nothing to do with display output on a device
I know that. That's why this was forked from 2391231 in the first place.
With the latest rebuilds: arm-image-installer 5.3-1.fc43 in combination with uboot-images-armv8 2025.10-1.fc43 and Fedora-Server-Host-Generic-43-1.4.aarch64.raw.xz works flawlessly. The resulting sdcard is bootable and Server works as expected. I could not test minimal and workstation so far. Nevertheless, I guess the bug can be closed.