If you use arm-installer to create a SD disk for a specific SBC (tested with Raspberry Pi 4, Pine64 RockPro, Radxa Pi 4+. Librecomputer Doc-rk3399-pc) you get the error message: = Writing image complete! mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. and no u-boot files are written to the disk. Only the Raspberry Pi 4 is able to boot. But if you try to expand the file system to the actual disk size, resizing the partition with e.g. cfdisk works fine, but resizing the the VG fails: $ sudo pvresize /dev/mmcblk0p3 [sudo] password for pb: Cannot use /dev/mmcblk0p3: device is not in devices file 0 physical volume(s) resized or updated / 0 physical volume(s) not resized The partitioning seems OK: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 59.5G 0 disk ├─mmcblk0p1 179:1 0 600M 0 part /boot/efi ├─mmcblk0p2 179:2 0 1G 0 part /boot └─mmcblk0p3 179:3 0 57.9G 0 part └─fedora-root 253:0 0 5.4G 0 lvm / Reproducible: Always Steps to Reproduce: 1. Create a disk boot disk using arm-image-installer 2. Put the CD card into the device and power on 3. Actual Results: Only the Raspberry Pi is able to boot, the other doesn't even boot. The Raspberry LVM configuration is broken, so you can't expand the minimal root file system to the required size. Expected Results: Every device should be able to boot with a correct and maintainable LVM configuration. The Server Edition is unusable at all. Therefore, this is likely to be a release blocker.
Please test the latest arm-image-installer
Does the raspberry pi or the rockpro64 boot if you just write out the image with dd?
> Please test the latest arm-image-installer I used arm-image-installer-3.9-2.fc39.noarch.rpm With Raspi I get: ================= # arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset" --media=/dev/mmcblk0 ===================================================== ************************************************** = NOTE: This host system uses the same VG name as = the AArch64 disk image. To avoid issues, the VG = on the image will be renamed to 'fedora-server'. ************************************************** = Selected Image: = Fedora-Server-39-1.2.aarch64.raw.xz = Selected Media : /dev/mmcblk0 ===================================================== ***************************************************** ***************************************************** ******** WARNING! ALL DATA WILL BE DESTROYED ******** ***************************************************** ***************************************************** Type 'YES' to proceed, anything else to exit now = Proceed? YES = Writing: = Fedora-Server-39-1.2.aarch64.raw.xz = To: /dev/mmcblk0 .... 7511998464 Bytes (7,5 GB, 7,0 GiB) kopiert, 371 s, 20,2 MB/s 1792+0 Datensätze ein 1792+0 Datensätze aus 7516192768 Bytes (7,5 GB, 7,0 GiB) kopiert, 371,378 s, 20,2 MB/s = Writing image complete! No command with matching syntax recognised. Run 'vgrename --help' for more information. mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. = No U-boot will be written. = Adding optional kernel parameters for board : = Parameter: nomodeset sed: /tmp/root/etc/default/grub kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden = Installation Complete! Insert into the board and boot. The system started to boot and then stopped with [ OK] Reached target basic.target [ 13.594896] dracut-initqueue[641]: Volume group "fedoira-server" not found [ 13.595294] dracut-initqueue[641]: Can not process volume group fedora-server With Pine64 RockPro I got the same error messages. > Does the raspberry pi or the rockpro64 boot if you just write out the image with dd? It does, bat I ran into the nomodeset issue. And I could manage to mount the LVM properly to add an ssh key successfully, yet. After fiddling a lot with the image and the generated disk, I think the issue is not the disk image but the arm-image-installer script.9ok0ol,ßpä handling the LVM. I'll will continue to try later after some urgent appointments here.
Using the example above 'sudo arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset" --media=/dev/sda' Works ok for me using arm-image-installer-3.9-2.fc39: = Fedora-Server-39-1.2.aarch64.raw.xz = To: /dev/sda .... 7507804160 bytes (7.5 GB, 7.0 GiB) copied, 90 s, 83.4 MB/s 1792+0 records in 1792+0 records out 7516192768 bytes (7.5 GB, 7.0 GiB) copied, 90.0547 s, 83.5 MB/s = Writing image complete! WARNING: VG name fedora is used by VGs 2YFa5V-5fbq-ZRLt-PhtJ-NK4q-yjye-Id03Rl and VuAdaw-9Fah-kfF7-U7K7-M4Rd-xZcJ-IMx34B. Fix duplicate VG names with vgrename uuid, a device filter, or system IDs. WARNING: VG name fedora is used by VGs 2YFa5V-5fbq-ZRLt-PhtJ-NK4q-yjye-Id03Rl and VuAdaw-9Fah-kfF7-U7K7-M4Rd-xZcJ-IMx34B. Fix duplicate VG names with vgrename uuid, a device filter, or system IDs. Processing VG fedora because of matching UUID 2YFa5V-5fbq-ZRLt-PhtJ-NK4q-yjye-Id03Rl Volume group "2YFa5V-5fbq-ZRLt-PhtJ-NK4q-yjye-Id03Rl" successfully renamed to "fedora-server" = No U-boot will be written. = Adding optional kernel parameters for board : = Parameter: nomodeset = Installation Complete! Insert into the board and boot. rpm -q arm-image-installer arm-image-installer-3.9-2.fc39.noarch Could you verify that you have the latest version of arm-image-installer and attach the output with '--debug' included? Thanks
pboy intended this for Final blocker consideration, so marking it as such.
As follow up of our release-blocker-meeting discussion I post my test environment (1) My test environmebt # cat /etc/os-release NAME="Fedora Linux" VERSION="39 (Server Edition)" ID=fedora VERSION_ID=39 # xz -t Fedora-Server-39-1.2.aarch64.raw.xz (no probs detected) # rpm -q arm-image-installer arm-image-installer-3.9-2.fc39.noarch (2) Write image to disk arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset" --media=/dev/mmcblk0 --debug [root@rbox assets]# arm-image-installer --image=Fedora-Server-39-1.2.aarch64.raw.xz --args "nomodeset" --media=/dev/mmcblk0 --debug + shift + '[' 0 -gt 0 ']' ++ whoami + '[' root '!=' root ']' + '[' /dev/mmcblk0 = '' ']' ++ mount ++ grep 'on / ' ++ awk '{printf $1"\n"}' + ROOTDISK=/dev/mapper/sysvg-root + case "$ROOTDISK" in + '[' /dev/mmcblk0 = /dev/mapper/sysvg-root ']' + '[' '' = cubietruck ']' + '[' '' '!=' '' ']' + '[' '!' -f Fedora-Server-39-1.2.aarch64.raw.xz ']' + '[' '!' -e /dev/mmcblk0 ']' + echo '' + echo ===================================================== ===================================================== + '[' -b /dev/fedora/root ']' + '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']' + echo '= Selected Image: ' = Selected Image: + echo '= Fedora-Server-39-1.2.aarch64.raw.xz' = Fedora-Server-39-1.2.aarch64.raw.xz + echo '= Selected Media : /dev/mmcblk0' = Selected Media : /dev/mmcblk0 + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + echo ===================================================== ===================================================== + echo ' ' + echo '*****************************************************' ***************************************************** + echo '*****************************************************' ***************************************************** + echo '******** WARNING! ALL DATA WILL BE DESTROYED ********' ******** WARNING! ALL DATA WILL BE DESTROYED ******** + echo '*****************************************************' ***************************************************** + echo '*****************************************************' ***************************************************** + '[' '' '!=' 1 ']' + echo ' ' + echo ' Type '\''YES'\'' to proceed, anything else to exit now ' Type 'YES' to proceed, anything else to exit now + echo ' ' + printf '= Proceed? ' = Proceed? + read PROCEED YES ++ echo YES ++ tr '[:lower:]' '[:upper:]' + '[' YES '!=' YES ']' + umount /dev/mmcblk0 /dev/mmcblk0p1 /dev/mmcblk0p2 /dev/mmcblk0p3 ++ fdisk -l /dev/mmcblk0 ++ grep LVM + '[' '/dev/mmcblk0p3 3328000 124735487 121407488 57.9G 8e Linux LVM' '!=' '' ']' ++ pvs ++ grep /dev/mmcblk0 ++ awk '{print $2}' + vgchange -a n '' + '[' Fedora-Server-39-1.2.aarch64.raw.xz '!=' '' ']' + echo '= Writing: ' = Writing: + echo '= Fedora-Server-39-1.2.aarch64.raw.xz ' = Fedora-Server-39-1.2.aarch64.raw.xz + echo '= To: /dev/mmcblk0 ....' = To: /dev/mmcblk0 .... + xzcat Fedora-Server-39-1.2.aarch64.raw.xz + dd of=/dev/mmcblk0 oflag=direct bs=4M status=progress iflag=fullblock7507804160 bytes (7.5 GB, 7.0 GiB) copied, 364 s, 20.6 MB/s 1792+0 records in 1792+0 records out 7516192768 bytes (7.5 GB, 7.0 GiB) copied, 364.517 s, 20.6 MB/s + sync + sleep 3 + echo '= Writing image complete!' = Writing image complete! + partprobe /dev/mmcblk0 + '[' -e /dev/mmcblk0p5 ']' + '[' -e /dev/mmcblk0p4 ']' + '[' -e /dev/mmcblk0p3 ']' + export FIRMPART=/dev/mmcblk0p1 + FIRMPART=/dev/mmcblk0p1 + BOOTPART=/dev/mmcblk0p2 + ROOTPART=/dev/mmcblk0p3 + PARTNUM=3 ++ blkid /dev/mmcblk0p3 -s TYPE -o value + FS_TYPE=LVM2_member + '[' LVM2_member = btrfs ']' + '[' '' '!=' '' ']' + sleep 5 + get_lvm_name ++ fdisk -l /dev/mmcblk0 ++ grep LVM + '[' '/dev/mmcblk0p3 3328000 14680063 11352064 5.4G 8e Linux LVM' '!=' '' ']' ++ pvs ++ grep /dev/mmcblk0 ++ awk '{print $2}' + LVM_NAME= + '[' '' = 1 ']' + mkdir /tmp/boot /tmp/root /tmp/fw + mount /dev/mmcblk0p2 /tmp/boot + '[' 0 -ne 0 ']' + mount /dev/mmcblk0p1 /tmp/fw + '[' 0 -ne 0 ']' + '[' '' = '' ']' + get_lvm_name ++ grep LVM ++ fdisk -l /dev/mmcblk0 + '[' '/dev/mmcblk0p3 3328000 14680063 11352064 5.4G 8e Linux LVM' '!=' '' ']' ++ pvs ++ grep /dev/mmcblk0 ++ awk '{print $2}' + LVM_NAME= + vgchange -a y ++ grep /tmp/root /proc/mounts + '[' '' = '' ']' + '[' '' '!=' '' ']' + mount /dev/mmcblk0p3 /tmp/root mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. ++ echo Fedora-Server-39-1.2.aarch64.raw.xz ++ grep IoT + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' = 1 ']' + '[' '' = 1 ']' + PREFIX=/tmp/root + '[' '' '!=' '' ']' + echo '= No U-boot will be written.' = No U-boot will be written. + TARGET=board + '[' '' '!=' '' ']' + '[' '' = 1 ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + '[' '' = 1 ']' + '[' '' '!=' '' ']' ++ getenforce + '[' Enforcing = Disabled ']' + '[' '' '!=' '' ']' + '[' nomodeset '!=' '' ']' + echo '= Adding optional kernel parameters for board : ' = Adding optional kernel parameters for board : + echo '= Parameter: nomodeset' = Parameter: nomodeset + add_kernel_parameter nomodeset + '[' -f /tmp/boot/extlinux/extlinux.conf ']' + '[' -f /tmp/fw/EFI/fedora/grub.cfg ']' + sed -i 's|GRUB_CMDLINE_LINUX="|& nomodeset |' /tmp/root/etc/default/grub sed: can't read /tmp/root/etc/default/grub: No such file or directory + '[' -f /tmp/fw/EFI/fedora/grubenv ']' + add_bls_parameter nomodeset + for bls in /tmp/boot/loader/entries/*.conf + sed -i 's|options|& nomodeset|' /tmp/boot/loader/entries/1df910c164244dc883cf45f9880ac3b9-6.5.6-300.fc39.aarch64.conf + '[' board = rpi4 ']' + '[' '' '!=' '' ']' + '[' '' '!=' '' ']' + sync + umount /tmp/root /dev/mmcblk0p2 /dev/mmcblk0p1 + '[' '' '!=' '' ']' + rmdir /tmp/root /tmp/boot /tmp/fw + '[' '' '!=' '' ']' + echo '' + echo '= Installation Complete! Insert into the board and boot.' = Installation Complete! Insert into the board and boot. + exit 0 (3) * Inseert into Rpi2 / 4gb and boot * Execute Firstboot * Set locale (4) Basic system works % ssh pb.158.125 The authenticity of host '192.168.158.125 (192.168.158.125)' can't be established. ED25519 key fingerprint is SHA256:U4/iqK91pXk/3Q0GcfUVbguYxcWUklE7Bh9YunbDwNo. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.158.125' (ED25519) to the list of known hosts. pb.158.125's password: Web console: https://localhost:9090/ or https://192.168.158.125:9090/ Last login: Mon Oct 30 20:30:11 2023 $ cat /etc/os-release NAME="Fedora Linux" VERSION="39 (Server Edition)" ID=fedora VERSION_ID=39 VERSION_CODENAME="" PLATFORM_ID="platform:f39" PRETTY_NAME="Fedora Linux 39 (Server Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:39" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=39 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=39 SUPPORT_END=2024-05-14 VARIANT="Server Edition" VARIANT_ID=server $ uname -a Linux localhost.localdomain 6.5.6-300.fc39.aarch64 #1 SMP PREEMPT_DYNAMIC Fri Oct 6 19:36:57 UTC 2023 aarch64 GNU/Linux # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 59.5G 0 disk ├─mmcblk0p1 179:1 0 600M 0 part /boot/efi ├─mmcblk0p2 179:2 0 1G 0 part /boot └─mmcblk0p3 179:3 0 5.4G 0 part └─fedora-root 253:0 0 5.4G 0 lvm / zram0 252:0 0 3.7G 0 disk [SWAP] expand partition using cfdisk works lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 59.5G 0 disk ├─mmcblk0p1 179:1 0 600M 0 part /boot/efi ├─mmcblk0p2 179:2 0 1G 0 part /boot └─mmcblk0p3 179:3 0 57.9G 0 part └─fedora-root 253:0 0 5.4G 0 lvm / zram0 252:0 0 3.7G 0 disk [SWAP] (4) LVM administration fails # pvresize /dev/mmcblk0p3 Cannot use /dev/mmcblk0p3: device is not in devices file 0 physical volume(s) resized or updated / 0 physical volume(s) not resized # vgs Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found. In Cockpit -> storage the device section is empty. The output of arm-image-installer is the same for my RockPro64, Roc-rk3399-pc, rock-pi-4a and rock-pi 4b+ But these models don't boot at all, of course (without u-boot) I would be really glad if you find I made a mistake and everything is fine. The Server test box is an i3-5th gen, 16gb, 2 HDs that I use for all my Fedora Server test actions.
The script renames the VG 'fedora' to 'fedora-server' when the host system used to write the image also uses the VG 'fedora', this is to avoid issues when resizing the disk image. After renaming the VG, it edits the bls snippet and grubenv to change to the new VG so the system boots. The older version where issues were first encountered, did not chnage the VG name, but did change the bls snippet and grubenv causing the system not to boot with the error: [ 13.594896] dracut-initqueue[641]: Volume group "fedora-server" not found That was fixed in the latest version: arm-image-installer-3.9-2.fc39 and the system should now boot. I am looking at why the 'nomodeset' didn't work in the output above.
(In reply to Peter Boy from comment #6) > As follow up of our release-blocker-meeting discussion I post my test > environment > > > > (4) LVM administration fails > > # pvresize /dev/mmcblk0p3 > Cannot use /dev/mmcblk0p3: device is not in devices file > 0 physical volume(s) resized or updated / 0 physical volume(s) not > resized > > # vgs > Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on > /dev/vda3 not found. This can be reproduced without using the script: Command: xzcat Fedora-Server-39-1.2.aarch64.raw.xz | sudo dd of=/dev/sdb oflag=direct bs=4M status=progress iflag=fullblock Then boot on an AArch64 system (used a RPi4), adding 'nomodeset' Output: [root@localhost ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/fedora-root xfs 5.4G 2.6G 2.9G 48% / devtmpfs devtmpfs 4.0M 0 4.0M 0% /dev tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs tmpfs 762M 9.2M 752M 2% /run tmpfs tmpfs 1.9G 24K 1.9G 1% /tmp /dev/sda2 xfs 960M 193M 768M 21% /boot /dev/sda1 vfat 599M 35M 564M 6% /boot/efi tmpfs tmpfs 381M 4.0K 381M 1% /run/user/1000 [root@localhost ~]# pvresize /dev/sda3 Cannot use /dev/sda3: device is not in devices file 0 physical volume(s) resized or updated / 0 physical volume(s) not resized [root@localhost ~]# lvs Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found. [root@localhost ~]# vgs Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found.
Looks like this is coming from: cat /etc/lvm/devices/system.devices # LVM uses devices listed in this file. # Created by LVM command lvmdevices pid 3077 at Thu Oct 26 04:02:32 2023 VERSION=1.1.4 IDTYPE=devname IDNAME=/dev/vda3 DEVNAME=/dev/vda3 PVID=elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 PART=3
Discussed during the 2023-10-30 blocker review meeting: [0] The decision to delay the classification of this as a blocker bug was made as it is unclear if there is actually a bug here (and if so, if it is in the image or the script) as pboy and pwhalen do not get the same results. The decision is delayed until the situation can be clarified. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-10-30/f39-blocker-review.2023-10-30-16.00.txt
After changing vda to sda in /etc/lvm/devices/system.devices, lvm commands work as expected: pvs PV VG Fmt Attr PSize PFree /dev/sda3 fedora lvm2 a-- 5.41g 0 [root@localhost ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root fedora -wi-ao---- 5.41g
so...what's the executive summary here? is this a blocker? can we fix it? why did you not see the same errors in your image write, Paul? Thanks!
(In reply to Adam Williamson from comment #12) > so...what's the executive summary here? is this a blocker? can we fix it? Not a blocker, and fixable. > why did you not see the same errors in your image write, Paul? Trying to work it out. > > Thanks! Another bug should be opened for the LVM issue, that is unexpected and not related.
(In reply to Paul Whalen from comment #8) > (In reply to Peter Boy from comment #6) > > As follow up of our release-blocker-meeting discussion I post my test (In reply to Paul Whalen from comment #8) > (In reply to Peter Boy from comment #6) > > As follow up of our release-blocker-meeting discussion I post my test > > environment > ... > > > (4) LVM administration fails > > > > # pvresize /dev/mmcblk0p3 > > Cannot use /dev/mmcblk0p3: device is not in devices file > > 0 physical volume(s) resized or updated / 0 physical volume(s) not > > resized > > > > # vgs > > Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on > > /dev/vda3 not found. > > This can be reproduced without using the script: > > Command: > xzcat Fedora-Server-39-1.2.aarch64.raw.xz | sudo dd of=/dev/sdb oflag=direct > bs=4M status=progress iflag=fullblock > > Then boot on an AArch64 system (used a RPi4), adding 'nomodeset' Sorry, a probably silly question: how did you "adding #nomodeset'"? I tried half the day using nbd, guestmount, and others without success. More important, then, this part is a problem of the image and not of the script. When we make a new image, wouldn't it be easier for users to add nomodeset in the image until the issue is resolved (and leave nomodeset off when we create a new image anyway). >The script renames the VG 'fedora' to 'fedora-server' when the host system used to write the image also uses the VG 'fedora', this is to avoid issues when resizing the disk image. After renaming the VG, it edits the bls snippet and grubenv to change to the new VG so the system boots. But my testbox doesn't use the name for a VG (because of various issues while testing) # vgs VG #PV #LV #SN Attr VSize VFree server 1 2 0 wz--n- 58.03g 43.03g sysvg 1 2 0 wz--n- 20.00g 4.00g usrvg 1 2 0 wz--n- 90.11g 50.11g > The older version where issues were first encountered, did not chnage the VG name, but did change the bls snippet and grubenv causing the system not to boot with the error: > > [ 13.594896] dracut-initqueue[641]: Volume group "fedora-server" not found > > That was fixed in the latest version: arm-image-installer-3.9-2.fc39 and the system should now boot. But I used arm-image-installer 3.9.2, downloaded from Koji on Saturday, when Peter Robinson asked me to use the new version. Or can there be a newer version 3.9-2? > After changing vda to sda in /etc/lvm/devices/system.devices, lvm commands work as expected: That counts as another issue in the image, I guess.
(In reply to Paul Whalen from comment #13) > (In reply to Adam Williamson from comment #12) > > so...what's the executive summary here? is this a blocker? can we fix it? > > Not a blocker, and fixable. > I'm not familiar with the procedural things, yet. But as fas as I understand we need a new image and should not release before we have tested the new image and script successfully.
pboy: aiui, nomodeset is only needed/useful on certain raspberry pi models, so it doesn't seem to make sense to wire it into the images.
(In reply to Peter Boy from comment #14) > > Then boot on an AArch64 system (used a RPi4), adding 'nomodeset' > > Sorry, a probably silly question: how did you "adding #nomodeset'"? > I tried half the day using nbd, guestmount, and others without success. In grub as the system boots, the cursor is invisible so its a bit tricky. > > >The script renames the VG 'fedora' to 'fedora-server' when the host system used to write the image also uses the VG 'fedora', this is to avoid issues when resizing the disk image. After renaming the VG, it edits the bls snippet and grubenv to change to the new VG so the system boots. > > But my testbox doesn't use the name for a VG (because of various issues > while testing) This is fixed in the latest version: > > The older version where issues were first encountered, did not chnage the VG name, but did change the bls snippet and grubenv causing the system not to boot with the error: > > > > [ 13.594896] dracut-initqueue[641]: Volume group "fedora-server" not found > > > > That was fixed in the latest version: arm-image-installer-3.9-2.fc39 and the system should now boot. > > But I used arm-image-installer 3.9.2, downloaded from Koji on Saturday, when > Peter Robinson asked me to use the new version. Or can there be a newer > version 3.9-2? Do you still see the error: [ 13.594896] dracut-initqueue[641]: Volume group "fedora-server" not found As I understand comment#6 the kernel parameter is not added but the system booted OK. The missing kernel parameter will be fixed before release but this should not be considered a blocker. > > > > After changing vda to sda in /etc/lvm/devices/system.devices, lvm commands work as expected: > > That counts as another issue in the image, I guess. This would be a bug, but not in the arm-image-installer.
(In reply to Adam Williamson from comment #16) > pboy: aiui, nomodeset is only needed/useful on certain raspberry pi models, > so it doesn't seem to make sense to wire it into the images. Basically you are right. But what do you need on server more than kernel framebuffer? And again and again there are problems with graphics drivers. That would have been done prophylactically. But agreed, maybe a bit too radical. More important: I just tested rc 1.4 Server on RPi4. The LVM issue is still there: [root@localhost ~]# vgs Devices file PVID elqwwdUQpHfHhxMiZvaYfiJUhyU83Vr2 last seen on /dev/vda3 not found. So, probably fixable, but not fixed yet! So I think, it is a blocker.
I am still confused by the big difference between how you and Paul are talking about whatever bug remains. Paul says it's some kind of minor thing that just prevents manual LVM operations on the system, but doesn't prevent deployment and normal use. You say it's a blocker. I don't understand why there's a difference here and I have no idea whether we're supposed to hold the release for it.
The problem is, you can't change anything of the LVM. You can't resize the root log Volume, you can't add additional LV for specific serices, et. postgres, and so on. The root file system is about 6 GB now with about 3 GB used by system. So, you can't do much. I suppose, the problem is in /etc/lvm/devices/system.devices There is noted a LVN VG on vda3. I suppose, that's a left over from the image generating VM on koji. On an SBC / RPi you have no vda. Moist likely you have a mmcblk0, or sda. As a system engineer, you will 'simple' remove /etc/lvm/devices/system.devices and let the system generate a new and proper one, or fix it manually or do whatever is needed. As a system admin, specifically a new one (likely with RPi) that maybe beyond your expectations and abilities. And according to our technical specifications, we want to provide a basically working system without much needed manually adjustments. So the solution is, to generate a new disk image which takes care of removing the content of /etc/lvm/devices and the cache. All previous disk image generating processes did that. And I did it when generating the VM image. And with next or first boot LVM generates the correct information. In my (admin and user biased) opinion, we can't release a Server Edition with such deficiencies, specifically where it's so easy to fix. I understand, that Paul as a system engineer may have a different perspective.
Maybe, there is an alternative. We can grab the current image and use mountimage or another guestfs-tool's tool to manually remove the file from the image before deployment. I would have to test that. But maybe, that's a bit too q&d
Tested with rc 1.5 Same issue with LVM
If I may put my two cents in here. (In reply to Peter Boy from comment #20) > The problem is, you can't change anything of the LVM. You can't resize the > root log Volume, you can't add additional LV for specific serices, et. > postgres, and so on. The root file system is about 6 GB now with about 3 GB > used by system. So, you can't do much. Technically speaking, you can pass the --resizefs option to arm-image-installer and thus resize the root filesystem to fill media device, but that's all you can do from there. > I suppose, the problem is in /etc/lvm/devices/system.devices > > There is noted a LVN VG on vda3. I suppose, that's a left over from the > image generating VM on koji. On an SBC / RPi you have no vda. Moist likely > you have a mmcblk0, or sda. > > As a system engineer, you will 'simple' remove > /etc/lvm/devices/system.devices and let the system generate a new and proper > one, or fix it manually or do whatever is needed. > > As a system admin, specifically a new one (likely with RPi) that maybe > beyond your expectations and abilities. And according to our technical > specifications, we want to provide a basically working system without much > needed manually adjustments. I agree with that and as I mentioned in the Fedora ARM room, it's troublesome that on a freshly installed system this issue exists.
(In reply to Hristo Marinov from comment #23) > If I may put my two cents in here. You are welcome :-) > Technically speaking, you can pass the --resizefs option to > arm-image-installer > and thus resize the root filesystem to fill media device, but that's all you > can do > from there. I didn'try, but I guess that wouldn't work either. arm-image-installer gives the message mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. It wouldn't be able to perform the resize of the pv and can't mount the file system. At the end it had to perform the same steps as to enlarge the VG in the running system. And that doesn't work without a fix.
I am still confused by the varying descriptions of this bug. The title says "is not bootable", and pboy still seems to be claiming this. But pwhalen says it boots just fine. Which is true?
In Europe it's a case of "Radio Eriwan", a case of both.# How many different SBC models de we support? Let's assume 40, the RPi4, RPi3 and 38 other models. For each of these 40 models we provide a specific set of u-boot files. As part of arm-image-installer these are each copied to a special locations of the generic distribution file to make it bootable. The LVM issue prevents the arm-image-installer to copy it to the location at all. That's is the error message about LVM. So, all these models are not bootable, with the exception of Raspberry. It's u-boot files are included in the disk image during the generation process as the default value. So, if you specity an RPi as target, arm-image-installer runs inteo the same issue, but it doesn't matter, bedause the files are already there. But after booting the RPi's run into the same issue, because the LVM issue makes the LVM managing tools unusable. You can fix the issue for the RPis after boot, but not for all the other 38 models. At the end, all SBC's are unusable with f39 server edition.
(In reply to Adam Williamson from comment #25) > I am still confused by the varying descriptions of this bug. The title says > "is not bootable", and pboy still seems to be claiming this. But pwhalen > says it boots just fine. Which is true? In terms of booting Fedora-Server-39-1.5.aarch64.raw.xz the only issue I have on my Raspberry Pi 4 Model B, Rev 1.4, 8 GB is the screen one: https://bugzilla.redhat.com/show_bug.cgi?id=2246428
(In reply to Peter Boy from comment #0) > If you use arm-installer to create a SD disk for a specific SBC (tested with > Raspberry Pi 4, Pine64 RockPro, Radxa Pi 4+. Librecomputer Doc-rk3399-pc) > you get the error message: Can you provide the details of the arm-image-installer for each separate device. Where does the firmware reside on the rk3399 devices.
> Can you provide the details of the arm-image-installer for each separate device. Where does the firmware reside on the rk3399 devices. I ran the same arm-image-installer with different targets: arm-image-installer --image=Fedora-Server-39-1.5.aarch64.raw.xz --target=rockpro64-rk3399 --media=/dev/mmcblk0 arm-image-installer --image=Fedora-Server-39-1.5.aarch64.raw.xz --target=rock-pi-4-rk3399 --media=/dev/mmcblk0 (this one for both pi-4a and pi-4b+ arm-image-installer --image=Fedora-Server-39-1.5.aarch64.raw.xz --target=roc-pc-rk3399 --media=/dev/mmcblk0 And I always got the same error message at the same spot: mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. The SPI was empty on all the boards beside the Rock Pi 4a which is currently my working device and where I installed a spi module provided by armbian. I deactivated the SPI by shorting two pins as described in the manual and as I have done more often. Later I installed tow-boot on the RockPro64 SPI and Fedora 39 booted just fine without Fedora provided u-boot and no need for nomodeset, but with the same LVM issue. I couldn't use the LVM tools with the same error message as all the time.
(In reply to Peter Boy from comment #26) > In Europe it's a case of "Radio Eriwan", a case of both.# > > > How many different SBC models de we support? Let's assume 40, the RPi4, RPi3 > and 38 other models. For each of these 40 models we provide a specific set > of u-boot files. As part of arm-image-installer these are each copied to a > special locations of the generic distribution file to make it bootable. > > The LVM issue prevents the arm-image-installer to copy it to the location at > all. That's is the error message about LVM. So, all these models are not > bootable, with the exception of Raspberry. It's u-boot files are included in > the disk image during the generation process as the default value. So, if > you specity an RPi as target, arm-image-installer runs inteo the same issue, > but it doesn't matter, bedause the files are already there. The script works ok for me on two different systems (one with logical volumes and one with btrfs), when writing the image to mSD's or SSD's. I have not been able to reproduce what I see in your output, but will try to fix it. > > But after booting the RPi's run into the same issue, because the LVM issue > makes the LVM managing tools unusable. The LVM issue is not related to the script and should be tracked in its own bug. We can add an entry to common bugs to either delete "/etc/lvm/devices/system.devices" or to edit and update the storage (change /dev/vda to /dev/sda or mmcblk0 depending on the device). > > > You can fix the issue for the RPis after boot, but not for all the other 38 > models. > > At the end, all SBC's are unusable with f39 server edition. I have tested the RPi4 and Pine64 Plus with F39 Server.
> mount: /tmp/root: unknown filesystem type 'LVM2_member'. > dmesg(1) may have more information after failed mount system call. If you are using old images that you've flashed previously does your system mount them all automatically? If so you may get this error message when the dd blows it away. > Later I installed tow-boot on the RockPro64 SPI and Fedora 39 booted just > fine without Fedora provided u-boot and no need for nomodeset, but with the > same LVM issue. I couldn't use the LVM tools with the same error message as > all the time. If you're using tow-boot or some firmware on SPI flash you should be passing --target=none not an actual device. You don't want firmware written out in the case of SPI firmware. On a device with firmware on SPI can you write out the image with the following command and test to see if it boots (same image can be used on all of the devices): xzcat Fedora-Server-39-1.5.aarch64.raw.xz | sudo dd status=progress bs=4M of=/dev/mmcblk0
> At the end, all SBC's are unusable with f39 server edition. All the devices I've tested work fine.
> Basically you are right. But what do you need on server more than kernel > framebuffer? And again and again there are problems with graphics drivers. What about AI? You can't use OpenCL on the server image if booted with nomodeset and I've been testing some stuff around AI/ML on Fedora arm
I have tested Fedora-Server-39-1.5.aarch64.raw.xz on RPi4 and NanoPi NEO2, both booted and operated as expected.
To try and summarize for the go/no-go meeting: there is confusion and disagreement here over exactly what issues cause exactly what problems. As best as I can tell, pboy seems to say there are at least two distinct issues in this bug (so we should probably split them out). One is the presence of the /etc/lvm/devices/system.devices file within the root filesystem of the image, which causes problems with running LVM operations with that filesystem as the root context if the currently-running kernel does not agree with the device names specified in that file. AFAICT, everyone agrees that this problem exists and the file should not be present in the images, but I'm not sure if everyone agrees what the practical consequences of that are. Peter and Paul say the consequences are not release-blocking. pboy is also saying that he has problems booting Server images at all on non-Pi hardware. From https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/QOYOQXORFO7B2IRB7NY23EIZ5IMZHVWS/ I believe he is saying that this problem happens when he writes a Server image, using arm-image-installer, from an existing Server system - which probably means the problem happens when writing the image from an environment that has an LVM VG using the same names as the image itself. (The arm-image-installer script is aware of this problem and tries to handle it, but there may still be a bug on that codepath). I haven't tested this myself yet, but assuming it's the case, that should be fixable without respinning the compose, I hope. Peter, Paul and Geoff all say they have successfully booted F39 Server images on Pi and non-Pi hardware.
> As best as I can tell, pboy seems to say there are at least two distinct issues in this bug agreed > One is the presence of the /etc/lvm/devices/system.devices file within the root filesystem of the image, which causes problems with running LVM operations agreed > but I'm not sure if everyone agrees what the practical consequences of that are. Peter and Paul say the consequences are not release-blocking Depends If we take the ARM SBC as a seriously usable device for a Fedora Server Edition we would create a new image file without that file, to release an ARM SBC Edition of the **same quality** as for the other architectures / installation media. If we take it as a toy and a nice to have but not seriously usable, we would release the current deficient (but fixable) image. I would prefer to take it seriously! But I know, several members take it more as a "toy" and consider it is not worth investing much time on it. So leave it as a toy for the adventurous nerd. No need to bring it to the same quality level as the "real and important" installation media. I would regret that, but can live with that, too, and would adjust our documentation accordingly. > I haven't tested this myself yet, but assuming it's the case, that should be fixable without respinning the compose, I hope. Correct. If you use arm-image-installer on a Fedora Server system, you create not-bootable non-RPi images (always). If you use arm-image-installer on a Workstation live image you create a bootable non-RPi image. It is a bit fragile (you have only one shot, if you repeat the process it breaks under specific cirumstances and you have to take extra measures to make it workable again). Therefore, this is definitely not a "blocker" issue because we have a seamlessly usable workaround (workstation live image) and the issue itself is hopefully resolvable with a future update of arm-image-installer. No need for a new release image (as far as we currently know).
(In reply to Peter Robinson from comment #33) > > Basically you are right. But what do you need on server more than kernel > > framebuffer? And again and again there are problems with graphics drivers. > > What about AI? You can't use OpenCL on the server image if booted with > nomodeset and I've been testing some stuff around AI/ML on Fedora arm OK, agreed. Didn't had that in mind.
(In reply to Peter Robinson from comment #31) > > mount: /tmp/root: unknown filesystem type 'LVM2_member'. > > dmesg(1) may have more information after failed mount system call. > > If you are using old images that you've flashed previously does your system > mount them all automatically? If so you may get this error message when the > dd blows it away. > > > Later I installed tow-boot on the RockPro64 SPI and Fedora 39 booted just > > fine without Fedora provided u-boot and no need for nomodeset, but with the > > same LVM issue. I couldn't use the LVM tools with the same error message as > > all the time. > > If you're using tow-boot or some firmware on SPI flash you should be passing > --target=none not an actual device. You don't want firmware written out in > the case of SPI firmware. > > On a device with firmware on SPI can you write out the image with the > following command and test to see if it boots (same image can be used on all > of the devices): > xzcat Fedora-Server-39-1.5.aarch64.raw.xz | sudo dd status=progress bs=4M > of=/dev/mmcblk0 Peter Boy: can you confirm the outcome of this.
> But I know, several members take it more as a "toy" and consider it is not > worth investing much time on it. So leave it as a toy for the adventurous > nerd. No need to bring it to the same quality level as the "real and > important" installation media. I don't believe anyone is seeing it as a "toy", I certainly am not. It would have ideally been better that these issues were picked up in the Beta time frame but that water is now well and truly under the bridge. We have three people confirming the images are working for them on multiple devices. I have provided alternative methods for you to test to see if we can get to the bottom of the issues your seeing but I am still awaiting on your updates. Also in my testing I do not require nomodeset for my Rockchips devices, in fact adding it provides blank screens on server (arm display pipelines are quite complex and blocking a single driver often causes more issues).
(In reply to Peter Robinson from comment #39) > > But I know, several members take it more as a "toy" and consider it is not > > worth investing much time on it. So leave it as a toy for the adventurous > > nerd. No need to bring it to the same quality level as the "real and > > important" installation media. > > I don't believe anyone is seeing it as a "toy", I certainly am not. Agreed, You certainly not. But I had some discussion in this direction. > We have three people confirming the images are working for them on multiple > devices. I have provided alternative methods for you to test to see if we > can get to the bottom of the issues your seeing but I am still awaiting on > your updates. The arm-image-installer side and the image generation is now clear for me. It is a problem of LVM on Server and Workstation. And probably how to handle that in the installer script. And we may try to find out, why it worked up to F38 and breaks now. > Also in my testing I do not require nomodeset for my Rockchips devices, in > fact adding it provides blank screens on server (arm display pipelines are > quite complex and blocking a single driver often causes more issues). My RockPro64 and my Roc-3399-pc don't need nomodeset, just the Rock-Pi That happend also about 2 years or so ago and you could resolve that. I would contact you later about this. I guess (and hope) you can fix this, too.
"The arm-image-installer side and the image generation is now clear for me. It is a problem of LVM on Server and Workstation." it doesn't seem entirely clear, since pwhalen cannot reproduce any such problem with the current version of the script. (I also couldn't, testing on my old laptop which still has LVM, though I think the name may not clash there as I think the VG is called fedora-vaioz, i.e. it includes the hostname of the system).
Discussed at 2023-11-02 F39 go/no-go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting/2023-11-02/f39-final-go_no_go-meeting.2023-11-02-17.04.log.txt . This was rejected, as our best understanding at present is that nothing covered here clearly constitutes a release blocker. The consequences of the stray file on the disk image don't really violate any criteria and we can document that folks should wipe it if they need to perform operations on the VG. We will continue to investigate problems with the installer script in specific scenarios, but pwhalen could not duplicate pboy's issues running 3.9 from a system with a 'fedora' VG, and any problems we find can be fixed with updates to the script without needing to hold the release.
> My RockPro64 and my Roc-3399-pc don't need nomodeset, just the Rock-Pi What variant of the Rock-Pi it is? Do you get U-Boot firmware display output, what output, when does it start/stop?
I believe I tracked down the cause of the /etc/lvm/devices/system.devices problem and have split it out into a separate report: https://bugzilla.redhat.com/show_bug.cgi?id=2247872
(In reply to Peter Robinson from comment #38) > (In reply to Peter Robinson from comment #31) > > If you're using tow-boot or some firmware on SPI flash you should be passing > > --target=none not an actual device. You don't want firmware written out in > > the case of SPI firmware. > > > > On a device with firmware on SPI can you write out the image with the > > following command and test to see if it boots (same image can be used on all > > of the devices): > > xzcat Fedora-Server-39-1.5.aarch64.raw.xz | sudo dd status=progress bs=4M > > of=/dev/mmcblk0 > > Peter Boy: can you confirm the outcome of this. Yes it works that way, with my RockPro64 using tow-boot and on my Rock-Pi-4a using the Armbian made SPI module (on Fedora we have none for tis model and tow-boot for Rock-Pi has a bug). But, just to be complete, a run into the LVM issue and have to delete that system.devices file.
(In reply to Peter Robinson from comment #43) > > My RockPro64 and my Roc-3399-pc don't need nomodeset, just the Rock-Pi > > What variant of the Rock-Pi it is? Do you get U-Boot firmware display > output, what output, when does it start/stop? I checked it again. My Rock Pi 4a (the older model revision 1.4) booted with the Armbian SPI module directly into Fedora. I saw no EFI screen, but the message I saw, was the "plymouth line" and everything went as usual and I could log in without display issues. My newer model, the Rock Pi 4b+ with the advanced Rockchip prozessor bootet from the mSD card in to Efi, then the screen clears and in the first line "Booting 'Fedora Linux (6.5.6-300.fc39.aarch64) 39 (Server Edition)'" is displayed. Then the screen goes blank. But the system is booted and get's an IP from my DHCP router.
I tried to create SD card with F39 RC 1.5 Server, using arm-image-installer-3.9.2.fc39.noarch. The first try was from fresh F39 Server installation, arm-image-installer output has the same error as experienced by pboy: mount: /tmp/root: unknown filesystem type 'LVM2_member'. dmesg(1) may have more information after failed mount system call. Raspberry Pi 400 starts to boot, but gets stuck on dracut-initqueue error: [ 13.2541531] dracut-initqueue[6401]: Volume group "fedora-server" not found [ 13.2545611] dracut-initqueue [6401]: Cannot process volume group The second try was from F39 Workstation Live, using the same command I created SD card with F39 Server and it boots as expected on Raspberry Pi 400.
(In reply to Lukas Brabec from comment #47) > I tried to create SD card with F39 RC 1.5 Server, using > ... > The second try was from F39 Workstation Live, using the same command I > created SD card with F39 Server and it boots as expected on Raspberry Pi 400. Thanks! That the same as I reported earlier. Glad to have someone else with the same experience. There are so many divergent reports.
(In reply to Lukas Brabec from comment #47) > I tried to create SD card with F39 RC 1.5 Server, using > arm-image-installer-3.9.2.fc39.noarch. > > The first try was from fresh F39 Server installation, arm-image-installer > output has the same error as experienced by pboy: > mount: /tmp/root: unknown filesystem type 'LVM2_member'. > dmesg(1) may have more information after failed mount system call. > > Raspberry Pi 400 starts to boot, but gets stuck on dracut-initqueue error: > [ 13.2541531] dracut-initqueue[6401]: Volume group "fedora-server" not found > [ 13.2545611] dracut-initqueue [6401]: Cannot process volume group > > The second try was from F39 Workstation Live, using the same command I > created SD card with F39 Server and it boots as expected on Raspberry Pi 400. I was able to reproduce this using Fedora 39 Server, the cause is the '/etc/lvm/devices/system.devices' file. Because it's included in all server installations it prevents the detection of PV's not listed: sudo pvscan --cache /dev/sda3 pvscan[1296] /dev/sda3 excluded by devices file (checking PVID). Deleting '/etc/lvm/devices/system.devices' and rerunning worked ok. Peter, could you give that a try?
Created attachment 1998164 [details] lvm bug
Can confirm. On fresh F39 Server installation, I unxzed the F39 Server image, used guestfish to mount it and delete '/etc/lvm/devices/system.devices', finally I xzed the image and used arm-image-installer to create a SD card. F39 Server booted on Raspberry Pi 400 as expected. The LVM2_member error is still present in the output of arm-image-installer, though.
so, wait, the problem is the presence of an `/etc/lvm/devices/system.devices` on the *host* or in the *image to be written*, or both?
(In reply to Adam Williamson from comment #52) > so, wait, the problem is the presence of an > `/etc/lvm/devices/system.devices` on the *host* or in the *image to be > written*, or both? In my testing, the existence of `/etc/lvm/devices/system.devices` on the *host* (F39 server) prevents the proper detection of the PV's on the disk image after being written, and therefor cannot mount/manipulate the image. After I deleted this file on the *host*, I was able to use the script as intended to write the image. The disk image, when running on hardware (not virt as it will also use vda), causes issues running lvm commands until `/etc/lvm/devices/system.devices` is either deleted or edited(change vda->sda).
(In reply to Adam Williamson from comment #52) > so, wait, the problem is the presence of an > `/etc/lvm/devices/system.devices` on the *host* or in the *image to be > written*, or both? The test I did for comment #c51 was done on x86_64 host with Fedora Server installed using server dvd (so not deployed from server image). I deleted the file `/etc/lvm/devices/system.devices` on aarch64 server image.
I can confirm, that arm-installer fails on Server Edition F39 if the directory /etc/lvm/devices/ contains a file system.devices. The script uses the commands pvs and vgchange to manage LVM partitions. The failure of these commands is independent of the arm-installer-procedure or the created arm boot device. If you create on Server Editions f38 & f39 an additional partition of your choice and create a pv, vg and lv with xfs filesystem, both the pvs and vgchange commands fail if the /etc/lvm/devices directory contains said file and work immediately without problems if it is removed. If I look at Mill's causality conditions, arm-installer cannot be the cause of the problem but most likely LVM2 (or an underlying system component). I will create a bug report against LVM2.
Devicesfile is maintained with 'lvmdevices' commands. If the user doesn't want to use this filtering logic he can set use_devicesfile=0 in lvm.conf Also the whole lvm2 can be configured with a default for this setting - configure --with-default-use-devices-file
Well, as it became clear, the failure of arm-image-installer is not a bug in arm-image-installer, but a new feature in lvm. Therefore, we can close this bug.