Bug 2203233
| Summary: | logical volumes fail to activate after installing VM using `edge-installer` artifact | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Micah Abbott <miabbott> | |
| Component: | ignition | Assignee: | Antonio Murdaca <amurdaca> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Yi He <yih> | |
| Severity: | high | Docs Contact: | Eliane Ramos Pereira <elpereir> | |
| Priority: | high | |||
| Version: | 9.3 | CC: | amurdaca, bschmaus, dornelas, elpereir, miabbott, perobins, yih | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | 9.3 | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2211206 (view as bug list) | Environment: | ||
| Last Closed: | 2023-06-02 09:55:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2211206 | |||
| Attachments: | ||||
Created attachment 1964041 [details]
full console output showing the system booting and then ultimately hanging
I've tried using `rd.break=<arg>` a couple of ways to break into the dracut shell. Using `rd.break=pre-trigger` is too late in the cycle and you end up waiting on the start job. Using `rd.break=pre-udev` appears to be too early, as there is no devices reported by `lsblk` at that time. After breaking in with `rd.break=pre-udev`, I can load the `virtio-blk` module which lets me see the block devices configured and the LVM configuration: ``` pre-udev:/# modprobe virtio_blk [ 10.426169] virtio_blk virtio3: [vda] 41943040 512-byte logical blocks (21.5 GB/20.0 GiB) [ 10.434348] vda: vda1 vda2 vda3 pre-udev:/# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 252:0 0 20G 0 disk |-vda1 252:1 0 600M 0 part |-vda2 252:2 0 1G 0 part `-vda3 252:3 0 18.4G 0 part pre-udev:/# lvm vgdisplay --- Volume group --- VG Name rhel System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 18.41 GiB PE Size 4.00 MiB Total PE 4713 Alloc PE / Size 4713 / 18.41 GiB Free PE / Size 0 / 0 VG UUID CQevqs-nVuA-0eWJ-gEIX-nwMU-eklY-MQWXj1 pre-udev:/# lvm lvdisplay --- Logical volume --- LV Path /dev/rhel/swap LV Name swap VG Name rhel LV UUID r0C3sK-wfei-KpL2-1IE7-qutU-6Q6g-pyVxc5 LV Write Access read/write LV Creation host, time localhost.localdomain, 2023-05-11 12:21:05 +0000 LV Status NOT available LV Size 2.00 GiB Current LE 512 Segments 1 Allocation inherit Read ahead sectors auto --- Logical volume --- LV Path /dev/rhel/root LV Name root VG Name rhel LV UUID uUbdKE-lbSR-yV6E-pGWJ-5Hd9-V0yq-d8uRVd LV Write Access read/write LV Creation host, time localhost.localdomain, 2023-05-11 12:21:05 +0000 LV Status NOT available LV Size 16.41 GiB Current LE 4201 Segments 1 Allocation inherit Read ahead sectors auto ``` But the activation of the VG/LVs seems to still hang if I try to activate them manually. (I loaded some device mapper modules based on what I saw on the healthy vanilla RHEL system) ``` pre-udev:/# modprobe dm_mod pre-udev:/# modprobe dm_mirror pre-udev:/# modprobe xfs [ 106.130449] SGI XFS with ACLs, security attributes, scrub, quota, no debug enabled pre-udev:/# lsmod Module Size Used by xfs 2048000 0 libcrc32c 16384 1 xfs crc32c_intel 24576 1 dm_mirror 28672 0 dm_region_hash 24576 1 dm_mirror dm_log 24576 2 dm_region_hash,dm_mirror virtio_blk 20480 0 dm_mod 204800 2 dm_log,dm_mirror fuse 176128 0 pre-udev:/# lvm vgchange -a y rhel ``` At this point, I can't get any more information from the host and have to hard reset it. For comparison, the vanilla RHEL system (using rd.break) looks like: ``` switch_root:/# lsmod Module Size Used by xfs 2048000 1 libcrc32c 16384 1 xfs sr_mod 28672 0 cdrom 81920 1 sr_mod sg 49152 0 ahci 49152 0 crct10dif_pclmul 16384 1 libahci 53248 1 ahci crc32_pclmul 16384 0 crc32c_intel 24576 1 libata 425984 2 libahci,ahci ghash_clmulni_intel 16384 0 virtio_net 69632 0 net_failover 24576 1 virtio_net virtio_blk 20480 2 virtio_console 40960 0 failover 16384 1 net_failover serio_raw 20480 0 sunrpc 696320 1 dm_mirror 28672 0 dm_region_hash 24576 1 dm_mirror dm_log 24576 2 dm_region_hash,dm_mirror dm_mod 204800 7 dm_log,dm_mirror switch_root:/# lvm pvdisplay --- Physical volume --- PV Name /dev/vda3 VG Name rhel PV Size 18.41 GiB / not usable 2.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 4713 Free PE 0 Allocated PE 4713 PV UUID 0YOB4g-Xs5N-3kw8-PBqj-9crK-F9YX-Rnu1OD switch_root:/# lvm vgdisplay --- Volume group --- VG Name rhel System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 18.41 GiB PE Size 4.00 MiB Total PE 4713 Alloc PE / Size 4713 / 18.41 GiB Free PE / Size 0 / 0 VG UUID Gc3yff-t5fK-tTri-N7i8-CeRy-4hPU-46Ekv7 switch_root:/# lvm lvdisplay --- Logical volume --- LV Path /dev/rhel/root LV Name root VG Name rhel LV UUID rKVmOg-637P-5pmH-Lyq2-TpuZ-PXAc-R1kkPI LV Write Access read/write LV Creation host, time localhost.localdomain, 2023-05-11 12:14:08 +0000 LV Status available # open 1 LV Size 16.41 GiB Current LE 4201 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Logical volume --- LV Path /dev/rhel/swap LV Name swap VG Name rhel LV UUID N5Y2qE-XUSW-Jmoa-lUzf-7iTJ-OyJj-qfrflp LV Write Access read/write LV Creation host, time localhost.localdomain, 2023-05-11 12:14:09 +0000 LV Status available # open 0 LV Size 2.00 GiB Current LE 512 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 switch_root:/# mount none on / type rootfs (rw) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=455597,mode=755,inode64) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,size=746076k,nr_inodes=819200,mode=755,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) none on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) none on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) none on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /dev/mapper/rhel-root on /sysroot type xfs (ro,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota) rpc_pipefs on /sysroot/var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) ``` Created attachment 1964066 [details]
console output with rd.debug set for failing RHEL for Edge scenario
If someone has better knowledge of `dracut`, have a gander at the latest attachment with the `rd.debug` output. I'm at the limit of my knowledge and ability to debug further, so looking for help. Created attachment 1964067 [details]
rdsosreport from failed RHEL for Edge scenario
Created attachment 1964068 [details]
console output with rd.debug and rd.timeout=30 set on RHEL for Edge failed scenario
Right after the last comment, I discovered `rd.timeout` which let me get to a shell after the LVs failed to activate and I could collect the `rdsosreport` just reproduced, will work from there so I can definitely activate the swap and root lv once I'm in the dracut shell - by using lvm vgchange. Once difference I see (but not sure it does make sense/help) is that I have only 2 partitions on vda (1 & 2) and both lv are on vda2 (instead of 3). Still fully unsure as to what's happening. @amurdaca Hi Antonio, is there any update/fix for this bug that I can work with to verify? I've created https://github.com/fedora-iot/ignition-edge/pull/1 - I'm doing one more test before review & merge - then we need to build in c9s and you could grab the RPM from there and do the test with the kickstart. Yi, could you please just get the PR, run `make rpm` and run the tests with that overridden RPM? I usually do: git clone https://github.com/fedora-iot/ignition-edge cd ignition-edge rm -rf rpmbuild && make rpm createrepo ./rpmbuild/RPMS/x86_64 then, add this snippet to /etc/osbuild-composer/repositories/rhel-92.json { "name": "ignition-edge-override", "baseurl": "file:///home/runcom/ignition-edge/rpmbuild/RPMS/x86_64", "check_gpg": false }, restart, and test with the following: $ cat minimal92.toml name = "minimal92" description = "minimal rhel for edge blueprint" version = "1.0.0" distro = "rhel-92" $ cat minimal92.toml name = "minimal92" description = "minimal rhel for edge blueprint" version = "1.0.0" distro = "rhel-92" [runcom@rhel9 test-bp]$ cat iso92.toml name = "iso92" description = "blueprint to create RHEL for Edge ISO" version = "2.0.0" distro = "rhel-92" [[customizations.user]] name = "core" password = "$6$IuPqY...." key = "ssh-rsa AAAAB3NzaC1yc2E..." groups = ["wheel"] and finally for simplified-installer + ignition: $ cat iso92.toml name = "iso92" description = "blueprint to create RHEL for Edge ISO" version = "2.0.0" distro = "rhel-92" [[customizations.user]] name = "core" password = "$6$IuPqY...." key = "ssh-rsa AAAAB3NzaC1yc2E..." groups = ["wheel"] [runcom@rhel9 test-bp]$ cat simplified-installer.toml name = "simplified_installer" description = "A rhel-edge simplified-installer image" version = "0.0.5" packages = [] modules = [] groups = [] [customizations] installation_device = "/dev/vda" [customizations.ignition.firstboot] url = "http://192.168.122.1:8000/config.ign" I'll go ahead and create a PR for c9s, but early testing before merging the PR is highly appreciated Thanks, I will test it. First test: build container build edge-installer install with GUI selecting the disk check if it boots Second test: build edge-simplified-installer install check that ignition works with the blueprint above Verified scratch build for edge-installer, cannot reproduce this bug. My steps: 1. git clone pr branch in https://github.com/fedora-iot/ignition-edge/pull/1, build rpm package and set it as source in osbuild-composer 2. Build an edge-container image [root@yih-r92 rhel-edge]# composer-cli blueprints show minimal92 name = "minimal92" description = "minimal rhel for edge blueprint" version = "1.0.0" packages = [] modules = [] groups = [] distro = "rhel-92" 3. Build an edge-installer image [root@yih-r92 rhel-edge]# composer-cli blueprints show edgeinstaller name = "edgeinstaller" description = "A base rhel-edge installer image" version = "0.0.1" packages = [] modules = [] groups = [] distro = "" [customizations] [[customizations.user]] name = "installeruser" description = "Added by installer blueprint" password = "$6$GRmb7..." key = "ssh-rsa AAAAB3NzaC1yc2EAAA...." home = "/home/installeruser/" groups = ["wheel"] 4. Download edge-installer image, install it with GUI by just selecting the disk. 5. VM can boot without error. Verified scratch build with edge-simplified-installer, ignition works well. My steps: 1. git clone pr branch in https://github.com/fedora-iot/ignition-edge/pull/1, build rpm package and set it as source in osbuild-composer 2. Run test script https://github.com/virt-s1/rhel-edge/blob/main/ostree-ignition.sh (covered ignition for both edge-simplified-installer and edge-raw-image), passed without error. ignition has been rebuilt with the fixes and since it doesn't have a rhel-9.2.0 branch, the fix for zstream is going to go into 9.3 automatically |
Created attachment 1964037 [details] error activating the LVs in the initrd on a RHEL for Edge system After installing a VM using the `edge-installer` artifact, the system hangs indefinitely in the initrd trying to activate the logical volumes that were created during the install process. See the attached screenshot for example of problem. If I re-attempt the install using non-LVM disk layout, the problem does not occur. Additionally, if I use the `image-installer` artifact to install a vanilla RHEL system using LVM for disk layout, the problem does not occur. Reproducer: 1. Create `edge-container` artifact with the following blueprint (the customizations may not be required) ``` $ cat minimal92.toml name = "minimal92" description = "minimal rhel for edge blueprint" version = "1.0.0" distro = "rhel-92" [[customizations.directories]] path = "/etc/foobar" mode = "0755" user = "root" group = "root" ensure_parents = false [[customizations.files]] path = "/etc/foobar/hello" mode = "0644" user = "root" group = "root" data = "Hello world!" ``` 2. Load and start the `edge-container` with podman 3. Create an `edge-installer` artifact with the following blueprint ``` name = "iso92" description = "blueprint to create RHEL for Edge ISO" version = "2.0.0" distro = "rhel-92" [[customizations.user]] name = "core" password = "$6$IuPqY...." key = "ssh-rsa AAAAB3NzaC1yc2E..." groups = ["wheel"] ``` 4. Install a VM using the `edge-installer` artifact, keeping the default disk layout settings (should be LVM) 5. Observe the problem of the LVs not activating during the first boot of the installed VM. You can use the same "iso92" blueprint above to create the `image-installer` artifact for a vanilla RHEL install. As noted above, the installed VM using the `image-installer` artifact should boot successfully. This leads me to suspect something is different in the initrd for the VM that was installed using the `edge-installer` artifact. I examined the initrd in the RHEL for Edge disk image and the vanilla RHEL disk image, but did not find any obvious/unexpected differences that would explain the error.