Description of problem: Running RHCOS VM with ignition failed with (from VM boot screen): systemd: emergency.service: Failed at step STDIN spawning /usr/bin/systemctl: Inappropriate ioctl for device - Observing vdsm.log seems that ignition payload path is updated to an empty path: 2020-03-08 15:30:16,858+0200 WARN (vm/7280d704) [virt.vm] (vmId='7280d704-c693-4855-bbea-54fc69ac0826') updating drive ua-1e75755e-5d16-4344-aaf9-e19ec3865725 path from /var/run/vdsm/payload/7280d704-c693-4855-bbea-54fc69ac0826.img to (storage:177) 2020-03-08 15:30:16,858+0200 WARN (vm/7280d704) [virt.vm] (vmId='7280d704-c693-4855-bbea-54fc69ac0826') updating drive ua-1e75755e-5d16-4344-aaf9-e19ec3865725 config path from /var/run/vdsm/payload/7280d704-c693-4855-bbea-54fc69ac0826.img to (storage:198) - and in domxmls the ovirt-vm:payload is empty: 2020-03-08 15:30:16,922+0200 INFO (jsonrpc/5) [api.host] FINISH dumpxmls return={'dom.... ... <ovirt-vm:payload>\n <ovirt-vm:volId>config-2</ovirt-vm:volId>\n <ovirt-vm:file path="openstack/latest/meta_data.json">ewogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIsCiAgImhvc3RuYW1lIiA6ICJSSENPU Version-Release number of selected component (if applicable): ovirt-engine-4.4.0-0.24.master.el8ev vdsm-4.40.5-1.el8ev.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 libvirt-daemon-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create new VM and attach new RHCOS qcow image to it. 2. Run VM with ignition payload. 3. Actual results: VM failed to boot (systemd: emergency.service: Failed at step STDIN spawning /usr/bin/systemctl: Inappropriate ioctl for device) Expected results: VM should boot with ignition payload configured. Additional info: engine.log, vdsm.log (VM ID 7280d704-c693-4855-bbea-54fc69ac0826), qemu.log and VM boot screenshot attached.
Created attachment 1668444 [details] engine.log
Created attachment 1668445 [details] vdsm.log
Created attachment 1668446 [details] qemu log
Created attachment 1668447 [details] VM boot screenshot
The bug seems to be unrelated straight to RHV ignition. The problem is with CD-ROM payload device. By the VDSM warning we see path switch from the image to read into empty string. Then, when we expect to see in the domxml for this device: <source file=\'/var/run/vdsm/payload/<vm-uuid>.img .. we get either without source file or in some times with <source file= .. (empty string) Here in the VDSM log(After the path change): <disk type=\'file\' device=\'cdrom\'>\n <driver name=\'qemu\' error_policy=\'report\'/>\n <target dev=\'sdb\' bus=\'sata\'/>\n <readonly/>\n <alias name=\'ua-1e75755e-5d16-4344-aaf9-e19ec3865725\'/>\n <address type=\'drive\' controller=\'0\' bus=\'0\' target=\'0\' unit=\'1\'/>\n </disk>\n We do see the payload sent in the ovirt metadata and the payload is correct (removing the ignition handling as a root cause). This also can harm any cd-rom payload related flows, such as cloud-init, sysprep(since 4.4)..
Continuing investigate, I've found out it's unrelated to engine and doesn't seem related to VDSM as well. In VDSM we get the domxml from libvirt and the warning above (path switch) is as a result from libvirt. The source tag for the disk is gone in libvirt domxml. In libvirt 5.6.0.6 it works, and not in 6.0.0.7. From the libvirt log: 2020-03-09 13:25:12.114+0000: 36262: error : virDomainDiskTranslateSourcePool:31616 : XML error: 'startupPolicy' is only valid for 'file' type volume 2020-03-09 13:25:12.114+0000: 36262: debug : qemuDomainCheckRemoveOptionalDisk:11282 : Dropping disk 'sdb' on domain 'rhcos_new' (UUID 'bfc35b9c-925b-4f43-b803-fa75b97432e7') due to inaccess ible source '/var/run/vdsm/payload/bfc35b9c-925b-4f43-b803-fa75b97432e7.img'
Created attachment 1668667 [details] libvirt comparison logs
I think this is a libvirt regression caused by: 63469116cc virDomainDiskTranslateSourcePool: split code to setup one storage source which tries to fix the bug 1804603. I will clone this bug shortly to libvirt so that we can fix it. You guys can later use this one to test the fix.
*** Bug 1808846 has been marked as a duplicate of this bug. ***
not relevant for upstream (uses libvirt-5.6.0-10) relevant for RHV on RHEL AV 8.2. Workaround is to downgrade to libvirt < 6.0.0-6
Verified: ovirt-engine-4.4.0-0.26.master.el8ev vdsm-4.40.7-1.el8ev.x86_64 libvirt-daemon-6.0.0-13.module+el8.2.0+6048+0fa476b4.x86_64 qemu-kvm-4.2.0-15.module+el8.2.0+6029+618ef2ec.x86_64
This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.