Description: v2v can't convert guest from vmware because of libvirt error 'argument key 'host' must not have null value [code=1 int1=-1]' Version-Release number of selected component (if applicable): virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libguestfs-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64 qemu-kvm-4.2.0-6.module+el8.2.0+5453+31b2b136.x86_64 How reducible: 100% Steps: 1.Convert a guest from vmware by virt-v2v # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data -b ovirtmgmt esx6.7-rhel7.7-x86_64 [ 0.6] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.7-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 2.3] Creating an overlay to protect the source from being modified [ 5.7] Opening the overlay virt-v2v: error: libguestfs error: could not create appliance through libvirt. Try running qemu directly without libvirt using this environment variable: export LIBGUESTFS_BACKEND=direct Original error from libvirt: internal error: argument key 'host' must not have null value [code=1 int1=-1] If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Actual result: As above description Expected result: v2v can't convert guest from vmware on rhel8.2 fast stream successfully Additional info: 1.Can't reproduce the problem with libvirt-5.6.0-7.module+el8.2.0+4670+07fe2774.x86_64, so it is a regression bug 2.Auto case log:https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/v2v/view/RHEL-8.2/job/v2v-RHEL-8.2-runtest-x86_64-acceptance-ovirt/18/testReport/rhel/convert_vm_to_ovirt/esx_vm_6_7_linux_latest7_arch_x86_64_raw_f_NFS_rhv_upload_rhv_direct_rhv_verifypeer_nopreallocated_it_vddk/
Created attachment 1652697 [details] XML passed to libvirt
"host" doesn't appear directly in the XML. I believe this must be a libvirt regression.
Dan has this debugging suggestion: (1) Open /etc/libvirt/libvirtd.conf (2) In this file add (eg at the end of the file): log_filters="1:qemu 1:qemu_monitor 1:json" log_outputs="1:file:/var/log/libvirt/libvirtd.log" (3) Restart the libvirtd service: systemctl restart libvirtd.service (4) Rerun the virt-v2v command which fails. (5) Check and attach the file /var/log/libvirt/libvirtd.log which should contain very verbose logging of the interactions with qemu. To reverse these steps you can remove the lines from /etc/libvirt/libvirtd.conf and restart libvirtd.
Created attachment 1652714 [details] libvirtd-log-for-comment3
An observation: There are only 3 places in the libvirt code which use "s:host" as a parameter of virJSONValueObject*: (1) qemuMonitorJSONBuildInetSocketAddress when building an NBD server connection or chardev. NBD is not used in this example, but we do use two virtio-serial connections to Unix domain sockets (ie. no host). (2) qemuBlockStorageSourceBuildJSONSocketAddress when building a connection to remote storage. In this instance all storage files are local, but the overlay file will be backed by a remote connection, but that would have a remote host name. So could be related to the virtio-serial connections? However if that was the case then simply "libguestfs-test-tool" would fail also.
After some discussion on IRC it turns out that a backing file with the following backing store string is used: nbd:unix:/tmp/sock:exportname=/ This string was parsed incorrectly by libvirt and the internal structures looked like an TCP based storage definition. Since the 'host' part was missing the JSON formatter used for -blockdev failed. This is indeed a libvirt bug in the backing store parser which started manifesting itself after the switch to -blockdev in libvirt.
https://www.redhat.com/archives/libvir-list/2020-January/msg00714.html
(In reply to Peter Krempa from comment #6) > After some discussion on IRC it turns out that a backing file with the > following backing store string is used: > > nbd:unix:/tmp/sock:exportname=/ That string should remain valid for back-compat reasons, but we should probably fix whatever generated it to instead prefer the modern URI syntax: nbd+unix:///?socket=/tmp/sock documented at https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
FYI it's generated here: https://github.com/libguestfs/virt-v2v/blob/2a0fb45f5f98f399e26926e56f45f9012cef8c84/v2v/nbdkit.ml#L351
Fixed upstream: commit 2775aada167fad5a508b16a8cadac1fca489c7be (HEAD -> master, origin/master, origin/HEAD) Author: Peter Krempa <pkrempa> Date: Thu Jan 16 13:49:37 2020 +0100 tests: virstorage: Add tests for NBD URI style syntax over UNIX Add few test cases for nbd+unix style URIs with few corner cases. The NBD URI syntax is documented at https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Eric Blake <eblake> Reviewed-by: Ján Tomko <jtomko> commit 5f2fa393f721861132500f717ce509bb66afcdb7 Author: Peter Krempa <pkrempa> Date: Thu Jan 16 12:34:13 2020 +0100 util: storagefile: Properly set transport type when parsing NBD strings When parsing legacy NBD backing file strings such as 'nbd:unix:/tmp/sock:exportname=/' we'd fail to set the transport to VIR_STORAGE_NET_HOST_TRANS_UNIX. This started to be a problem once we actually started to generate config of the backing store on the command line with -blockdev as the JSON code would try to format it as TCP and fail with: internal error: argument key 'host' must not have null value Set the type properly and add a test. This bug was found by the libguestfs test suite in: https://bugzilla.redhat.com/show_bug.cgi?id=1791614 Signed-off-by: Peter Krempa <pkrempa> Reported-by: Ming Xie <mxie> Reviewed-by: Eric Blake <eblake> Tested-by: Richard W.M. Jones <rjones>
Verify the bug with builds: virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libguestfs-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libvirt-6.0.0-2.module+el8.2.0+5513+34927b6c.x86_64 qemu-kvm-4.2.0-7.module+el8.2.0+5520+4e5817f3.x86_64 Steps: 1.Convert a guest from vmware to rhv by virt-v2v # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data -b ovirtmgmt esx6.7-rhel7.7-x86_64 [ 0.9] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.7-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA [ 2.5] Creating an overlay to protect the source from being modified [ 5.7] Opening the overlay [ 12.6] Inspecting the overlay [ 29.7] Checking for sufficient free disk space in the guest [ 29.7] Estimating space required on target for each disk [ 29.7] Converting Red Hat Enterprise Linux Server 7.7 Beta (Maipo) to run on KVM virt-v2v: warning: guest tools directory ‘linux/el7’ is missing from the virtio-win directory or ISO. Guest tools are only provided in the RHV Guest Tools ISO, so this can happen if you are using the version of virtio-win which contains just the virtio drivers. In this case only virtio drivers can be installed in the guest, and installation of Guest Tools will be skipped. virt-v2v: This guest has virtio drivers installed. [ 120.4] Mapping filesystem data to avoid copying unused and blank areas [ 120.8] Closing the overlay [ 121.0] Assigning disks to buses [ 121.0] Checking if the guest needs BIOS or UEFI to boot [ 121.0] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 122.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.baq7Kk/nbdkit0.sock", "file.export": "/" } (raw) (100.00/100%) [ 811.4] Creating output metadata [ 813.0] Finishing off 2.Check the guest after finishing conversion, checkpoints of guest are passed Result: V2V can convert guest from VMware successfully now, so move the bug from ON_QA to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017