Bug 1791614

Created attachment 1652697 [details]
XML passed to libvirt

"host" doesn't appear directly in the XML.  I believe this must be a libvirt regression.

Dan has this debugging suggestion:

(1) Open /etc/libvirt/libvirtd.conf

(2) In this file add (eg at the end of the file):

log_filters="1:qemu 1:qemu_monitor 1:json"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"

(3) Restart the libvirtd service:  systemctl restart libvirtd.service

(4) Rerun the virt-v2v command which fails.

(5) Check and attach the file /var/log/libvirt/libvirtd.log which should
contain very verbose logging of the interactions with qemu.

To reverse these steps you can remove the lines from /etc/libvirt/libvirtd.conf
and restart libvirtd.

Created attachment 1652714 [details]
libvirtd-log-for-comment3

An observation: There are only 3 places in the libvirt code which
use "s:host" as a parameter of virJSONValueObject*:

(1) qemuMonitorJSONBuildInetSocketAddress when building an NBD server
connection or chardev.  NBD is not used in this example, but we do
use two virtio-serial connections to Unix domain sockets (ie. no host).

(2) qemuBlockStorageSourceBuildJSONSocketAddress when building a
connection to remote storage.  In this instance all storage files
are local, but the overlay file will be backed by a remote connection,
but that would have a remote host name.

So could be related to the virtio-serial connections?  However if that
was the case then simply "libguestfs-test-tool" would fail also.

After some discussion on IRC it turns out that a backing file with the following backing store string is used:

nbd:unix:/tmp/sock:exportname=/

This string was parsed incorrectly by libvirt and the internal structures looked like an TCP based storage definition. Since the 'host' part was missing the JSON formatter used for -blockdev failed.

This is indeed a libvirt bug in the backing store parser which started manifesting itself after the switch to -blockdev in libvirt.

https://www.redhat.com/archives/libvir-list/2020-January/msg00714.html

(In reply to Peter Krempa from comment #6)
> After some discussion on IRC it turns out that a backing file with the
> following backing store string is used:
> 
> nbd:unix:/tmp/sock:exportname=/

That string should remain valid for back-compat reasons, but we should probably fix whatever generated it to instead prefer the modern URI syntax:

nbd+unix:///?socket=/tmp/sock

documented at https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md

FYI it's generated here:
https://github.com/libguestfs/virt-v2v/blob/2a0fb45f5f98f399e26926e56f45f9012cef8c84/v2v/nbdkit.ml#L351

Fixed upstream:

commit 2775aada167fad5a508b16a8cadac1fca489c7be (HEAD -> master, origin/master, origin/HEAD)
Author: Peter Krempa <pkrempa>
Date:   Thu Jan 16 13:49:37 2020 +0100

    tests: virstorage: Add tests for NBD URI style syntax over UNIX
    
    Add few test cases for nbd+unix style URIs with few corner cases.
    
    The NBD URI syntax is documented at
    https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>
    Reviewed-by: Ján Tomko <jtomko>

commit 5f2fa393f721861132500f717ce509bb66afcdb7
Author: Peter Krempa <pkrempa>
Date:   Thu Jan 16 12:34:13 2020 +0100

    util: storagefile: Properly set transport type when parsing NBD strings
    
    When parsing legacy NBD backing file strings such as
    'nbd:unix:/tmp/sock:exportname=/' we'd fail to set the transport to
    VIR_STORAGE_NET_HOST_TRANS_UNIX. This started to be a problem once we
    actually started to generate config of the backing store on the command
    line with -blockdev as the JSON code would try to format it as TCP and
    fail with:
    
     internal error: argument key 'host' must not have null value
    
    Set the type properly and add a test.
    
    This bug was found by the libguestfs test suite in:
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1791614
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reported-by: Ming Xie <mxie>
    Reviewed-by: Eric Blake <eblake>
    Tested-by: Richard W.M. Jones <rjones>

Verify the bug with builds:
virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64
libguestfs-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64
libvirt-6.0.0-2.module+el8.2.0+5513+34927b6c.x86_64
qemu-kvm-4.2.0-7.module+el8.2.0+5520+4e5817f3.x86_64


Steps:
1.Convert a guest from vmware to rhv by virt-v2v
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data -b ovirtmgmt   esx6.7-rhel7.7-x86_64
[   0.9] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.7-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   2.5] Creating an overlay to protect the source from being modified
[   5.7] Opening the overlay
[  12.6] Inspecting the overlay
[  29.7] Checking for sufficient free disk space in the guest
[  29.7] Estimating space required on target for each disk
[  29.7] Converting Red Hat Enterprise Linux Server 7.7 Beta (Maipo) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el7’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[ 120.4] Mapping filesystem data to avoid copying unused and blank areas
[ 120.8] Closing the overlay
[ 121.0] Assigning disks to buses
[ 121.0] Checking if the guest needs BIOS or UEFI to boot
[ 121.0] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 122.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.baq7Kk/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 811.4] Creating output metadata
[ 813.0] Finishing off

2.Check the guest after finishing conversion, checkpoints of guest are passed


Result:
  V2V can convert guest from VMware successfully now, so move the bug from ON_QA to VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017