Bug 1831969

Summary: vddk: vddk7.0 not supported
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: mxie <mxie>
Component: nbdkitAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.2CC: eblake, jen, jsaucier, jsuchane, juzhou, mzhan, ptoscano, rjones, tyan, tzheng, virt-maint, xiaodwan, zili
Target Milestone: rcKeywords: Triaged
Target Release: 8.3   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: nbdkit-1.16.2-3.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1847304 (view as bug list) Environment:
Last Closed: 2020-07-28 07:12:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1833376, 1847304    
Attachments:
Description Flags
v2v-vddk7.0.log none

Description mxie@redhat.com 2020-05-06 03:32:52 UTC
Created attachment 1685485 [details]
v2v-vddk7.0.log

Description of problem:
Can't convert guest from VMware via vddk7.0 by virt-v2v

Version-Release number of selected component (if applicable):
VMware-vix-disklib-7.0.0-15832853.x86_64.tar.gz
virt-v2v-1.40.2-22.module+el8.2.0+6029+618ef2ec.x86_64
libguestfs-1.40.2-22.module+el8.2.0+6029+618ef2ec.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from ESXi7.0 via vddk7.0 by virt-v2v but conversion is failed due to vddk error
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/root/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   esx7.0-rhel8.2-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -oo rhv-cluster=NFS -os nfs_data -b ovirtmgmt
nbdkit: error: libvixDiskLib.so.6: cannot open shared object file: No such file or directory

If 'libvixDiskLib.so.6' is located on a non-standard path you may need to
set $LD_LIBRARY_PATH or edit /etc/ld.so.conf.

See the nbdkit-vddk-plugin(1) man page for details.
virt-v2v: error: nbdkit VDDK plugin is not installed or not working.  It is 
required if you want to use VDDK.

It looks like you did not set the right path in the ‘-io vddk-libdir’ 
option, or your copy of the VDDK directory is incomplete.  There should be 
a library called ’<libdir>/lib64/libvixDiskLib.so.?’.

See also the virt-v2v-input-vmware(1) manual.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Check files in vddk7.0 as below
# ls /root/vmware-vix-disklib-distrib/lib64
libcares.so          libglib-2.0.so.0     libsqlite3.so.0.8.6        libvixDiskLibVim.so
libcares.so.2        libgmodule-2.0.so    libssl.so                  libvixDiskLibVim.so.7
libcrypto.so         libgmodule-2.0.so.0  libssl.so.1                libvixDiskLibVim.so.7.0.0
libcrypto.so.1       libgobject-2.0.so    libssl.so.1.0.2            libvixMntapi.so
libcrypto.so.1.0.2   libgobject-2.0.so.0  libssoclient.so            libvixMntapi.so.1
libcurl.so           libgthread-2.0.so    libstdc++.so               libvixMntapi.so.1.1.0
libcurl.so.4         libgthread-2.0.so.0  libstdc++.so.6             libvmacore.so
libdiskLibPlugin.so  libgvmomi.so         libvddkVimAccess.so        libvmomi.so
libexpat.so          liblookup-types.so   libvddkVimAccess.so.7      libxml2.so
libffi.so            libpcre.so           libvddkVimAccess.so.7.0.0  libxml2.so.2
libffi.so.6          libpcre.so.1         libvim-types.so            libz.so
libgcc_s.so          libpcre.so.1.2.9     libvixDiskLib.so           libz.so.1
libgcc_s.so.1        libsqlite3.so        libvixDiskLib.so.7
libglib-2.0.so       libsqlite3.so.0      libvixDiskLib.so.7.0.0


Actual results:
As above description

Expected results:
Can convert guest from VMware via vddk7.0 by virt-v2v

Additional info:
Can convert guest from ESXi7.0 via vddk6.7 by virt-v2v

Comment 1 Pino Toscano 2020-05-06 08:15:16 UTC
The logic for locating the libvixDiskLib.so file is in the vddk plugin of nbdkit, so reassigning.

The first thing is that it seems VDDK 7 has a different SONAME for its libvixDiskLib.so library; other changes might be needed.

Comment 2 Richard W.M. Jones 2020-05-06 08:42:07 UTC
Simple fix for AV 8.2.1:

https://github.com/libguestfs/nbdkit/commit/7f53999179af98aa47569c6771517f7dfa08c5d0

I will also include this in AV 8.3 rebase.

Comment 6 mxie@redhat.com 2020-05-11 09:21:16 UTC
Verify the bug with below builds:
virt-v2v-1.40.2-22.module+el8.2.0+6029+618ef2ec.x86_64
libguestfs-1.40.2-22.module+el8.2.0+6029+618ef2ec.x86_64
libvirt-6.0.0-19.module+el8.2.1+6538+c148631f.x86_64
qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
nbdkit-1.16.2-3.module+el8.2.1+6510+5adafada.x86_64
VMware-vix-disklib-7.0.0-15832853.x86_64.tar.gz


Steps:
1.Convert a guest from ESXi7.0 via vSphere7.0 and vddk7.0 by virt-v2v,v2vconversion can finish successfully and checkpoints of guest are passed after finishing conversion
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   esx7.0-rhel8.2-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt
[   0.5] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel8.2-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   2.3] Creating an overlay to protect the source from being modified
[   3.0] Opening the overlay
[   7.3] Inspecting the overlay
[  20.1] Checking for sufficient free disk space in the guest
[  20.1] Estimating space required on target for each disk
[  20.1] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el8’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[ 115.7] Mapping filesystem data to avoid copying unused and blank areas
[ 116.2] Closing the overlay
[ 116.4] Assigning disks to buses
[ 116.4] Checking if the guest needs BIOS or UEFI to boot
[ 116.4] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 117.6] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.EDVB49/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 543.3] Creating output metadata
[ 544.8] Finishing off


2.Convert a guest from ESXi7.0 via vddk7.0 by virt-v2v, v2v conversion can finish successfully but there is vddk error and checkpoints of guest are passed after finishing conversion
# virt-v2v  -ic esx://root.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=84:5D:EF:AE:38:66:81:FA:67:B6:EA:2B:73:C1:36:87:B5:D9:45:14 -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -n ovirtmgmt -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oo rhv-verifypeer=true -ip /home/esxpw esx7.0-win2016-x86_64
[   0.5] Opening the source -i libvirt -ic esx://root.199.217/?no_verify=1 esx7.0-win2016-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=84:5D:EF:AE:38:66:81:FA:67:B6:EA:2B:73:C1:36:87:B5:D9:45:14
[   1.6] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Other error encountered: HTTP error response: Bad Request at 4844.
[   2.1] Opening the overlay
nbdkit: vddk[2]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Other error encountered: HTTP error response: Bad Request at 4844.
[   6.5] Inspecting the overlay
[  10.0] Checking for sufficient free disk space in the guest
[  10.0] Estimating space required on target for each disk
[  10.0] Converting Windows Server 2016 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
x86_64).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  18.9] Mapping filesystem data to avoid copying unused and blank areas
[  19.6] Closing the overlay
[  19.8] Assigning disks to buses
[  19.8] Checking if the guest needs BIOS or UEFI to boot
[  19.8] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  21.1] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.UZe9zA/nbdkit0.sock", "file.export": "/" } (raw)
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Other error encountered: HTTP error response: Bad Request at 4844.
    (100.00/100%)
[ 799.4] Creating output metadata
[ 800.9] Finishing off

Additional info for step2:
2.1 There is same nbdkit error during converting a guest from ESXi6.5/ESXi6.7 via vddk7.0 by virt-v2v
2.2 There is same nbdkit error during converting a guest from ESXi6.0 via vsphere6.0 and vddk7.0 by virt-v2v


3.Convert a guest from ESXi6.7 via vSphere6.7 and vddk7.0 by virt-v2v, v2v conversion can finish successfully and checkpoints of guest are passed after finishing conversion
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   esx6.7-rhel7.8-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt
[   0.4] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.8-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   2.2] Creating an overlay to protect the source from being modified
[   2.8] Opening the overlay
[   6.9] Inspecting the overlay
[  25.3] Checking for sufficient free disk space in the guest
[  25.3] Estimating space required on target for each disk
[  25.3] Converting Red Hat Enterprise Linux Server 7.8 (Maipo) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el7’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[ 120.6] Mapping filesystem data to avoid copying unused and blank areas
[ 121.1] Closing the overlay
[ 121.4] Assigning disks to buses
[ 121.4] Checking if the guest needs BIOS or UEFI to boot
[ 121.4] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 122.5] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.GvI4pq/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 781.9] Creating output metadata
[ 783.4] Finishing off


Hi Richard,

   There is vddk error info during converting guest from ESXi host directly via vddk7.0, may I open a bug?

Comment 7 Richard W.M. Jones 2020-05-11 10:59:18 UTC
> There is vddk error info during converting guest from ESXi host directly via vddk7.0, may I open a bug?
> nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Other error encountered: HTTP error response: Bad Request at 4844.

It looks a lot like a VDDK bug, rather than something we can do anything about.
I also checked and there is no client side setting that would appear to
affect this.

So ...  you could file a bug to track this, but we may end up having to
close it as CANTFIX or something similar, or it may be that the only way
to fix it is to wait for VMware to update VDDK.

Comment 8 mxie@redhat.com 2020-05-11 12:21:55 UTC
Thanks rjones, file bug1834267 to track vddk error problem, move the bug from ON_QA to VERIFIED according to comment6~comment8

Comment 10 errata-xmlrpc 2020-07-28 07:12:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3172

Comment 11 Brett Thurber 2020-09-21 14:17:00 UTC
*** Bug 1879509 has been marked as a duplicate of this bug. ***