Bug 1817050

Thanks Martin for helping to debug the problem, if add '-io vddk-transports=nbd' in command line to convert guest from VMware with non-admin vCenter user and vddk by virt-v2v, the v2v conversion can finish successfully.Besides, Martin said the transport mode is nbd in his v2v conversion, maybe this is the reason why he can't reproduce the bug every time like me. so I think the bug is caused by transport mode: nbdssl. 

# virt-v2v  -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 -o rhv-upload -os nfs_data -of raw -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt esx6.7-win10-i386  -ip /home/passwd -io vddk-transports=nbd
[   0.7] Opening the source -i libvirt -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-i386 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -io vddk-transports=nbd
[   2.6] Creating an overlay to protect the source from being modified
[   5.5] Opening the overlay
[  12.5] Inspecting the overlay
[  16.2] Checking for sufficient free disk space in the guest
[  16.2] Estimating space required on target for each disk
[  16.2] Converting Windows 10 Enterprise to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0 
i386).  virt-v2v looks for this driver in 
/usr/share/virtio-win/virtio-win.iso

The guest will be configured to use a basic VGA display driver.
virt-v2v: This guest has virtio drivers installed.
[  26.2] Mapping filesystem data to avoid copying unused and blank areas
[  26.8] Closing the overlay
[  27.2] Assigning disks to buses
[  27.2] Checking if the guest needs BIOS or UEFI to boot
[  27.2] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[  28.4] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.UyDkvu/nbdkit0.sock", "file.export": "/" } (raw)
    (100.00/100%)
[1406.6] Creating output metadata

I wonder what original transport VMware was trying to use?  It doesn't
actually say very much in the original log, only:

  nbdkit: debug: VixDiskLib: Available transport modes: file:san:hotadd:nbdssl:nbd.
  ...
  nbdkit: vddk[1]: debug: 2020-03-25T21:38:01.289+08:00 warning -[7F236BFFF700] [Originator@6876 sub=transport] SAN transport mode requires a snapshot.
  ...
  nbdkit: vddk[1]: debug: VixDiskLib: Unable to locate appropriate transport mode to open disk. Error 13 (You do not have access rights to this file) (Unexpected error when trying to retrieve token for disk [esx6.7-matrix] esx6.7-win10-i386/esx6.7-win10-i386.vmdk.) at 4905.

Let's imagine that it runs through the transport modes in priority order.
SAN is tried and fails with the message above.  Then HotAdd is tried.  We
don't see any messages about it, could it have been trying HotAdd and failing?
Then maybe NBDSSL is tried, but AFAIK VMware's "NBD" and "NBDSSL" are the same
and shouldn't need different permissions.

Then we know it works if mxie forces nbd.

mxie: What is the error if you force nbdssl (-io vddk-transports=nbdssl) ?
and what is the error if you force hotadd (-io vddk-transports=hotadd) ?

VMware works in mysterious ways.

Hi rjones,
  
   Can find transport mode is nbdssl in virt-v2v debug log when convert guest from vmware via vddk with admin vCenter user.
   # cat virt-v2v-1.40.2-21.module+el8.2.0+5851-guest5-time.log |grep "transport mode:"
   Mar 04 21:13:00 nbdkit: vddk[1]: debug: transport mode: nbdssl
   Mar 04 21:13:06 nbdkit: vddk[2]: debug: transport mode: nbdssl
   Mar 04 21:16:29 nbdkit: vddk[3]: debug: transport mode: nbdssl

   Please refer to below test result, seems virt-v2v uses nbdssl as default transport mode in vddk conversion no matter what kind of vcenter user used (admin/non-admin)

# virt-v2v  -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 -o rhv-upload -os nfs_data -of raw -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt esx6.7-win10-i386  -ip /home/passwd -io vddk-transports=nbdssl
[   0.6] Opening the source -i libvirt -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-i386 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -io vddk-transports=nbdssl
[   2.3] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib_Open: [esx6.7-matrix] esx6.7-win10-i386/esx6.7-win10-i386.vmdk: You do not have access rights to this file
qemu-img: /var/tmp/v2vovl22e475.qcow2: Requested export not available
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

#  virt-v2v  -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 -o rhv-upload -os nfs_data -of raw -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt esx6.7-win10-i386  -ip /home/passwd -io vddk-transports=hotadd
[   0.6] Opening the source -i libvirt -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win10-i386 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -io vddk-transports=hotadd
[   2.2] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib_ConnectEx: One of the parameters was invalid
qemu-img: /var/tmp/v2vovlac1c03.qcow2: Requested export not available
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

How this works:

(1) VDDK loads its "Advanced Transport" module.

(2) VDDK selects a priority list of transports.  If it loaded the Advanced Transport
module successfully (which it should always do if VDDK is installed correctly), then
the list of transports will be: file:san:hotadd:nbdssl:nbd (ie. local "file",
if that fails, "san", if that fails, "hotadd" etc ...)

(3) After we successfully open the VDDK connection, the nbdkit plugin reports
which transport mode was selected:

https://github.com/libguestfs/nbdkit/blob/d6ee94d1e0da9f7eef4d78eb5b94b4cbc88eccf5/plugins/vddk/vddk.c#L680

Unfortunately if it fails VDDK doesn't give any useful information about what
transport was being selected or about why it really failed.

Using the -io vddk-transports=XXX flag tells VDDK to override step (2), ie. you
can specify the priority list of transports yourself.

It's completely weird that it fails with "nbdssl", but works with "nbd".
As far as I'm aware both modes use the same TCP port (902) and are the
same with respect to permissions required.  But:

> VMware works in mysterious ways.

BTW we would expect that -io vddk-transports=nbdssl:nbd would try "nbdssl"
and fail over to "nbd" (which we expect to work).  However I bet this won't
work.  We already know that the default priority list contains "...:nbdssl:nbd"
but that failover isn't working.  This will be yet another bug in VDDK.

(In reply to Richard W.M. Jones from comment #5)
Not only that, but if we both try it on the same vCenter with different users, but same permissions set there, then it works for me, but does not work for mxie.  If I choose anything else than nbd in vddk-transports, then it selects nbd anyway.  Yet for another user with completely the same settings it fails in different ways.

See also: https://bugzilla.redhat.com/show_bug.cgi?id=1844926
(Same bug but for VMware 7)

Can reproduce the bug in rhel8.4 even though add '-io vddk-transports=nbd' 

Test the bug with builds:
virt-v2v-1.42.0-6.module+el8.4.0+8855+a9e237a9.x86_64

Steps:
# virt-v2v -ic vpx://vsphere.local%5clz.198.169/data/10.73.199.217/?no_verify=1  esx7.0-win2019-x86_64   -o rhv-upload -os nfs_data -of raw -b ovirtmgmt -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cluster=Default -oo rhv-direct -ip /home/passwd -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem -io vddk-transports=nbd
[   0.3] Opening the source -i libvirt -ic vpx://vsphere.local%5clz.198.169/data/10.73.199.217/?no_verify=1 esx7.0-win2019-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -io vddk-transports=nbd
[   2.0] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib_Open: [esx7.0-matrix] esx7.0-win2019-x86_64/esx7.0-win2019-x86_64.vmdk: Insufficient permissions in the host operating system
qemu-img: /var/tmp/v2vovlb391f4.qcow2: Requested export not available
Could not open backing image.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

It's strange that v2v can't convert guest from VMware with non-admin account and vddk6.7 on rhel8.5 AV for the first time sometimes, but the second time it will be successful, however, the probability of such a situation is not high

#virt-v2v -ic vpx://vsphere.local%5cmini-v2v.198.169/data/10.73.199.217/?no_verify=1 -ip /root/mini-v2v  -it vddk -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  esx7.0-rhel8.4-x86_64 -io vddk-libdir=/home/vddk6.7
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cmini-v2v.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel8.4-x86_64 -it vddk  -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -io vddk-libdir=/home/vddk6.7
[   1.7] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib_Open: [esx7.0-matrix] esx7.0-rhel8.4-x86_64/esx7.0-rhel8.4-x86_64.vmdk: You do not have access rights to this file
qemu-img: /var/tmp/v2vovl23e6ff.qcow2: Requested export not available
Could not open backing image.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


#  virt-v2v -ic vpx://vsphere.local%5cmini-v2v.198.169/data/10.73.199.217/?no_verify=1 -ip /root/mini-v2v  -it vddk -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  esx7.0-rhel8.4-x86_64 -io vddk-libdir=/home/vddk6.7
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cmini-v2v.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel8.4-x86_64 -it vddk  -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -io vddk-libdir=/home/vddk6.7
[   1.6] Creating an overlay to protect the source from being modified
[   2.3] Opening the overlay
[   7.6] Inspecting the overlay
[  22.3] Checking for sufficient free disk space in the guest
[  22.3] Estimating space required on target for each disk
[  22.3] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  83.7] Mapping filesystem data to avoid copying unused and blank areas
[  84.4] Closing the overlay
[  84.7] Assigning disks to buses
[  84.7] Checking if the guest needs BIOS or UEFI to boot
[  84.7] Initializing the target -o libvirt -os default
[  84.7] Copying disk 1/1 to /var/lib/libvirt/images/esx7.0-rhel8.4-x86_64-sda (raw)
^C  (16.15/100%)

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

This bug was closed in error by a process that we do not control. Reopening.

Does the issue go away if the following three permissions are granted? (In addition to those listed at <https://libguestfs.org/virt-v2v-input-vmware.1.html#vcenter:-non-administrator-role>.)

- Global > DisableMethods
- Global > EnableMethods
- Global > License

https://www.veeam.com/kb1937
https://campus.barracuda.com/product/backup/doc/78808493/how-to-resolve-vddk-error-3014-insufficient-permissions-in-the-host-operating-system/
https://kb.vmware.com/s/article/2063054

(In reply to Laszlo Ersek from comment #17)
> Does the issue go away if the following three permissions are granted? (In
> addition to those listed at
> <https://libguestfs.org/virt-v2v-input-vmware.1.html#vcenter:-non-
> administrator-role>.)
> 
> - Global > DisableMethods
> - Global > EnableMethods
> - Global > License
> 
> https://www.veeam.com/kb1937
> https://campus.barracuda.com/product/backup/doc/78808493/how-to-resolve-vddk-
> error-3014-insufficient-permissions-in-the-host-operating-system/
> https://kb.vmware.com/s/article/2063054

Hi Laszlo,

  It doesn't work, details pls check below:


Package versions:
virt-v2v-2.0.4-1.el9.x86_64
libguestfs-1.48.1-1.el9.x86_64
guestfs-tools-1.48.0-1.el9.x86_64
nbdkit-server-1.30.4-1.el9.x86_64
libnbd-1.12.2-1.el9.x86_64
libvirt-libs-8.3.0-1.el9.x86_64
qemu-img-7.0.0-2.el9.x86_64


Steps:
1.Create non-admin user in VMware environemt with below permissions

Datastore

    Browse datastore
    Low level file operations

Global

    Disable methods
    Enable methods
    Licenses

Sessions

    Validate session

Virtual machine

    Guest operations
        Guest operation program execution
    Provisioning
        Allow disk access
        Allow read-only disk access


2.Convert a guest from VMware with above non-admin vCenter user and vddk7.0.3 by virt-v2v

#  virt-v2v  -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt -ip /home/passwd -os nfs_data esx6.7-rhel8.5-x86_64
[   0.0] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel8.5-x86_64
[   1.9] Opening the source
nbdkit: vddk[1]: error: VixDiskLib_Open: [esx6.7-matrix] esx6.7-rhel8.5-x86_64/esx6.7-rhel8.5-x86_64.vmdk: Insufficient permissions in the host operating system
virt-v2v: error: libguestfs error: could not create appliance through 
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: internal error: process exited while 
connecting to monitor: 2022-05-16T08:36:19.812929Z qemu-kvm: -blockdev 
{"driver":"nbd","server":{"type":"unix","path":"/tmp/v2v.sgziV9/in0"},"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: 
Requested export not available [code=1 int1=-1]

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Thanks - I made the simple docs change here:

https://listman.redhat.com/archives/libguestfs/2022-May/028915.html

Upstream in commit 2bf8fae2a6.

Verified with the versions:
virt-v2v-2.0.6-2.el9.x86_64
libguestfs-1.48.3-3.el9.x86_64
libvirt-8.4.0-3.el9.x86_64
nbdkit-server-1.30.6-1.el9.x86_64
libnbd-1.12.4-1.el9.x86_64
qemu-img-7.0.0-7.el9.x86_64


1. Setting the permission of the non-admin user as below:

            Datastore:
             - Browse datastore
             - Low level file operations

            Sessions:
             - Validate session

            Virtual Machine:
              Interaction:
                - Guest operating system management by VIX API
              Provisioning:
                - Allow disk access
                - Allow read-only disk access
                - Allow virtual machine download

            Cryptographic operations:
             - Decrypt
             - Direct Access


2. convert the VM with non-admin user via v2v.
# virt-v2v  -ic vpx://vsphere.local%5cvwu.73.141/data/10.73.75.219/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd  -b ovirtmgmt -ip /v2v-ops/vwupasswd -os nfs_data esx6.7-rhel8.6-x86_64
[   0.2] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cvwu.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel8.6-x86_64
[   2.3] Opening the source
[  10.7] Inspecting the source
[  30.0] Checking for sufficient free disk space in the guest
[  30.0] Converting Red Hat Enterprise Linux 8.6 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 180.7] Mapping filesystem data to avoid copying unused and blank areas
[ 181.8] Closing the overlay
[ 182.1] Assigning disks to buses
[ 182.1] Checking if the guest needs BIOS or UEFI to boot
[ 182.1] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 203.6] Copying disk 1/1
█ 100% [****************************************]
[ 526.5] Creating output metadata
[ 549.4] Finishing off

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968

*** Bug 2151882 has been marked as a duplicate of this bug. ***

*** Bug 2152088 has been marked as a duplicate of this bug. ***