Bug 1832805 - v2v conversion will be failed because of memory allocation failure if convert guest from ESXi6.0 or ESXi6.5 via vddk, or use vddk6.5 to convert guest from VMware
Summary: v2v conversion will be failed because of memory allocation failure if convert...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: virt-v2v
Version: 8.3
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: rc
: 8.3
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-07 10:14 UTC by mxie@redhat.com
Modified: 2020-11-17 17:48 UTC (History)
13 users (show)

Fixed In Version: virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-17 17:48:34 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
v2v-vddk-rhel8.3.log (1.02 MB, text/plain)
2020-05-07 10:14 UTC, mxie@redhat.com
no flags Details
convert-guest-from-esx6.0-via-vddk-on-rhel8.3_fast.log (1.44 MB, text/plain)
2020-05-20 02:53 UTC, mxie@redhat.com
no flags Details

Description mxie@redhat.com 2020-05-07 10:14:25 UTC
Created attachment 1686127 [details]
v2v-vddk-rhel8.3.log

Description of problem:
V2V can't convert guest from VMware via vddk on rhel8.3 

Version-Release number of selected component (if applicable):
qemu-kvm-4.2.0-19.module+el8.3.0+6478+69f490bb.x86_64
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
libguestfs-1.42.0-1.module+el8.3.0+6496+d39ac712.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
nbdkit-1.20.1-1.module+el8.3.0+6491+7f1d5b30.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from Vmware via vddk by virt-v2v but conversion is failed with error
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.5-rhel8.2-x86_64 -ip /home/passwd
[   0.2] Opening the source -i libvirt -ic vpx://root.73.141:443/data/10.73.196.89/?no_verify=1 esx6.5-rhel8.2-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   1.8] Creating an overlay to protect the source from being modified
[   5.1] Opening the overlay
[  12.2] Inspecting the overlay
[  21.3] Checking for sufficient free disk space in the guest
[  21.3] Estimating space required on target for each disk
[  21.3] Converting Red Hat Enterprise Linux 8.2 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  65.3] Mapping filesystem data to avoid copying unused and blank areas
[  65.8] Closing the overlay
[  66.0] Assigning disks to buses
[  66.0] Checking if the guest needs BIOS or UEFI to boot
[  66.0] Initializing the target -o libvirt -os default
[  66.1] Copying disk 1/1 to /var/lib/libvirt/images/esx6.5-rhel8.2-x86_64-sda (raw)
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
qemu-img: error while reading sector 89088: Input/output error

virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.Convert a guest from Vmware without vddk by virt-v2v, conversin can finish without error
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 esx6.5-rhel8.1-x86_64 -ip /home/passwd
[   0.2] Opening the source -i libvirt -ic vpx://root.73.141:443/data/10.73.196.89/?no_verify=1 esx6.5-rhel8.1-x86_64
[   2.8] Creating an overlay to protect the source from being modified
[   3.3] Opening the overlay
[  37.9] Inspecting the overlay
[ 109.7] Checking for sufficient free disk space in the guest
[ 109.7] Estimating space required on target for each disk
[ 109.7] Converting Red Hat Enterprise Linux 8.1 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 910.0] Mapping filesystem data to avoid copying unused and blank areas
[ 912.4] Closing the overlay
[ 912.7] Assigning disks to buses
[ 912.7] Checking if the guest needs BIOS or UEFI to boot
[ 912.7] Initializing the target -o libvirt -os default
[ 912.7] Copying disk 1/1 to /var/lib/libvirt/images/esx6.5-rhel8.1-x86_64-sda (raw)
    (100.00/100%)
[1312.8] Creating output metadata
[1312.8] Finishing off



Actual results:
As above description

Expected results:
V2V can convert guest from VMware via vddk on rhel8.3 

Additional info:
1.Can't reproduce the bug when qemu-kvm version is 4.2.0-20, so I think the bug is caused by qemu-kvm
Packages:
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64

# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA esx6.5-rhel8.1-x86_64 -ip /home/passwd
[   0.1] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 esx6.5-rhel8.1-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   2.0] Creating an overlay to protect the source from being modified
[   5.3] Opening the overlay
[  14.8] Inspecting the overlay
[  24.1] Checking for sufficient free disk space in the guest
[  24.1] Estimating space required on target for each disk
[  24.1] Converting Red Hat Enterprise Linux 8.1 (Ootpa) to run on KVM
virt-v2v: warning: guest tools directory ‘linux/el8’ is missing from 
the virtio-win directory or ISO.

Guest tools are only provided in the RHV Guest Tools ISO, so this can 
happen if you are using the version of virtio-win which contains just the 
virtio drivers.  In this case only virtio drivers can be installed in the 
guest, and installation of Guest Tools will be skipped.
virt-v2v: This guest has virtio drivers installed.
[  77.3] Mapping filesystem data to avoid copying unused and blank areas
[  78.0] Closing the overlay
[  78.3] Assigning disks to buses
[  78.3] Checking if the guest needs BIOS or UEFI to boot
[  78.3] Initializing the target -o libvirt -os default
[  78.4] Copying disk 1/1 to /var/lib/libvirt/images/esx6.5-rhel8.1-x86_64-sda (raw)
    (100.00/100%)
[ 275.8] Creating output metadata
Pool default refreshed

Domain esx6.5-rhel8.1-x86_64 defined from /tmp/v2vlibvirtfdb8e8.xml

[ 276.0] Finishing off

Comment 1 Richard W.M. Jones 2020-05-07 10:18:38 UTC
Isn't this caused by the usual configuration problem on the VMware side?

http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-memory-limits

Comment 2 mxie@redhat.com 2020-05-07 10:29:29 UTC
(In reply to Richard W.M. Jones from comment #1)
> Isn't this caused by the usual configuration problem on the VMware side?
> 
> http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-
> memory-limits

I don't think so, I can't reproduce bug on rhel8.2 when testing virt-v2v with same VMware environment

Comment 4 Richard W.M. Jones 2020-05-10 10:09:48 UTC
I've move this to libguestfs for now.  Eventually will go under the
virt-v2v component once we get it added to BZ.

Comment 6 mxie@redhat.com 2020-05-20 02:49:33 UTC
Hi rjones,

   Please ignore comment0 and comment5 because I can reproduce the bug on rhel8.3 fast stream when convert guest from ESXi6.0 via vddk by virt-v2v, details please refer to below, so I will change the bug title as "Virt-v2v can't convert guest from ESXi6.0 via vddk because of memory allocation failed"

  
Packages:
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
libguestfs-1.42.0-1.module+el8.3.0+6496+d39ac712.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.1-1.module+el8.3.0+6491+7f1d5b30.x86_64
VMware-vix-disklib-6.5.2-6195444.x86_64.tar.gz
VMware-vix-disklib-7.0.0-15832853.x86_64.tar.gz


Automation test:
https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/v2v/view/RHEL-8.3/job/v2v-RHEL-8.3-runtest-x86_64-acceptance-ovirt/5/testReport/rhel/convert_vm_to_ovirt/esx_vm_6_0_linux_latest6_arch_x86_64_raw_f_NFS_rhv_upload_rhv_direct_rhv_verifypeer_nopreallocated_it_vddk/


Steps to Reproduce by manual testing:
1.Convert guest from ESXi6.0 via vddk6.5 by virt-v2v 
# virt-v2v -ic vpx://root.73.148/data/10.73.72.61/?no_verify=1 -it vddk -io vddk-libdir=/root/vmware-vix-disklib-distrib -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D esx6.0-rhel6.10-x86_64 -ip /home/passwd -o local -os /home
[   0.2] Opening the source -i libvirt -ic vpx://root.73.148:443/data/10.73.72.61/?no_verify=1 esx6.0-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/root/vmware-vix-disklib-distrib -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   1.7] Creating an overlay to protect the source from being modified
[   5.0] Opening the overlay
[  12.5] Inspecting the overlay
[  24.4] Checking for sufficient free disk space in the guest
[  24.4] Estimating space required on target for each disk
[  24.4] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 107.4] Mapping filesystem data to avoid copying unused and blank areas
[ 107.9] Closing the overlay
[ 108.2] Assigning disks to buses
[ 108.2] Checking if the guest needs BIOS or UEFI to boot
[ 108.2] Initializing the target -o local -os /home
[ 108.2] Copying disk 1/1 to /home/esx6.0-rhel6.10-x86_64-sda (raw)
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
qemu-img: error while reading at byte 117047296: Input/output error

virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


2.Convert guest from ESXi6.0 via vddk7.0 by virt-v2v 
# virt-v2v -ic vpx://root.73.148/data/10.73.72.61/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D esx6.0-rhel6.10-x86_64 -ip /home/passwd -o local -os /home
[   0.2] Opening the source -i libvirt -ic vpx://root.73.148:443/data/10.73.72.61/?no_verify=1 esx6.0-rhel6.10-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=AA:F5:4C:48:C9:BF:75:1A:94:41:61:4C:D5:EC:DF:46:48:B5:9B:4D
[   1.7] Creating an overlay to protect the source from being modified
nbdkit: vddk[1]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
[   5.4] Opening the overlay
nbdkit: vddk[2]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
[  13.7] Inspecting the overlay
[  48.9] Checking for sufficient free disk space in the guest
[  48.9] Estimating space required on target for each disk
[  48.9] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 330.2] Mapping filesystem data to avoid copying unused and blank areas
[ 330.7] Closing the overlay
[ 331.0] Assigning disks to buses
[ 331.0] Checking if the guest needs BIOS or UEFI to boot
[ 331.0] Initializing the target -o local -os /home
[ 331.0] Copying disk 1/1 to /home/esx6.0-rhel6.10-x86_64-sda (raw)
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
nbdkit: vddk[3]: error: VixDiskLib: VixDiskLib_Connect: Get CEIP status failed, treat CEIP as disabled. Invalid request. at 4844.
nbdkit: vddk[3]: error: [NFC ERROR]NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes
nbdkit: vddk[3]: error: VixDiskLib_Read: Memory allocation failed. Out of memory.
qemu-img: error while reading at byte 117047296: Input/output error

virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 7 mxie@redhat.com 2020-05-20 02:53:56 UTC
Created attachment 1690054 [details]
convert-guest-from-esx6.0-via-vddk-on-rhel8.3_fast.log

Comment 8 mxie@redhat.com 2020-05-21 09:57:15 UTC
Hi rjones,

    As v2v can convert guest from ESXi6.0 without vddk, and our esxi6.0 server has <maxMemory>50331648</maxMemory>, the bug may be caused by nbdkit-1.20.1-1

Packages:
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
libguestfs-1.42.0-1.module+el8.3.0+6496+d39ac712.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64


1.Log into ESXi6.0 host and check ESXi6.0 host about the limit for the NFC service

 #cat /etc/vmware/hostd/config.xml |grep '<nfcsvc>' -A 5
      <nfcsvc>
         <path>libnfcsvc.so</path>
         <enabled>true</enabled>
         <maxMemory>50331648</maxMemory>
         <maxStreamMemory>10485760</maxStreamMemory>
      </nfcsvc>

2.Convert guest from ESXi6.0 without vddk by v2v
# virt-v2v -ic vpx://root.73.148/data/10.73.72.61/?no_verify=1  esx6.0-rhel6.10-x86_64 -ip /home/passwd   -o local -os /home
[   0.2] Opening the source -i libvirt -ic vpx://root.73.148:443/data/10.73.72.61/?no_verify=1 esx6.0-rhel6.10-x86_64
[   2.6] Creating an overlay to protect the source from being modified
[   3.1] Opening the overlay
[  45.4] Inspecting the overlay
[ 121.7] Checking for sufficient free disk space in the guest
[ 121.7] Estimating space required on target for each disk
[ 121.7] Converting Red Hat Enterprise Linux Workstation release 6.10 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1178.6] Mapping filesystem data to avoid copying unused and blank areas
[1180.6] Closing the overlay
[1180.9] Assigning disks to buses
[1180.9] Checking if the guest needs BIOS or UEFI to boot
[1180.9] Initializing the target -o local -os /home
[1180.9] Copying disk 1/1 to /home/esx6.0-rhel6.10-x86_64-sda (raw)
    (100.00/100%)
[1686.2] Creating output metadata
[1686.2] Finishing off


Additional info:
Can't reproduce the bug on rhel8.2 fast train,pls refer to below automated test report which is tested with nbdkit-1.16.2-3.module+el8.2.1 
https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/v2v/view/RHEL-8.2/job/v2v-RHEL-8.2-runtest-x86_64-acceptance-ovirt/34/testReport/rhel/convert_vm_to_ovirt/esx_vm_6_0_linux_latest6_arch_i386_raw_f_NFS_rhv_upload_rhv_direct_rhv_verifypeer_nopreallocated_it_vddk/

Comment 11 Richard W.M. Jones 2020-05-22 11:16:10 UTC
I just realised that all the testing I did in comment 9 and comment 10
was wrong because I forgot to add -it vddk.  I'll start again, but in
the meantime ignore those comments.

Comment 12 Richard W.M. Jones 2020-05-22 14:27:14 UTC
CAN reproduce it using:

$ rpm -q libguestfs virt-v2v nbdkit qemu-kvm qemu-img 
libguestfs-1.42.0-1.module+el8.3.0+6496+d39ac712.x86_64
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
nbdkit-1.20.2-1.scrmod+el8.3.0+6697+3b959a4c.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
qemu-img-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64

I will downgrade versions to see if it's caused by a particular package next.

Comment 13 mxie@redhat.com 2020-05-23 06:49:04 UTC
Add some research results from zili and me for the bug

Package version:
virt-v2v-1.42.0-3.module+el8.3.0+6497+b190d2a5.x86_64
libguestfs-1.42.0-1.module+el8.3.0+6496+d39ac712.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.1-1.module+el8.3.0+6491+7f1d5b30.x86_64
VMware-vix-disklib-6.5.2-6195444.x86_64.tar.gz
VMware-vix-disklib-6.7.3-14389676.x86_64.tar.gz
VMware-vix-disklib-7.0.0-15832853.x86_64.tar.gz


Convert rhel/windows guest from different ESXi hosts via vpx:// and different version vddks, got below result:

               ESXi6.0                  ESXi6.5                     ESXi6.7                   ESXi7.0                
VDDK6.5     windows/rhel fail       windows/rhel fail          windows/rhel fail             windows/rhel fail
VDDK6.7     windows/rhel fail       windows/rhel fail          windows/rhel pass             windows/rhel pass
VDDK7.0     windows/rhel fail       windows/rhel fail          windows/rhel pass             windows/rhel pass


According to above result, we can see v2v conversion will be failed because of Memory allocation failed if convert guest from ESXi6.0 or ESXi6.5 via vddk, or use vddk6.5 to convert guest from VMware on rhel8.3 fast train

Comment 14 Richard W.M. Jones 2020-05-24 15:06:49 UTC
Downgrading to this makes the problem go away:

libguestfs-1.40.2-22.module+el8.3.0+6423+e4cb6418.x86_64
nbdkit-1.16.2-2.module+el8.3.0+6423+e4cb6418.x86_64

Upgrading only this and the problem is still gone:

nbdkit-1.20.2-1.scrmod+el8.3.0+6697+3b959a4c.x86_64

Finally, upgrading libguestfs/virt-v2v 1.40 -> 1.42 causes the bug!

Could it be https://github.com/libguestfs/virt-v2v/commit/2aa78ade2d48e926b7b04050338ebd8a0c5e3f05 ?
I'll try a git bisect on libguestfs RHEL 8.3 branch next.

Comment 15 Richard W.M. Jones 2020-05-27 18:23:33 UTC
I am still doing the bisect (which takes about a day per step), but my
theory at the moment is that it's caused by:
https://github.com/libguestfs/virt-v2v/commit/1becdced2d9c247282912d9c79d0558a8f3410bf
Before this commit, the readahead filter would not be enabled (because
we would misdetect that it was not available).  After this commit
the readahead filter would be correctly enabled.  This would cause larger
requests to be made via VDDK which might cause these problems on the
VMware server.

Comment 16 Richard W.M. Jones 2020-05-28 09:23:13 UTC
Can finally confirm:

1becdced2d9c247282912d9c79d0558a8f3410bf is the first bad commit
commit 1becdced2d9c247282912d9c79d0558a8f3410bf
Author: Tomáš Golembiovský <tgolembi>
Date:   Mon Jan 20 14:05:41 2020 +0100

    nbdkit: fix condition in probe_filter
    
    The tests assume probe_filter returns true if the filter is available
    (and not the other way around).
    
    Signed-off-by: Tomáš Golembiovský <tgolembi>

:040000 040000 8505cebddc8585f41d601b4b159974dbeb085b48 3185dc4ad579291f996ca11ebd21dfa399c18a2b M	v2v

NOTE: This is NOT a buggy commit!  What has happen is that Tomáš's
correct fix has enabled one of the filters (most likely readahead) and
that filter has broken VDDK.

Comment 17 Richard W.M. Jones 2020-05-28 11:23:53 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2020-May/thread.html#00107

Comment 21 mxie@redhat.com 2020-06-01 05:12:39 UTC
Verify the bug with below builds:
virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.2-1.module+el8.3.0+6764+cc503f20.x86_64
virtio-win-1.9.11-1.el8.noarch
VMware-vix-disklib-6.5.2-6195444.x86_64.tar.gz
VMware-vix-disklib-6.7.3-14389676.x86_64.tar.gz
VMware-vix-disklib-7.0.0-15832853.x86_64.tar.gz

Steps:
1.Convert a guest from different ESXi(>=6.0) hosts via different versions of VDDK, summary result as below:

             ESXi6.0        ESXi6.5       ESXi6.7        ESXi7.0                
VDDK6.5       PASS           PASS          PASS           PASS        
VDDK6.7       PASS           PASS          PASS           PASS 
VDDK7.0       PASS           PASS          PASS           PASS 


Result:
   V2V conversion can finish successfully when convert a guest from different ESXi(>=6.0) hosts via different versions of VDDK, so move the bug from ON_QA to VERIFIED

Comment 24 errata-xmlrpc 2020-11-17 17:48:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137


Note You need to log in before you can comment on or make changes to this bug.