Bug 2035154

Summary: virt-v2v wrongly tries to finalize the transfer before closing the connections to imageio
Product: Red Hat Enterprise Linux 9 Reporter: mxie <mxie>
Component: virt-v2vAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: high Docs Contact:
Priority: medium    
Version: 9.0CC: chhu, hongzliu, juzhou, kkiwi, lersek, mzhan, nsoffer, rjones, tyan, tzheng, vwu, xiaodwan
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-v2v-1.45.98-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 13:41:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mxie@redhat.com 2021-12-23 05:35:25 UTC
Description of problem:
virt-v2v wrongly tries to finalize the transfer before closing the connections to imageio

Version-Release number of selected component (if applicable):
virt-v2v-1.45.95-2.el9.x86_64
libguestfs-1.46.1-1.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
libvirt-libs-7.10.0-1.el9.x86_64
qemu-img-6.2.0-1.el9.x86_64
nbdkit-server-1.28.3-2.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Log into rhv4.4 server and node, then configure ovirt-imageio as below

1.1 Download ovirt-imageio from https://github.com/oVirt/ovirt-imageio/suites/4680140720/artifacts/127451989, then update ovirt-imageio packages for rhv server and node

#  rpm -qa |grep ovirt-imageio
ovirt-imageio-common-debuginfo-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-daemon-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-debugsource-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-client-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-common-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64

1.2 # cat /etc/ovirt-imageio/conf.d/99-bz2032324.conf
[daemon]
inactivity_timeout = 600

1.3 # systemctl restart ovirt-imageio


2.Convert guests from VMware to rhv4.4 via rhv-upload by v2v

2.1 Set environment variable because of 2034160
# export LIBGUESTFS_BACKEND=direct

2.2 Convert a guest from ESXi7.0 via vddk7.0.2 + rhv-upload by v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd  -o rhv-upload -oc https://dell-per740-48.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -oo rhv-cluster=NFS -os nfs_data -b ovirtmgmt  esx7.0-rhel8.5-x86_64 
[   1.8] Opening the source
[   6.6] Inspecting the source
[  15.5] Checking for sufficient free disk space in the guest
[  15.5] Converting Red Hat Enterprise Linux 8.5 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 117.9] Mapping filesystem data to avoid copying unused and blank areas
[ 118.8] Closing the overlay
[ 118.9] Assigning disks to buses
[ 118.9] Checking if the guest needs BIOS or UEFI to boot
[ 140.4] Copying disk 1/1
█ 100% [****************************************]
[ 799.1] Creating output metadata
Traceback (most recent call last):
  File "/tmp/v2v.DGyhoK/rhv-upload-finalize.py", line 172, in <module>
    finalize_transfer(connection, transfer_id, disk_id)
  File "/tmp/v2v.DGyhoK/rhv-upload-finalize.py", line 130, in finalize_transfer
    raise RuntimeError(
RuntimeError: timed out waiting for transfer c3cf7326-2e6b-4dfc-8928-cad6ba504cf2 to finalize, transfer is finalizing_success
virt-v2v: error: failed to finalize the transfers, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

2.3 Convert a guest from ESXi6.5 via vddk7.0.2 + rhv-upload by v2v, get same result with step2.2
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -ip /home/passwd  -o rhv-upload -oc https://dell-per740-48.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -oo rhv-cluster=NFS -os nfs_data -b ovirtmgmt esx6.5-win11-x86_64 -v -x |& tee > ovirt-imageio-daemon-2.4.1-inactivity_timeout_600.log


Actual results:
As above description

Expected results:
virt-v2v should finalize the transfer after closing the connections to imageio.

Additional info:

Comment 2 Richard W.M. Jones 2021-12-23 09:07:19 UTC
See: https://bugzilla.redhat.com/show_bug.cgi?id=2032324#c43
and following comments.

Comment 3 Klaus Heinrich Kiwi 2021-12-23 12:42:27 UTC
(In reply to Richard W.M. Jones from comment #2)
> See: https://bugzilla.redhat.com/show_bug.cgi?id=2032324#c43
> and following comments.

Thanks Rich. I'll assign it to you as a high-severity, regression bug so that we don't loose sight of it - https://bugzilla.redhat.com/show_bug.cgi?id=2032324#c44

Comment 4 Richard W.M. Jones 2022-02-14 10:38:13 UTC
This was fixed by:
https://github.com/libguestfs/virt-v2v/commit/d69ba56b2f4bc642ce59bfc6bdd5c137480bf8c3

Comment 8 mxie@redhat.com 2022-02-15 15:47:24 UTC
Verify the bug with below builds:
virt-v2v-1.45.98-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
libvirt-libs-8.0.0-4.el9.x86_64
qemu-img-6.2.0-8.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.5-1.el9.x86_64

Steps:
1.Log into rhv4.4 server and node, then configure ovirt-imageio as below

1.1 Download ovirt-imageio from https://github.com/oVirt/ovirt-imageio/suites/4680140720/artifacts/127451989, then update ovirt-imageio packages for rhv server and node

#  rpm -qa |grep ovirt-imageio
ovirt-imageio-common-debuginfo-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-daemon-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-debugsource-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-client-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64
ovirt-imageio-common-2.4.1-0.202112161451.gitd8eaabc.el8.x86_64

1.2 # cat /etc/ovirt-imageio/conf.d/99-bz2032324.conf
[daemon]
inactivity_timeout = 600

1.3 
# systemctl restart ovirt-imageio
# systemctl restart ovirt-engine


2. Convert a guest from ESXi7.0 via vddk7.0.2 + rhv-upload without rhv-direct by v2v
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  -ip /home/passwd   -o rhv-upload  -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx6.5-win2022-x86_64 
[   0.0] Setting up the source: -i libvirt -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk esx6.5-win2022-x86_64
[   2.1] Opening the source
[   9.3] Inspecting the source
[  14.6] Checking for sufficient free disk space in the guest
[  14.6] Converting Windows Server 2022 Standard to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  23.4] Mapping filesystem data to avoid copying unused and blank areas
[  24.9] Closing the overlay
[  25.1] Assigning disks to buses
[  25.1] Checking if the guest needs BIOS or UEFI to boot
[  25.1] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[  49.0] Copying disk 1/1
█ 100% [****************************************]
[1074.5] Creating output metadata
[1081.9] Finishing off

3. Convert a guest from ESXi7.0 via vddk7.0.2 + rhv-upload with rhv-direct=true by v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd  -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt esx7.0-rhel8.5-x86_64  -oo rhv-direct=true 
[   0.0] Setting up the source: -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel8.5-x86_64
[   1.9] Opening the source
[   7.3] Inspecting the source
[  16.4] Checking for sufficient free disk space in the guest
[  16.4] Converting Red Hat Enterprise Linux 8.5 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 113.6] Mapping filesystem data to avoid copying unused and blank areas
[ 114.5] Closing the overlay
[ 114.8] Assigning disks to buses
[ 114.8] Checking if the guest needs BIOS or UEFI to boot
[ 114.8] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 128.1] Copying disk 1/1
█ 100% [****************************************]
[ 605.7] Creating output metadata
[ 616.4] Finishing off


Result:
   The bug has been fixed, move bug from ON_QA to VERIFIED

Comment 10 errata-xmlrpc 2022-05-17 13:41:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: virt-v2v), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2566