Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2011713

Summary: Enable modular virt-v2v (2.0) in RHEL 9 GA
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: virt-v2vAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: high Docs Contact:
Priority: medium    
Version: 9.0CC: chhu, juzhou, kkiwi, lersek, marcandre.lureau, mkletzan, mxie, rjones, tyan, tzheng, xiaodwan
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-v2v-1.45.97-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 13:41:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2029751    
Bug Blocks: 1961107, 2032112    

Description Richard W.M. Jones 2021-10-07 07:46:01 UTC 
Description of problem:

For RHEL 9 Beta we tried to introduce the new modular virt-v2v
(bug 1798105) but that didn't work out.  The modular virt-v2v
work is now upstream and I'm unhappy about shipping RHEL 9 with
the old code and then trying to update to the new version in
a RHEL 9 minor release.  This is because modular virt-v2v removes
some small features which could be regarded as regressions if
we remove them in a RHEL minor.  So I'd like to push to upgrading
to modular virt-v2v in RHEL 9 GA.

The command line works almost exactly the same way as for
old virt-v2v, so hopefully test plans won't need to be
changed.

The main visible differences on the command line are:

* -of option is no longer implicit from input format, and
  you may need to supply it

* --debug-overlays was removed, this was a debugging option
  and was never intended for general use

* --no-copy was removed (use -o null instead)

* --print-estimate option was removed, in modular virt-v2v
  there will be better ways to get estimates

* virt-v2v-copy-to-local was removed

Version-Release number of selected component (if applicable):

The version will be virt-v2v 2.0.x
Comment 2 Richard W.M. Jones 2021-10-07 07:47:20 UTC 
*** Bug 1798105 has been marked as a duplicate of this bug. ***
Comment 5 mxie@redhat.com 2021-12-01 16:45:39 UTC 
Test the bug with below builds:
virt-v2v-1.45.91-1.el9.x86_64
libguestfs-1.46.0-5.el9.x86_64
guestfs-tools-1.46.1-5.el9.x86_64
nbdkit-1.28.2-2.el9.x86_64
libvirt-libs-7.9.0-1.el9.x86_64
qemu-img-6.1.0-7.el9.x86_64


Steps: 
1.Compare the performance between virt-v2v-1.45.91-1 and virt-v2v-1.45.3-3

1.1 Convert same guest from same VMware env via same vddk+rhv-upload one by one 
-----------------------------------------------------------------------------
v2v_version   ESXi7.0_vddk7.0.2        ESXi7.0+vddk6.7          ESXi7.0+vddk6.5               

1.45.91-1     Convert guest: 3m20s     Convert guest: 3m14s     Convert guest: 2m19s
              Copying disk:  12m25s    Copying disk:  19m39s    Copying disk:  13m44s  
                               
 
1.45.3-3      Convert guest: 2m43s     Convert guest: 1m12s     Convert guest: 2m14s 
              Copying disk:  4m35s     Copying disk:  14m9s     Copying disk:  12m12s 

-----------------------------------------------------------------------------
v2v_version   ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7         ESXi6.7+vddk6.5      
          
1.45.91-1     Convert guest: 2m32s    Convert guest: 2m41s     Convert guest: 3m26s
              Copying disk:  13m8s    Copying disk:  11m28s    Copying disk:  18m39s 


1.45.3-3      Convert guest: 2m13s    Convert guest: 2m11s     Convert guest: 1m9s
              Copying disk:  11m19s   Copying disk:  4m41s     Copying disk:  9m47s 
------------------------------------------------------------------------------

1.2 Convert guests with below three ways at the same time 
-----------------------------------------------------------------------------
v2v_version   vmx+ssh_rhv-upload        rhv_to_local              ova_to_rhv 
          
1.45.91-1     Convert guest: 1m58s      Convert guest: 24m47s     Convert guest: 2m3s
              Copying disk:  8m55s      Copying disk:  4m47s      Copying disk:  21m16s 


1.45.3-3      Convert guest: 2m8s       Convert guest: 20m9s      Convert guest: 1m47s
              Copying disk:  5m44s      Copying disk:  14m15s     Copying disk:  5m37s 
------------------------------------------------------------------------------

2.Convert a guest from VMware to OSP16.2 by v2v
# virt-v2v -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd -o openstack -oo server-id=v2v-appliance-rhel9_0 esx7.0-win2022-x86_64
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Failed to set volume read-only access mode flag: Invalid volume: Volume c7137e76-61f1-434d-8b8f-05cd7f4ba097 status must be available to update readonly flag, but current status is: creating. (HTTP 400) (Request-ID: req-5bdaee0e-2fbe-48ef-b527-3d97e3878512)
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
[   0.0] Opening the source
[  23.7] Inspecting the source
[  40.8] Checking for sufficient free disk space in the guest
[  40.8] Converting Windows Server 2022 Standard to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  67.2] Mapping filesystem data to avoid copying unused and blank areas
[  80.4] Closing the overlay
[  80.7] Assigning disks to buses
[  80.7] Checking if the guest needs BIOS or UEFI to boot
[ 103.9] Copying disk 1/1
█ 100% [****************************************]
[ 493.7] Creating output metadata
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
[ 501.5] Finishing off
/usr/lib/python3.9/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3.9/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes

2.1 Launch the volume as instance on OSP env after v2v conversion, checkpoints of guest are passed


3.Check if -no-copy was removed from v2v

# man virt-v2v |grep no-copy -B 3
       -o null
           Set the output method to null.

           The guest is converted and copied (unless you also specify --no-copy), but the results are thrown away and no metadata is written.

# virt-v2v --help |grep no-copy
nothing

# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd  esx7.0-rhel8.5-x86_64  --no-copy -o null
virt-v2v: unrecognized option '--no-copy'
Try ‘virt-v2v --help’ or consult virt-v2v(1) for more information.


4.Check if "--debug-overlays" "virt-v2v-copy-to-local" and 'print-estimate' have been removed from virt-v2v(actually, they have been removed since version 1.45.2-1, which is fixed in bug1950634)
# man virt-v2v |grep 'debug-overlays'
nothing
# man virt-v2v |grep 'print-estimate'
nothing
# virt-v2v --help  |grep 'print-estimate'
nothing
# virt-v2v --help |grep 'debug-overlays'
nothing
# man virt-v2v-input-xen |grep virt-v2v-copy-to-local
nothing

# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd  esx7.0-rhel8.5-x86_64  --print-estimate
virt-v2v: unrecognized option '--print-estimate'
Try ‘virt-v2v --help’ or consult virt-v2v(1) for more information.

# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd  esx7.0-rhel8.5-x86_64 --debug-overlays
virt-v2v: unrecognized option '--debug-overlays'
Try ‘virt-v2v --help’ or consult virt-v2v(1) for more information.

# virt-v2v-copy-to-local
bash: virt-v2v-copy-to-local: command not found...


5.Check if -of option is no longer implicit from input format

Input mode   If needs -of 

ova          NO
vpx          NO
esx          NO
vmx+ssh      NO
xen          NO  


6.Convert a guest from Xen to json format by v2v
#  virt-v2v -ic xen+ssh://root.224.33 xen-pv-rhel6.10-x86_64  -o json -os /home
[   0.0] Opening the source
[   7.3] Inspecting the source
[  14.3] Checking for sufficient free disk space in the guest
[  14.3] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 120.5] Mapping filesystem data to avoid copying unused and blank areas
[ 122.6] Closing the overlay
[ 122.9] Assigning disks to buses
[ 122.9] Checking if the guest needs BIOS or UEFI to boot
[ 128.0] Copying disk 1/1
█ 100% [****************************************]
[ 509.1] Creating output metadata
[ 509.2] Finishing off

# cat /home/xen-pv-rhel6.10-x86_64.json |grep vga
    "video": "vga",


7.Check v2v man pages, found some mistakes

7.1 
# man virt-v2v |grep in-place
        virt-v2v --in-place
       place conversion (--in-place) only uses the -i* options and modifies the source guest in-place.  (See "In-place conversion" below.)
       --in-place
       You must have sufficient free space in the host directory used to store large temporary overlays (except in --in-place mode).  To find out
        virt-v2v -ic qemu:///system converted_vm --in-place

#  virt-v2v -i ova esx7.0-win2019-x86_64  --in-place
virt-v2v: error: XXX --in-place option is not implemented yet

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

7.2 
# man virt-v2v-output-rhv |grep raw
                               [-op PASSWORD] [-of raw]
       -of raw
           Currently you must use -of raw and you cannot use -oa preallocated.

# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.5 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  -ip /home/passwd   -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx6.7-win11-x86_64   -oa preallocated

[   0.0] Opening the source
[   8.2] Inspecting the source
[  20.2] Checking for sufficient free disk space in the guest
[  20.2] Converting Windows 10 Enterprise to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  39.0] Mapping filesystem data to avoid copying unused and blank areas
[ 134.8] Closing the overlay
[ 135.1] Assigning disks to buses
[ 135.1] Checking if the guest needs BIOS or UEFI to boot
[ 162.3] Copying disk 1/1
^C 28% [************----------------------------]

7.3 
# man virt-v2v |grep release-notes
       virt-v2v-release-notes-1.42(1) — Release notes for this release.




Hi rjones,

   (1) Please check the result of step1, the overall performance of virt-v2v-1.45.91-1 is not as good as virt-v2v-1.45.3-3, especially copying disk speed of virt-v2v-1.45.91-1 is much slower than virt-v2v-1.45.3-3 in most of v2v conversions , only when convert guest from VMware without vddk to local, the performance of virt-v2v-1.45.91-1 is better than virt-v2v-1.45.3-3

   (2) Please check the result of step3,  --no-copy info is still shown in the introduction of -o null option in v2v man page
   
   (3) Please check the result of step7, found some mistakes in v2v related man pages, first one is '--in-place' info exists in v2v man page, second one is info about '-of raw' and '-oa preallocated' is incorrect in virt-v2v-output-rhv, the last one is I think release notes info should be removed or update the releae info to 1.45.91-1 because  virt-v2v-release-notes-1.42 is too old
Comment 7 Richard W.M. Jones 2021-12-01 20:08:31 UTC 
I'm sure there are still performance regressions.  I need to work on tuning
the nbdcopy parameters.

About the other two points:

(2) --no-copy: Yes, that's a bug, fixed here:
https://github.com/libguestfs/virt-v2v/commit/027eb509818ac3ac4d450bf85a31a7b8b3c1732d

(3) --in-place should not be mentioned or available at all.  I think that's 
a problem with the downstream patch which is supposed to remove this, but
doesn't.  I'll fix this in the next modular version.

-of raw / -oa preallocated: I think this is still correct advice, but I'll have to
ask Nir Soffer about it.  In the new virt-v2v -oa preallocated now works, but I
think it'll be extremely slow with -o rhv-upload, and may not be advisable.

Release notes: Fixed here, thanks:
https://github.com/libguestfs/virt-v2v/commit/ddc96fc5aae4c0249f7485c728ccb102e8301424
Comment 8 Richard W.M. Jones 2021-12-02 15:46:45 UTC 
virt-v2v-1.45.92-1.el9

- Rebase to upstream 1.45.92.
- Change video type to VGA (instead of QXL).
- Remove --in-place support properly.
- Remove -o glance support properly.
- Fix quoting with openssh >= 8.7 (RHEL) / 8.8
  resolves: rhbz#2011713, rhbz#1961107, rhbz#2027673
Comment 9 mxie@redhat.com 2021-12-07 09:20:42 UTC 
Test the bug with below builds:
virt-v2v-1.45.93-1.el9.x86_64
libguestfs-1.46.0-5.el9.x86_64
guestfs-tools-1.46.1-5.el9.x86_64
nbdkit-1.28.2-2.el9.x86_64
libvirt-libs-7.9.0-1.el9.x86_64
qemu-img-6.1.0-7.el9.x86_64
openssh-8.7p1-5.el9.x86_64

Steps:
1.Check v2v related man pages if the doc mistakes of comment5 are fixed
1.1 # man virt-v2v |grep in-place
nothing

1.2 # man virt-v2v |grep no-copy -B 3
nothing

1.3 # man virt-v2v |grep release-notes
       virt-v2v-release-notes-1.42(1) — Release notes for older 1.42 release.

2. Convert 6 guests from different VMware env via vddk+rhv-upload at the same time to compare the performance between virt-v2v-1.45.91-1 and virt-v2v-1.45.3-3.

v2v_version   ESXi7.0+vddk7.0.2        ESXi7.0+vddk6.7          ESXi7.0+vddk6.5               

1.45.93-1     Convert guest: 4m14s     Convert guest: 4m22s     Convert guest: 2m18s
              Copying disk:  6m56s     Copying disk:   5m21s    Copying disk:  7m51s  
                               
 
1.45.3-3      Convert guest: 2m19s     Convert guest: 2m55s     Convert guest: 2m14s 
              Copying disk:  11m46s    Copying disk:  7m22s     Copying disk:  3m51s 

-------------------------------------------------------------------------------------------

v2v_version   ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7          ESXi6.7+vddk6.5      
          
1.45.93-1     Convert guest: 4m19s     Convert guest: 4m33s     Convert guest: 6m23s
              Copying disk:  11m31s    Copying disk:  7m2s      Copying disk:  9m51s 


1.45.3-3      Convert guest: 1m27s    Convert guest: 1m35s      Convert guest: 3m44s
              Copying disk:  14m2s    Copying disk:  11m47s     Copying disk:  9m48s 


Hi rjones,
   
   (1) Please check the result of step1, the release-notes info is not removed or updated yet in virt-v2v man page
   (2) Please check the result of step2, the copying disk performance of virt-v2v-1.45.93-1 is much better than virt-v2v-1.45.3-3 when vddk version is larger than 6.5, but the copying disk performance of virt-v2v-1.45.93-1 is slightly inferior than virt-v2v-1.45.3-3 when vddk version is 6.5, besides， the converting performance of virt-v2v-1.45.93-1 is still not good as virt-v2v-1.45.3-3.
Comment 10 Richard W.M. Jones 2021-12-07 10:21:59 UTC 
>    (1) Please check the result of step1, the release-notes info is not removed or updated yet in virt-v2v man page

This is OK - the release notes are not yet ready upstream.

>    (2) ...

The performance should be improved in the next version (1.45.94) available
later today.

I'm not worried about the performance of VDDK <= 6.5 since customers can easily
upgrade VDDK versions if they have performance problems, and VDDK <= 6.5 doesn't
support extents and asynch which is essential for good performance.
Comment 12 Richard W.M. Jones 2021-12-07 13:44:57 UTC 
Sorry, one further fix is required, I'll add it to the RHEL 9 build
and do another respin.

https://github.com/libguestfs/virt-v2v/commit/ac59d3b2310511b1537d408b675b19ec9a5d384e
Comment 13 Richard W.M. Jones 2021-12-07 14:18:35 UTC 
Another note:

Old virt-v2v (using qemu-img convert) did not flush the file to disk in
output modes that write to local storage (eg. -o local, -o libvirt, -o openstack
and others).  Modular virt-v2v (using nbdcopy) always tries to flush the
output at the end.

In my testing this can sometimes take quite some time: eg. a 15GB Windows
image that I'm testing took 53 seconds to flush out at the end, so obviously
new virt-v2v appeared to be quite a bit slower than old virt-v2v, even though
they were basically taking the same time.  It depends on the speed of the
local disk, so with faster local disks it may not be as noticable.

I talked to Eric about this and I think I'm not going to change this.  We
want virt-v2v to flush data on exit, since it is safer.
Comment 14 mxie@redhat.com 2021-12-10 10:07:23 UTC 
Test the bug with below builds:
virt-v2v-1.45.94-3.el9.x86_64
libguestfs-1.46.0-5.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
nbdkit-server-1.28.3-2.el9.x86_64
libvirt-libs-7.10.0-1.el9.x86_64
qemu-img-6.1.0-8.el9.x86_64
virtio-win-1.9.19-5.el9_b.noarch


Steps:
1. Convert 4 guests from different VMware env via vddk+rhv-upload at the same time to compare the performance between virt-v2v-1.45.94-3 and virt-v2v-1.45.3-3

v2v_version   ESXi7.0+vddk7.0.2       ESXi7.0+vddk6.7        ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7     

1.45.94-3     Convert guest: 1m24s    Convert guest: 1m40s   Convert guest: 52s       Convert guest: 1m1s     
              Copying disk:  13m54s   Copying disk:  8m52s   Copying disk:  17m33s    Copying disk:  17m58s  
                               
 
1.45.3-3      Convert guest: 1m39s    Convert guest: 1m39s   Convert guest: 44s       Convert guest: 56s 
              Copying disk:  8m10s    Copying disk:  5m54s   Copying disk:  11m20s    Copying disk:   9m34s 
 


3. Convert guests with below  ways to compare the performance between virt-v2v-1.45.94-3 and virt-v2v-1.45.3-3
-----------------------------------------------------------------------------
v2v_version     rhv_to_json               ova_to_rhv 
          
1.45.94-3      Convert guest: 15m46s     Convert guest: 54s
               Copying disk:  8m24s      Copying disk:  16m39s 


1.45.3-3       Convert guest: 9m42s      Convert guest: 53s
               Copying disk:  49m4s      Copying disk:  5m3s 
------------------------------------------------------------------------------ 

Hi rjones,

    (1)Converting performance of virt-v2v-1.45.94-3 is almost same with virt-v2v-1.45.3-3 when convert guest from vmware via vddk now, but converting performance of virt-v2v-1.45.94-3 is still worse than virt-v2v-1.45.3-3 when convert guest from vmware without vddk
    (2)Seems copying disk performance of virt-v2v-1.45.94-3 is regression again when target is rhv, copying disk performance of virt-v2v-1.45.94-3 is better than virt-v2v-1.45.3-3 only when target is local
    (3)BTW, debug log of virt-v2v-1.45.94-3 has no virt-v2v and libvirt version, which is not convenient to debug problem I think, is it expected?
Comment 16 Richard W.M. Jones 2021-12-13 15:23:08 UTC 
(1) There's still a bit more of a difference / slowdown than I want.  Note
that new virt-v2v always flushes the data to disk, old virt-v2v did not.
It may be that this would make a difference (try "sync" after running virt-v2v),
but I don't think that's the complete answer.

(2) I will have to try copying rhv when I get back in the new year.

(3) Yes, that is a bug / regression, please file it!  Thanks.
Comment 17 mxie@redhat.com 2021-12-14 04:14:17 UTC 
(In reply to Richard W.M. Jones from comment #16)
> (1) There's still a bit more of a difference / slowdown than I want.  Note
> that new virt-v2v always flushes the data to disk, old virt-v2v did not.
> It may be that this would make a difference (try "sync" after running
> virt-v2v),
> but I don't think that's the complete answer.

Does v2v flush the data to disk in version 1.45.93-1 because this version has good copying disk performance during conversion.

> (3) Yes, that is a bug / regression, please file it!  Thanks.
File bug2032112 to track, thanks.
Comment 18 Richard W.M. Jones 2021-12-14 09:04:04 UTC 
Yes all versions >= 1.45.9x flush the disk to permanent storage at the end.

At some point I'm going to need to put all these results in a big
spreadsheet because tracking them across BZ comments is hard.  A
job for January.
Comment 19 mxie@redhat.com 2022-01-11 11:27:57 UTC 
Test the bug with below builds:
virt-v2v-1.45.96-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
nbdkit-server-1.28.4-1.el9.x86_64
libvirt-libs-8.0.0-0rc1.1.el9.x86_64
qemu-img-6.2.0-3.el9.x86_64
virtio-win-1.9.19-5.el9_b.noarch
python3-ovirt-engine-sdk4-4.4.15-1.el9ev.x86_64


Steps:
1. Convert 4 guests from different VMware via vddk+rhv-upload at the same time to compare the performance between virt-v2v-1.45.96-1 and virt-v2v-1.45.3-3

v2v_version   ESXi7.0+vddk7.0.2        ESXi7.0+vddk6.7         ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7     

1.45.96-1     Convert guest: 1m2s      Convert guest: 3m32s    Convert guest: 1m5s      Convert guest: 5m2s     
              Copying disk:  41m45s    Copying disk:  36m10s   Copying disk:  40m2s     Copying disk:  36m19s  
                               
 
1.45.3-3      Convert guest: 1m8s      Convert guest: 3m7s     Convert guest: 1m8s      Convert guest: 4m13s 
              Copying disk:  10m47s    Copying disk:  9m58s    Copying disk:  9m45s     Copying disk:  9m59s 
 


3. Convert guests with below  ways to compare the performance between virt-v2v-1.45.94-3 and virt-v2v-1.45.3-3
-----------------------------------------------------------------------------
v2v_version     rhv_to_json               vmx+ssh_to_local 
          
1.45.96-1      Convert guest: 16m46s      Convert guest: 1m22s
               Copying disk:  13m30s      Copying disk: 50s 


1.45.3-3       Convert guest: 9m48s       Convert guest: 1m10s
               Copying disk:  8m20s       Copying disk:  2m9s 
------------------------------------------------------------------------------ 

Test result:

    (1)Converting performance of virt-v2v-1.45.96-1 is almost same with virt-v2v-1.45.3-3 when convert guest from vmware via vddk and vmx+ssh 
    (2)Converting performance of virt-v2v-1.45.96-1 is not as good as virt-v2v-1.45.3-3 when convert guest from vmware without vddk, use bug2039255 to track
    (3)Copying disk performance of virt-v2v-1.45.96-1 is better than virt-v2v-1.45.3-3 when target is local
    (4)Copying disk performance of virt-v2v-1.45.96-1 is not as good as virt-v2v-1.45.3-3 when convert guest to rhv via rhv-upload, use bug2039255 to track
    (5)Copying disk performance of virt-v2v-1.45.96-1 is not as good as virt-v2v-1.45.3-3 when target is json, use bug2039255 to track

    On the whole, virt-v2v-1.45.96-1.el9.x86_64 can convert guest successfully except bug2027598
Comment 21 mxie@redhat.com 2022-01-26 04:16:49 UTC 
Verify the bug with below builds:
virt-v2v-1.45.97-2.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
libvirt-libs-8.0.0-1.el9.x86_64
qemu-img-6.2.0-4.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.3-1.el9.x86_64
virtio-win-1.9.19-5.el9_b.noarch


Steps:
1.Convert a guest from VMware host via vddk to rhv by v2v
# virt-v2v -ic esx://root.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=01:53:47:87:7E:09:3E:C4:7A:68:1B:A8:EF:84:93:1E:36:17:45:AE --password-file /home/esxpw   -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   0.0] Setting up the source: -i libvirt -ic esx://root.199.217/?no_verify=1 -it vddk esx7.0-rhel7.9-rhel8.4-multiple_linux_os
[   1.6] Opening the source
[  25.0] Inspecting the source

***
Dual- or multi-boot operating system detected.  Choose the root filesystem
that contains the main operating system from the list below:

 [1] /dev/rhel_bootp-73-199-22/root (Red Hat Enterprise Linux Server 7.9 (Maipo))
 [2] /dev/rhel_bootp-73-199-6/root (Red Hat Enterprise Linux 8.4 (Ootpa))

Enter a number between 1 and 2, or ‘exit’: 2
[  51.0] Checking for sufficient free disk space in the guest
[  51.0] Converting Red Hat Enterprise Linux 8.4 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 125.1] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-sda1 failed.  Usually you can ignore this 
message.  To find out more read "Trimming" in virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
virt-v2v: warning: fstrim on guest filesystem 
/dev/mapper/osprober-linux-rhel_bootp--73--199--22-root failed.  Usually 
you can ignore this message.  To find out more read "Trimming" in 
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: FITRIM ioctl failed: Operation 
not permitted
[ 127.4] Closing the overlay
[ 127.7] Assigning disks to buses
[ 127.7] Checking if the guest needs BIOS or UEFI to boot
[ 127.7] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 149.0] Copying disk 1/1
█ 100% [****************************************]
[1096.8] Creating output metadata
[1159.9] Finishing off

2.Convert a guest from Xen to local by v2v
# virt-v2v -ic xen+ssh://root.224.33/?no_verify=1 xen-hvm-rhel7.9-x86_64 
[   0.0] Setting up the source: -i libvirt -ic xen+ssh://root.224.33/?no_verify=1 xen-hvm-rhel7.9-x86_64
[   2.2] Opening the source
[   7.5] Inspecting the source
[  20.0] Checking for sufficient free disk space in the guest
[  20.0] Converting Red Hat Enterprise Linux Server 7.9 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 159.5] Mapping filesystem data to avoid copying unused and blank areas
[ 160.4] Closing the overlay
[ 160.7] Assigning disks to buses
[ 160.7] Checking if the guest needs BIOS or UEFI to boot
[ 160.7] Setting up the destination: -o libvirt
[ 163.6] Copying disk 1/1
█ 100% [****************************************]
[ 282.8] Creating output metadata
[ 282.9] Finishing off

3.Convert a guest from VMware to rhv with adding -oo rhv-verifypeer=true and -oo rhv-cafile=/home/ca.pem 
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  -ip /home/passwd esx7.0-rhel9.0-x86_64 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -oo rhv-cluster=Default -os nfs_data -b ovirtmgmt -oo rhv-verifypeer=true -oo rhv-cafile=/home/ca.pem 
[   0.0] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel9.0-x86_64
[   1.8] Opening the source
[   7.0] Inspecting the source
[  12.4] Checking for sufficient free disk space in the guest
[  12.4] Converting Red Hat Enterprise Linux 9.0 Alpha (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 107.0] Mapping filesystem data to avoid copying unused and blank areas
[ 107.9] Closing the overlay
[ 108.2] Assigning disks to buses
[ 108.2] Checking if the guest needs BIOS or UEFI to boot
[ 108.2] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 129.1] Copying disk 1/1
█ 100% [****************************************]
[ 716.5] Creating output metadata
[ 779.2] Finishing off

4.Convert a guest from VMware via vmx+ssh to rhv by v2v
# virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/esx6.7-function/Auto-esx6.7-rhel7.7-multi-disks/Auto-esx6.7-rhel7.7-multi-disks.vmx -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt 
[   0.0] Setting up the source: -i vmx ssh://root.75.219/vmfs/volumes/esx6.7-function/Auto-esx6.7-rhel7.7-multi-disks/Auto-esx6.7-rhel7.7-multi-disks.vmx
[  28.4] Opening the source
[  37.1] Inspecting the source
[  50.6] Checking for sufficient free disk space in the guest
[  50.6] Converting Red Hat Enterprise Linux Server 7.7 Beta (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 199.6] Mapping filesystem data to avoid copying unused and blank areas
[ 202.8] Closing the overlay
[ 203.4] Assigning disks to buses
[ 203.4] Checking if the guest needs BIOS or UEFI to boot
[ 203.4] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 459.7] Copying disk 1/21
█ 100% [****************************************]
[ 958.5] Copying disk 2/21
█ 100% [****************************************]
[ 974.8] Copying disk 3/21
█ 100% [****************************************]
[ 991.0] Copying disk 4/21
█ 100% [****************************************]
[1008.0] Copying disk 5/21
█ 100% [****************************************]
[1024.6] Copying disk 6/21
█ 100% [****************************************]
[1041.2] Copying disk 7/21
█ 100% [****************************************]
[1057.8] Copying disk 8/21
█ 100% [****************************************]
[1074.3] Copying disk 9/21
█ 100% [****************************************]
[1091.0] Copying disk 10/21
█ 100% [****************************************]
[1107.8] Copying disk 11/21
█ 100% [****************************************]
[1124.4] Copying disk 12/21
█ 100% [****************************************]
[1141.3] Copying disk 13/21
█ 100% [****************************************]
[1158.0] Copying disk 14/21
█ 100% [****************************************]
[1174.6] Copying disk 15/21
█ 100% [****************************************]
[1191.2] Copying disk 16/21
█ 100% [****************************************]
[1208.0] Copying disk 17/21
█ 100% [****************************************]
[1224.7] Copying disk 18/21
█ 100% [****************************************]
[1241.3] Copying disk 19/21
█ 100% [****************************************]
[1258.2] Copying disk 20/21
█ 100% [****************************************]
[1275.0] Copying disk 21/21
█ 100% [****************************************]
[1291.7] Creating output metadata
[1499.6] Finishing off

5.Check the guests of above steps after v2v conversion, checkpoints of guest are passed

Result:
  No problem found, move the bug from ON_QA to VERIFIED
Comment 23 errata-xmlrpc 2022-05-17 13:41:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: virt-v2v), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2566