FWIW some upstream analysis:

https://listman.redhat.com/archives/libguestfs/2022-January/thread.html#00055
https://listman.redhat.com/archives/libguestfs/2022-January/thread.html#00057
https://listman.redhat.com/archives/libguestfs/2022-January/thread.html#00058

So one thing that came out of the upstream analysis is that modular
virt-v2v always flushes the data out to disk, whereas old virt-v2v
did not do that.

As a result, to get fair comparisons you *must* do a "sync" after
virt-v2v, and include the time taken for sync in the total time.  For
example:

  $ virt-v2v -i ... -o ...
  $ time sync

  real    0m48.795s
  user    0m0.000s
  sys     0m0.062s

And add 48 seconds to the total time.

It probaby won't close the gap given the large differences shown in comment 0.

I'm not sure if we have a specific bug for -i disk -> -o rhv-upload, but
I'm finally able to reproduce the slow down in this case locally, and
it is very clear and reproducible.

$ time ./run virt-v2v -i disk fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test3 

  Virt-v2v 1.44.2:           1m22
  Virt-v2v 1.45.97@18b11018: 2m20

The test guest is the standard performance guest described here:
https://listman.redhat.com/archives/libguestfs/2022-January/msg00055.html

Not much clue what's going on here, but I posted some questions upstream:
https://listman.redhat.com/archives/libguestfs/2022-February/msg00109.html

It turns out that we were tickling a 60 second timeout in the oVirt code
because modular virt-v2v changed the order in which some operations were
done (in a neutral way - this is really a problem in oVirt).  Long story
is here: https://listman.redhat.com/archives/libguestfs/2022-February/thread.html#00111

I have a small patch which restores performance so it's now about the
same as old virt-v2v within measurement error (sometimes a bit faster).

Virt-v2v 1.44.2:

$ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410.home.annexia.org/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test11 -of raw 
[   0.4] Opening the source -i disk /var/tmp/fedora-35.qcow2
[   0.5] Creating an overlay to protect the source from being modified
[   0.5] Opening the overlay
[   8.3] Inspecting the overlay
[  10.9] Checking for sufficient free disk space in the guest
[  10.9] Estimating space required on target for each disk
[  10.9] Converting Fedora Linux 35 (Thirty Five) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[  36.9] Mapping filesystem data to avoid copying unused and blank areas
[  37.7] Closing the overlay
[  38.4] Assigning disks to buses
[  38.4] Checking if the guest needs BIOS or UEFI to boot
[  38.4] Initializing the target -o rhv-upload -oc https://ovirt4410.home.annexia.org/ovirt-engine/api -op /tmp/ovirt-passwd -os ovirt-data
[  39.7] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/run/user/1000/v2vnbdkit.kQgYQb/nbdkit3.sock", "file.export": "/" } (raw)
    (100.00/100%)
[  72.8] Creating output metadata
[  73.3] Finishing off

real	1m13.644s
user	0m1.832s
sys	0m4.845s

Virt-v2v 1.45.97 + patch:

$ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410.home.annexia.org/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test10 -of raw
[   0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2
[   1.0] Opening the source
[   9.0] Inspecting the source
[  11.7] Checking for sufficient free disk space in the guest
[  11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM
virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown 
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[  38.1] Mapping filesystem data to avoid copying unused and blank areas
[  39.1] Closing the overlay
[  39.7] Assigning disks to buses
[  39.7] Checking if the guest needs BIOS or UEFI to boot
[  39.7] Setting up the destination: -o rhv-upload -oc https://ovirt4410.home.annexia.org/ovirt-engine/api -os ovirt-data
[  59.5] Copying disk 1/1
█ 100% [****************************************]
[  71.2] Creating output metadata
[  74.1] Finishing off

real	1m14.365s
user	0m8.183s
sys	0m13.769s

The patch is here:

https://listman.redhat.com/archives/libguestfs/2022-February/msg00128.html

This patch is now upstream in:
https://github.com/libguestfs/virt-v2v/commit/d69ba56b2f4bc642ce59bfc6bdd5c137480bf8c3

I have a further question for Nir and it may be possible to gain
significantly more performance, so I'm going to leave this bug in
POST for now:
https://listman.redhat.com/archives/libguestfs/2022-February/msg00130.html

I've created an upstream oVirt bug to discuss the slow disk creation
problem: bug 2053103

Refer to comment 9,move the bug back to ASSIGNED.

(In reply to Richard W.M. Jones from comment #3)
> So one thing that came out of the upstream analysis is that modular
> virt-v2v always flushes the data out to disk, whereas old virt-v2v
> did not do that.

Old virt-v2v used qemu-img convert, and it always flushes at the end. It would be
a bad if it didn't.

Here is an example:

$ nbdkit -f -v file file=dst.raw 2>&1 | grep flush
nbdkit: file[1]: debug: file: can_flush
nbdkit: file.0: debug: file: flush

$ qemu-img convert -n -W fedora-35.raw nbd://localhost

flush was called.

When using old virt-v2v, we always had 2 flushes in imageio logs. One flush
came from qemu-img, the other one from nbdkit (bug).

> As a result, to get fair comparisons you *must* do a "sync" after
> virt-v2v, and include the time taken for sync in the total time.  For
> example:
> 
>   $ virt-v2v -i ... -o ...
>   $ time sync
> 
>   real    0m48.795s
>   user    0m0.000s
>   sys     0m0.062s
> 
> And add 48 seconds to the total time.

This should not be needed to compare times.

(In reply to mxie from comment #9)
I think we need logs to understand whats going on. I'm not sure about the flows
involving vddk, but the the local import to rhv, it wil help if you attach
here the v2v log and imageio logs from the host performing the import.

(In reply to Nir Soffer from comment #11)
> (In reply to Richard W.M. Jones from comment #3)
> > So one thing that came out of the upstream analysis is that modular
> > virt-v2v always flushes the data out to disk, whereas old virt-v2v
> > did not do that.
> 
> Old virt-v2v used qemu-img convert, and it always flushes at the end. It
> would be
> a bad if it didn't.
> 
> Here is an example:
> 
> $ nbdkit -f -v file file=dst.raw 2>&1 | grep flush
> nbdkit: file[1]: debug: file: can_flush
> nbdkit: file.0: debug: file: flush
> 
> $ qemu-img convert -n -W fedora-35.raw nbd://localhost
> 
> flush was called.

This isn't always true.  Old virt-v2v in modes such as -o local actually
did something like this (ie. writing directly to the output file):

$ qemu-img convert -n -W overlay.qcow2 guest.img

If you strace the qemu-img command you'll see it doesn't fsync the output.

You can also try this with virt-v2v 1.44.2:

$ sync; ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o local -os /var/tmp/ -of raw; time sync

and you'll see the final sync command takes a few seconds (depending on the size of
the input and speed of the disk).

I think qemu-img convert behaves differently if the output is an NBD server.
It appears if the server advertises flush then it will send it at the end.

> When using old virt-v2v, we always had 2 flushes in imageio logs. One flush
> came from qemu-img, the other one from nbdkit (bug).
> 
> > As a result, to get fair comparisons you *must* do a "sync" after
> > virt-v2v, and include the time taken for sync in the total time.  For
> > example:
> > 
> >   $ virt-v2v -i ... -o ...
> >   $ time sync
> > 
> >   real    0m48.795s
> >   user    0m0.000s
> >   sys     0m0.062s
> > 
> > And add 48 seconds to the total time.
> 
> This should not be needed to compare times.

Maybe not for -o rhv-upload, but it definitely is for other outputs.

(In reply to Nir Soffer from comment #12)
> (In reply to mxie from comment #9)
> I think we need logs to understand whats going on. I'm not sure about the
> flows
> involving vddk, but the the local import to rhv, it wil help if you attach
> here the v2v log and imageio logs from the host performing the import.

Yes I'd like to see the logs too.

(In reply to Richard W.M. Jones from comment #13)
> (In reply to Nir Soffer from comment #11)
> > (In reply to Richard W.M. Jones from comment #3)
> > > So one thing that came out of the upstream analysis is that modular
> > > virt-v2v always flushes the data out to disk, whereas old virt-v2v
> > > did not do that.
> > 
> > Old virt-v2v used qemu-img convert, and it always flushes at the end. It
> > would be
> > a bad if it didn't.
> > 
> > Here is an example:
> > 
> > $ nbdkit -f -v file file=dst.raw 2>&1 | grep flush
> > nbdkit: file[1]: debug: file: can_flush
> > nbdkit: file.0: debug: file: flush
> > 
> > $ qemu-img convert -n -W fedora-35.raw nbd://localhost
> > 
> > flush was called.
> 
> This isn't always true.  Old virt-v2v in modes such as -o local actually
> did something like this (ie. writing directly to the output file):
>
> $ qemu-img convert -n -W overlay.qcow2 guest.img
> 
> If you strace the qemu-img command you'll see it doesn't fsync the output.

Yes, this very bad:

$ strace -f -e fdatasync qemu-img convert fedora-35.raw dst.raw 2>&1 | grep fdatasync

$ strace -f -e fdatasync qemu-img convert -t unsafe fedora-35.raw dst.raw 2>&1 | grep fdatasync

Fortunately RHV always use -t none so we always have a flush:

$ strace -f -e fdatasync qemu-img convert -t writeback fedora-35.raw dst.raw 2>&1 | grep fdatasync
[pid 30951] fdatasync(8)                = 0

$ strace -f -e fdatasync qemu-img convert -t none fedora-35.raw dst.raw 2>&1 | grep fdatasync
[pid 31187] fdatasync(8)                = 0

Kevin, avoiding flushes during the copy makes sense, but flushing at the end sounds
like a better default to me for the use case of qemu-img convert. Should we file
qemu-img bug for this?

(In reply to mxie from comment #14)
> (In reply to Nir Soffer from comment #12)
> > (In reply to mxie from comment #9)
> > I think we need logs to understand whats going on. I'm not sure about the
> > flows
> > involving vddk, but the the local import to rhv, it wil help if you attach
> > here the v2v log and imageio logs from the host performing the import.
> 
> All v2v conversions of comment9 are executed on standalone v2v server rather
> than rhv node and v2v debugs logs of comment9 are in
> http://fileshare.englab.nay.redhat.com/pub/section3/libvirtmanual/mxie/pre-
> verify-bug2039255/

Are you using -oo rhv-direct=true when using -o rhv-upload?

If you don't every request to imageio server go via the imageio proxy on 
engine host. This is typically 50%s slower compare with sending to directly
to the host.

With small requests (moduular virt-v2v use 256k instead of 2m), this create
huge overhead that may explain the slow down.

This is also *not* the recommended usage and testing this in context of
performance testing is not good use of our time.

When testing performance should always use:

    virt-v2v -o rhv-upload -oo rhv-direct=true

And we should run virt-v2v on a RHV hypervisor node (not on the manager node).

Testing on another host which is not part of the cluster is nice to have.

Testing without -oo rhv-direct is nice to have for completeness, but I don't
think we should spend time on this use case.

> Are you using -oo rhv-direct=true when using -o rhv-upload?
> If you don't every request to imageio server go via the imageio proxy on 
> engine host. This is typically 50%s slower compare with sending to directly
> to the host.
> 
> With small requests (moduular virt-v2v use 256k instead of 2m), this create
> huge overhead that may explain the slow down.
> 
> This is also *not* the recommended usage and testing this in context of
> performance testing is not good use of our time.
> 
> When testing performance should always use:
> 
>     virt-v2v -o rhv-upload -oo rhv-direct=true

Hi Richard, do you think it's necessary to add -oo rhv-direct=true for rhv-upload conversions to retest the step1 of commnet9?

> And we should run virt-v2v on a RHV hypervisor node (not on the manager
> node).
> Testing on another host which is not part of the cluster is nice to have.

The v2v version to be tested is rhel9 build but rhv4.4 node is based on rhel8, so can't run v2v on rhv node

My testing is using 4 separate machines:

VMware ESXi ----> virt-v2v ----> oVirt host (& oVirt Engine separately)
                  Fedora 36       RHEL 8.5        RHEL 8.5

I'm _not_ using rhv-direct, because I thought that this only works when you
run virt-v2v on the oVirt host.  Also for the same reason as mxie above,
I cannot test recent virt-v2v on RHEL 8.

I'll also note: https://bugzilla.redhat.com/show_bug.cgi?id=2033096

I should say that when comparing virt-v2v 1.44 and virt-v2v 1.45.98,
I'm running both on the same Fedora 36 / Rawhide.  So even if using
the rhv proxy is bad, I'm not sure it can be the cause of the problem.

Sorry, I accidentally cancelled the needinfo set for Kevin on comment 15.

(In reply to Richard W.M. Jones from comment #18)
> My testing is using 4 separate machines:
> 
> VMware ESXi ----> virt-v2v ----> oVirt host (& oVirt Engine separately)
>                   Fedora 36       RHEL 8.5        RHEL 8.5
> 
> I'm _not_ using rhv-direct, because I thought that this only works when you
> run virt-v2v on the oVirt host.  Also for the same reason as mxie above,
> I cannot test recent virt-v2v on RHEL 8.

-oo rhv-direct=true works and should be the default in virt-v2v to avoid this
confusion.

(In reply to Richard W.M. Jones from comment #19)
> I should say that when comparing virt-v2v 1.44 and virt-v2v 1.45.98,
> I'm running both on the same Fedora 36 / Rawhide.  So even if using
> the rhv proxy is bad, I'm not sure it can be the cause of the problem.

I think amplify the problem - for every nbd command, we:
1. send http request to the proxy
2. the proxy send http request to the host
3. the host send nbd command to qemu-nbd
4. the host return reply to the proxy
5. the proxy return reply to us

I see 1.8x speed up when using local virt-v2v, communicating with imageio
via unix socke by increasing request size from 256k to 4m. When working
with remove server via a proxy, the overhead for each request is much larger.

Of course there may be an issue on the input side, which changed a lot, 
but comment 9 show clear issue on the output side:

> v2v_version     local_to_rhv_upload              
>          
> 1.45.98-1      Convert guest: 2m49s     
>                Copying disk:  7m57s     
>
>
> 1.45.3-3       Convert guest: 2m29s      
>                Copying disk:  3m51s

(In reply to Nir Soffer from comment #21)
> -oo rhv-direct=true works and should be the default in virt-v2v to avoid this
> confusion.

I just realised that I _am_ in fact using -oo rhv-direct!  It was hiding
in the very long (6 line) virt-v2v command I'm using.

If it works, then yes we should use it, and as you can see from bug 2033096
the aim is to make that the default.

Out of interest I just reran my tests with and without -oo rhv-direct
to see if it could account for the difference.  Results below for me.
These are all local vddk -> -o rhv-upload, using VDDK 7.0.3 and ESXi 7:

                      -oo rhv-direct=true     -oo rhv-direct=false
  virt-v2v 1.45.98     5m15                    5m19
  virt-v2v 1.45.3      6m11                    5m51
  virt-v2v 1.44.2      5m11                    4m59

At the moment I cannot reliably reproduce this bug.

(In reply to Richard W.M. Jones from comment #23)
> Out of interest I just reran my tests with and without -oo rhv-direct
> to see if it could account for the difference.  Results below for me.
> These are all local vddk -> -o rhv-upload, using VDDK 7.0.3 and ESXi 7:
> 
>                       -oo rhv-direct=true     -oo rhv-direct=false
>   virt-v2v 1.45.98     5m15                    5m19
>   virt-v2v 1.45.3      6m11                    5m51
>   virt-v2v 1.44.2      5m11                    4m59

The timing look identical - looks like -oo rhv-direct is broken. We either
always use direct more or never.

Do you have imageio logs for these tests? We see the client address in
the CLOSE log:

    2022-02-14 19:28:00,215 INFO    (Thread-125) [http] CLOSE connection=125 client=::ffff:192.168.122.10 ...

With -oo rhv-direct=true:

client=local
client=local
client=::ffff:192.168.0.139
client=local
client=local
client=::ffff:192.168.0.139
client=::ffff:192.168.0.139
client=::ffff:192.168.0.139
client=::ffff:192.168.0.139
client=local

(192.168.0.139 == IP address of machine running virt-v2v)

With -oo rhv-direct=false:

client=local
client=local
client=::ffff:192.168.0.210
client=local
client=local
client=local
client=::ffff:192.168.0.210
client=::ffff:192.168.0.210
client=::ffff:192.168.0.210
client=::ffff:192.168.0.210

(192.168.0.210 == IP address of oVirt engine)

(In reply to Richard W.M. Jones from comment #26)
Your test looks correct. So this show that the bottleneck is the input
side - if you send data slow enough, it does not matter if you use the proxy
or not.

Because nbdcopy uses async read and writes, the reads are never blocked by
slow writes. In imageio client we have 4 threads, but every one is block
either on read or on write, so slow write slows also reading from source.

This is what I see on my test system, using oVirt 4.5, uploading local file
(fedora 35 + 3g of random data) from my laptop to oVirt system running as 
vms on the laptop.

I tested these combinations:

#   combination                                        seconds  
--------------------------------------------------------------
1  Simulating virt-v2v[1] request size 256k             16.98
2  Simulating virt-v2v[1] request size 4m                9.33
3  Simulating virt-v2v[1] via proxy, request size 256k  30.60
4  Simulating virt-v2v[1] via proxy, request size 4m    15.21
5  Normal upload[2] read size 4m                         8.49
6  Normal upload[2] read size 256k                       7.77
7  Normal upload[2] via proxy, read size 4m             14.68
8  Normal upload[2] via proxy, read size 256k           12.63

[1] detecting zeroes and sending small (256k) http requests. Send one HTTP
    request per one NBD read.

[2] The normal upload uses read size only for reading from qemu-nbd, sending one
    HTTP request per extent (splitting large extents to 128m chunks).

You can see that there is a huge difference between sending one request
per nbd read and one request per extent, and using the proxy is much 
slower. Also it the advantage of larger request size is very clear.

(In reply to Nir Soffer from comment #15)
> Kevin, avoiding flushes during the copy makes sense, but flushing at the end
> sounds like a better default to me for the use case of qemu-img convert. Should we
> file qemu-img bug for this?

I'm not sure about the right behaviour there, it completely depends on what you're going to do with the copy next. As this is with 'cache=unsafe', I think it makes sense to optimise for the shortest runtime of qemu-img and avoid the flush - it might be just a temporary file that you continue processing from the page cache and flushing would only be unnecessary overhead. If you do want to have the image safe on disk, you can always call 'sync' on the file next.

Verify the bug with below builds:
virt-v2v-1.45.99-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
libvirt-libs-8.0.0-4.el9.x86_64
qemu-img-6.2.0-8.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.5-1.el9.x86_64
virtio-win-1.9.19-5.el9_b.noarch


Steps:
1. Convert three different guests from ESXi7.0 to rhv(rhv-upload) via different version vddk at the same time to compare the performance between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3, besides, considering add -oo rhv-direct/rhv-proxy option to command line when convert guest with different v2v version

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd  -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt esx7.0-win2022-x86_64 (-oo rhv-direct/rhv-proxy=true)

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.7 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd   -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx7.0-rhel8.5-x86_64 (-oo rhv-direct/rhv-proxy=true)

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.5 -io  vddk-thumbprint=xx:xx:xx:... -ip /home/passwd   -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx7.0-win11-x86_64 (-oo rhv-direct/rhv-proxy=true)

-------------------------------------------------------------------------------------------------
v2v_version                ESXi7.0+vddk7.0.2        ESXi7.0+vddk6.7          ESXi7.0+vddk6.5       

1.45.99-1                  Convert guest: 57s       Convert guest: 2m25s     Convert guest: 46s      
                           Copying disk:  9m44s     Copying disk:  10m4s     Copying disk:  15m57s     
                               
1.45.3-3(rhv-direct=true)  Convert guest: 55s       Convert guest: 2m19s     Convert guest: 1m     
                           Copying disk:  9m30s     Copying disk:  9m25s     Copying disk:  12m48s


--------------------------------------------------------------------------------------------------
v2v_version                ESXi7.0+vddk7.0.2        ESXi7.0+vddk6.7          ESXi7.0+vddk6.5       

1.45.99-1(rhv-proxy=true)  Convert guest: 55s       Convert guest: 2m26s     Convert guest: 55s      
                           Copying disk:  10m33s    Copying disk:  10m33s    Copying disk:  16m7s     
                               
1.45.3-3                   Convert guest: 59s       Convert guest: 2m14s     Convert guest: 54s     
                           Copying disk:  10m26s    Copying disk:  10m17s    Copying disk:  13m40s


Hi Richard,

   According to the current test results, the performance of virt-v2v-1.45.99-1 has been greatly improved, which is basically similar to that of virt-v2v-1.45.3-3, but you can see that the performance of virt-v2v-1.45.99-1 is still a little worse than that of the old version virt-v2v-1.45.3-3 when vddk version is 7.0.2 and 6.7，the performance gap is obvious when vddk version is 6.5, do you think the performance difference is acceptable? If yes, I will continue to test the other scenarios to compare their performance.

Thanks for the testing.  Some broader points first:

 - I'm only really concerned about VDDK 7 & ESXi 7.  By the time people are
   really using this in RHEL 9 (probably 9.1) they will have upgraded to both.

 - We only need to do performance testing with -oo rhv-direct (which is now
   the default).  -oo rhv-proxy is designed for situations where you don't
   have direct network access to the storage, and those are always going to
   be slow because it has to go through a proxy.

 - By the way, -oo rhv-direct[=true] is the default so it's no longer needed.
   -oo rhv-proxy[=true] is the opposite (use a proxy, slow).

I think the performance numbers look fine now.  It's still a few % slower,
but modular virt-v2v is more capable for a few reasons:

 - modular virt-v2v will allow an external program to do the copying, which
   means that whole system performance will be better (eventually, once we
   implement this fully)

 - nbdcopy doesn't trash the page cache when copying to local, again a
   benefit to whole system performance that's not visible for single copies

> the performance gap is obvious when vddk version is 6.5

VDDK 6.5 didn't support extents, so it'll end up copying much more data.
If customers ever hit this case we'll tell them to upgrade to the latest
VDDK, which is usually a simple thing to do.

As VDDK 6.5 didn't support extents, regardless of its impact on virt-v2v-1.45.99-1 performance, continue to verify the bug with below builds:

virt-v2v-1.45.99-1.el9.x86_64
libguestfs-1.46.1-2.el9.x86_64
guestfs-tools-1.46.1-6.el9.x86_64
libvirt-libs-8.0.0-4.el9.x86_64
qemu-img-6.2.0-8.el9.x86_64
nbdkit-1.28.5-1.el9.x86_64
libnbd-1.10.5-1.el9.x86_64
virtio-win-1.9.19-5.el9_b.noarch


Steps:
1. Convert two different guests from ESXi6.7 to rhv(rhv-upload) via vddk7.0.2 and vddk6.7 at the same time to compare the performance between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt esx7.0-win2022-x86_64 

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.7 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx7.0-rhel8.5-x86_64 

-----------------------------------------------------------------------------
v2v_version                ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7                

1.45.99-1                  Convert guest: 1m1s      Convert guest: 2m31s          
                           Copying disk:  9m44s     Copying disk:  10m19s         
                               
1.45.3-3(rhv-direct=true)  Convert guest: 1m2s      Convert guest: 2m23s         
                           Copying disk:  7m34s     Copying disk:  7m36s     

-----------------------------------------------------------------------------
v2v_version                ESXi6.7+vddk7.0.2        ESXi6.7+vddk6.7                

1.45.99-1(rhv-proxy=true)  Convert guest: 51s       Convert guest: 2m26s      
                           Copying disk:  10m4s     Copying disk:  10m41s         
                               
1.45.3-3                   Convert guest: 55s       Convert guest: 2m14s        
                           Copying disk:  7m40s     Copying disk:  7m51s    
-----------------------------------------------------------------------------

2. Convert two different guests from ESXi6.5 to rhv(rhv-upload) via vddk7.0.2 and vddk6.7 at the same time to compare the performance between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.2 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt esx7.0-win2022-x86_64 

#virt-v2v -ic vpx://root@center_ip/data/esxi_host/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk6.7 -io  vddk-thumbprint=xx:xx:xx:...  -ip /home/passwd -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt  esx7.0-rhel8.5-x86_64 

-----------------------------------------------------------------------------
v2v_version                ESXi6.5+vddk7.0.2         ESXi6.5+vddk6.7                

1.45.99-1                  Convert guest: 55s        Convert guest: 4m2s          
                           Copying disk:  12m37s     Copying disk:  33m39s         
                               
1.45.3-3(rhv-direct=true)  Convert guest: 3m35s      Convert guest: 5m17s         
                           Copying disk:  10m12s     Copying disk:  21m26s     

-----------------------------------------------------------------------------
v2v_version                ESXi6.5+vddk7.0.2         ESXi6.5+vddk6.7                

1.45.99-1(rhv-proxy=true)  Convert guest: 53s        Convert guest: 2m28s      
                           Copying disk:  12m58s     Copying disk:  34m1s         
                               
1.45.3-3                   Convert guest: 3m41s      Convert guest: 4m50s        
                           Copying disk:  10m34s     Copying disk:  21m20s    
-----------------------------------------------------------------------------


3.Convert guests in below ways at the same time to compare the performance between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3

3.1 

#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd -o local -os /home esx7.0-win2019-x86_64

#virt-v2v -i vmx -it ssh ssh://root.75.219/vmfs/volumes/esx6.7-matrix/esx6.7-rhel8.5-x86_64/esx6.7-rhel8.5-x86_64.vmx  -ip /home/passwd -o rhv -os 10.73.195.48:/home/nfs_export 

-----------------------------------------------------------------------------
v2v_version     VMware_to_local               vmx+ssh_to_rhv 
          
1.45.99-1      Convert guest: 22m31s       Convert guest: 2m3s
               Copying disk:  74m49s       Copying disk:  6m38s 


1.45.3-3       Convert guest: 10m25s       Convert guest: 2m
               Copying disk:  42m41s       Copying disk:  5m15s 
-----------------------------------------------------------------------------

3.2 The performance gap between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3 is too large in step3.1, so convert another guest from VMware to local without vddk to compare their performance again

#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd -o local -os /home esx7.0-sles15sp2-x86_64 

-----------------------------------------------------------------------------
v2v_version     VMware_to_local               

1.45.99-1       Convert guest: 13m38s      
                Copying disk:  13m17s       


1.45.3-3        Convert guest: 9m55s       
                Copying disk:  8m13s      
-----------------------------------------------------------------------------


4.Convert a guest from ova file to openstack to compare the performance between virt-v2v-1.45.99-1 and virt-v2v-1.45.3-3

#virt-v2v -i ova /media/tools/ova-images/esx7.0-win2019-x86_64 -o openstack -oo server-id=v2v-appliance 

-----------------------------------------------------------------------------
v2v_version     ova_to_openstack               

1.45.99-1       Convert guest: 3m59s      
                Copying disk:  6m53s       


1.45.3-3        Convert guest: 3m51s       
                Copying disk:  7m5s      
-----------------------------------------------------------------------------



Hi Richard,

   (1)Please check the result of step1, the performance of virt-v2v-1.45.99-1 is a little worse than that of virt-v2v-1.45.3-3 when convert guest from ESXi6.7 to rhv(rhv-upload) with vddk7.0.2 and vddk6.7,，their performance gap is almost between 2 and 3 minutes.
   
   (2)Please check the result of step2, when convert guest from ESXi6.5 to rhv(rhv-upload) with vddk the performance of virt-v2v-1.45.99-1 is basically similar to that of virt-v2v-1.45.3-3 if vddk version is 7.0.2，but the performance of virt-v2v-1.45.99-1 is much worse than that of virt-v2v-1.45.3-3 if vddk version is 6.7.

   (3)Please check the result of step3, can see that the performance of virt-v2v-1.45.99-1 is much worse than that of virt-v2v-1.45.3-3 when convert guest from VMware to local without vddk.

Thanks for doing this comprehensive testing.  I want to chop out some tests
that I don't think are useful for performance testing:

- Anything that uses rhv-proxy (!rhv-direct) is always going to be slow
  because all data goes through a proxy, so there's no point testing it.
  This mode is only needed for corner cases where you don't have direct
  access to the ovirt hosts.

- Also anything with VDDK < 7.0 is not worth testing because even with
  ancient RHV or VMware it's easy to upgrade VDDK.

- Also any test that doesn't use VDDK (ie. is using ssh or https) as these
  are known to be much slower and we only provide them for people who don't
  or can't use VDDK for licensing reasons.

(NB I'm just talking about *performance* testing.  We still need to test
that all these different modes work.)

That leaves these performance tests:

(from Step 1)

v2v_version                ESXi6.7+vddk7.0.2

1.45.99-1                  Convert guest: 1m1s
                           Copying disk:  9m44s
                               
1.45.3-3(rhv-direct=true)  Convert guest: 1m2s
                           Copying disk:  7m34s

 - Modular virt-v2v is about 25% slower, and there's no obvious reason looking
   at the logs.  I suspect that tuning the request size might help.  Let's look
   at this in RHEL 9.1.

(from Step 2)

v2v_version                ESXi6.5+vddk7.0.2

1.45.99-1                  Convert guest: 55s
                           Copying disk:  12m37s
                               
1.45.3-3(rhv-direct=true)  Convert guest: 3m35s
                           Copying disk:  10m12s

 - Modular virt-v2v is marginally faster overall.  What's odd about this is
   actually that old virt-v2v took so long to do the conversion.  Old virt-v2v
   spends ages doing QueryAllocatedBlocks requests during conversion (modular
   virt-v2v does not).  I'm not sure I understand what's going on there.

(from Step 3.1)

 - Tests use curl or ssh, not VDDK.

(from Step 3.2)

 - Tests use curl, not VDDK.

(from Step 4)

v2v_version     ova_to_openstack               

1.45.99-1       Convert guest: 3m59s      
                Copying disk:  6m53s       

1.45.3-3        Convert guest: 3m51s       
                Copying disk:  7m5s

 - There's not a very large difference here, but -i ova in modular virt-v2v is
   known to be a bit slower.  We can work on making this better for RHEL 9.1.

My conclusion is I'm not very worried :-)  Nothing is broken.  The performance
differences are small.  We understand well which input and output modes are fast
(VDDK) and which are slow (curl & ssh).  We'll work on performance improvements
in RHEL 9.1.

> (from Step 2)
>  - Modular virt-v2v is marginally faster overall.  What's odd about this is
>    actually that old virt-v2v took so long to do the conversion.  Old virt-v2v
>    spends ages doing QueryAllocatedBlocks requests during conversion (modular
>    virt-v2v does not).  I'm not sure I understand what's going on there.

Oh I think I see.  This is ESX 6.5 which had weird problems with extent
mapping.  Note that ESX 6.5 will be out of support in Oct 2022, which
is before RHEL 9.1 is released so I doubt many customers will be using this
combination.

Thanks rjones, I think the bug can be moved to VERIFIED status according to comment36 ~ comment41

(In reply to mxie from comment #38)
> 3.1 
> 
> #virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -ip
> /home/passwd -o local -os /home esx7.0-win2019-x86_64
> 
> #virt-v2v -i vmx -it ssh
> ssh://root.75.219/vmfs/volumes/esx6.7-matrix/esx6.7-rhel8.5-x86_64/
> esx6.7-rhel8.5-x86_64.vmx  -ip /home/passwd -o rhv -os
> 10.73.195.48:/home/nfs_export 
> 
> -----------------------------------------------------------------------------
> v2v_version     VMware_to_local               vmx+ssh_to_rhv 
>           
> 1.45.99-1      Convert guest: 22m31s       Convert guest: 2m3s
>                Copying disk:  74m49s       Copying disk:  6m38s 
> 
> 
> 1.45.3-3       Convert guest: 10m25s       Convert guest: 2m
>                Copying disk:  42m41s       Copying disk:  5m15s 
> -----------------------------------------------------------------------------

I was looking into the cases above (curl -> null|local, ssh -> null|local).
I said above that we don't really care about performance testing here, and that's
true, but I wanted to see if we are missing any easy wins.

I set up a simple test without VMware and compared 1.45.3 and 1.45.99 (methodology
at end).  However I was not able to see a case where modular virt-v2v is slower
beyond measurement errors, and in fact it's a bit faster in some cases:

version       curl -> null    ssh -> null       curl -> local       ssh -> local
1.45.3        190.0           221.5             186.7               220.6
1.45.99       187.5           191.9             187.4               191.8

Methodology:

For both cases, I put a Fedora 20 virt-builder image on to a local web server.

For curl, create /var/tmp/input.xml from the example in the virt-v2v manual,
modifying the disk section:

            <disk type='network' device='disk'>
              <driver name='qemu' type='raw'/>
              <source protocol='http' name='/fedora-20.img'>
                <host name='webserver' port='80'/>
              </source>
              <target dev='hda' bus='ide'/>
            </disk>

For ssh, I modified an existing .vmx file to point to the Fedora
image and hosted that on the same webserver.

$ virt-v2v -i libvirtxml /var/tmp/input.xml [ -o null | -o local -os /var/tmp ]
$ virt-v2v -i vmx -it ssh ssh://webserver/public_html/fedora-20.vmx [ -o null | -o local -os /var/tmp ]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: virt-v2v), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:2566