Bug 1778481

Summary: [v2v] Optimize virt-v2v to eliminate small holes with vmware
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Ilanit Stein <istein>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.1CC: mxie, mzhan, nsoffer, ptoscano, tzheng, xiaodwan, zili
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.40.2-21.module+el8.2.0+5851+8d6a931b Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-05 09:51:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch for RHEL 7.7 tested by QE
none
virt-v2v-1.40.2-20.module+el8.2.0+5433-reproduce.log
none
virt-v2v-1.40.2-21.module+el8.2.0+5851-verify.log none

Description Ilanit Stein 2019-12-01 14:07:38 UTC 
Description of problem:
Copy from patch https://github.com/libguestfs/virt-v2v/commit/2aa78ade2d48e926b7b04050338ebd8a0c5e3f05: 

"
qemu-img convert" detects zeroes in allocated areas and punch holes in
the destination image. This may save space on the destination image, but
slows down conversion when using outputs such as rhv-upload, which have
very large overhead per requests.

Using the -S flag, we can treat small areas filled with zeroes as data,
limiting the number of requests, and speeding the operation.
"

We tested in QE this patch, on RHV-4.3.7/
and these patched packages on RHEL-7.7 RHV host:
virt-v2v-1.40.2-6.el7_7.3.x86_64
libguestfs-1.40.2-6.el7_7.3.x86_64

For a VM with 100 GB/66 GB data, v2v disk conversion, using vddk method took 18 minutes (CFME UI time), and without the patch it took 20 minutes.

For 10 VMS, with 100GB/66 GB data, v2v disk conversion, using vddk method took 31 minutes (CFME UI time), and without the patch it took 33 minutes.

For 20 VMS, with 100GB/66 GB data, v2v disk conversion, using vddk method took 55 minutes (CFME UI time), and without the patch it took 55 minutes.

Based on the above improvement, and as we didn't experience any regression,
we recommend to add it this patch.

It can be back ported as well to RHEL-7.7. 


Version-Release number of selected component (if applicable):
RHEL-8.1, RHEL-7.7
Comment 1 Nir Soffer 2019-12-01 15:10:20 UTC 
Created attachment 1641059 [details]
Patch for RHEL 7.7 tested by QE
Comment 2 Pino Toscano 2019-12-02 09:09:03 UTC 
> It can be back ported as well to RHEL-7.7.

This is hardly happening.
Comment 3 Richard W.M. Jones 2019-12-02 10:33:16 UTC 
ACKed for RHEL AV 8.2.0.

As for RHEL 7, I also think that is tricky.  We are trying not to make any
feature changes in RHEL 7 (7.7 itself has long been released, so that is
definitely impossible, 7.8 is in testing so the development deadline for that
has passed too, 7.9 won't be released until late 2020).
Comment 7 mxie@redhat.com 2020-03-05 08:53:52 UTC 
Reproduce the bug with builds:
virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64
libguestfs-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64
libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
nbdkit-1.16.2-2.module+el8.2.0+5664+dd92f997.x86_64


Steps to reproduce:
Scenario1:
1.1 Convert one guest from VMware to rhv by virt-v2v
#  virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload  -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data esx6.7-rhel8.2-x86_64 -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -v -x |& ts > virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log

1.2 Check v2v debug log to calculate how long "qemu-img 'convert" command will be finished
#cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log |grep "(0.00/100%)" -B 1
Mar 05 13:03:15 qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovlf26ff0.qcow2' 'json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.03Idtl/nbdkit0.sock", "file.export": "/" }'
nbdkit: debug: accepted connection

#cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log |grep "(100.00/100%)" -A 2
--
nbdkit: python[1]: debug: python: flush
Mar 05 13:28:46 nbdkit: python[1]: debug: python: flush
Mar 05 13:28:46 nbdkit: python[1]: debug: client sent NBD_CMD_DISC, closing connection

Result1.2
  Can see there is no -S flag in "qemu-img 'convert'" command and this process takes 25m31s without -S flag to finish.

1.3 Calculate how many write requests happens during executing "qemu-img 'convert'" command
# sed -n '/Mar 05 13:03:15/,/Mar 05 13:28:46/p' virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log | wc -l
45176

1.4 Check v2v debug log to calculate how long v2v will take to convert a guest from VMware to rhv
#cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log
.....
Mar 05 13:01:56 virt-v2v: libguestfs 1.40.2rhel=8,release=20.module+el8.2.0+5433+9e1420c8,libvirt (x86_64)
....
Mar 05 13:28:59 nbdkit: debug: VixDiskLibVim: VixDiskLibVim_Exit: Clean up.

Result1.4
   v2v takes 27m03s to finish conversion


Scenario2:
2.1 Convert five different guests from VMware to rhv by virt-v2v at the same time, the steps are same with scenario1, details info please refer to virt-v2v-1.40.2-20.module+el8.2.0+5433-reproduce.log

Summary test result of scenario2 as below:
               Time of qemu-img 'convert        write requests of qemu-img 'convert     Total time of conversion

Guest1              57m19s                                45167                               58m44s 
Guest2              46m25s                                38163                               48m38s
Guest3              1h03m06s                              28245                               1h03m54s 
Guest4              1h05m34s                              41868                               1h07m57s
Guest5              59m39s                                46734                               1h02m29s 

Average value       58m25s                                40035                                1h02s

*****************************

Summary reproduced result as below:

                         Time of qemu-img 'convert        write requests of qemu-img 'convert    Total time of conversion

Convert 1 guest            25m31s                                45176                                27m03s 

Convert 5 guests           58m25s                                40035                                1h02s


-----------------------------------------------------------------------------------------------------------------

Verify the bug with builds:
virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64
libguestfs-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64
libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
nbdkit-1.16.2-2.module+el8.2.0+5664+dd92f997.x86_64

Steps:
Scenario1:
1.1 Convert one guest from VMware to rhv by virt-v2v
#  virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload  -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd  -b ovirtmgmt --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data esx6.7-rhel8.2-x86_64 -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -v -x |& ts > virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log

1.2 Check v2v debug log to calculate how long "qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k'....." will be finished
# cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log |grep "(0.00/100%)" -B 1
Mar 04 14:46:43 qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k' '/var/tmp/v2vovl5f98d3.qcow2' 'json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.dHCJk7/nbdkit0.sock", "file.export": "/" }'
nbdkit: debug: accepted connection

# cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log |grep "(100.00/100%)" -A 2
--
nbdkit: python[1]: debug: python: flush
Mar 04 15:00:40 nbdkit: python[1]: debug: python: flush
Mar 04 15:00:40 nbdkit: python[1]: debug: client sent NBD_CMD_DISC, closing connection

Result1.2
  "qemu-img 'convert'" command takes 13m57s with -S flag and value '64k' to finish.

1.3 Calculate how many write requests happens during executing "qemu-img 'convert'" command
# sed -n '/Mar 04 14:46:43/,/Mar 04 15:00:40/p' virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log | wc -l
35942

1.4 Check v2v debug log to calculate how long v2v will take to convert a guest from VMware to rhv
# cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log
.....
Mar 04 14:45:11 virt-v2v: libguestfs 1.40.2rhel=8,release=21.module+el8.2.0+5851+8d6a931b,libvirt (x86_64)
....
Mar 04 15:00:54 nbdkit: debug: VixDiskLibVim: VixDiskLibVim_Exit: Clean up.


Result1.4
   v2v takes 15m43s to finish conversion


Scenario2:
2.1 Convert five different guests from VMware to rhv by virt-v2v at the same time, the steps are same with scenario1, details info please refer to virt-v2v-1.40.2-21.module+el8.2.0+5851-verify.log

Summary test result scenario2 as below:

               Time of qemu-img 'convert        write requests of qemu-img 'convert     Total time of conversion

Guest1              41m39s                                35952                               43m27s 
Guest2              44m41s                                35365                               49m15s
Guest3              37m57s                                28245                               38m50s 
Guest4              49m16s                                41260                               51m43s
Guest5              48m35s                                41074                               52m43s

Average value       43m46s                                36379                               46m42s

***************************************

Summary verify result as below:

                         Time of qemu-img 'convert        write requests of qemu-img 'convert    Total time of conversion

Convert 1 guest              13m57s                                     35942                            15m43s 

Convert 5 guests             43m46s                                     36379                            46m42s

----------------------------------------------------------------------------------------------------------

Final Result:

   (1) virt-v2v-1.40.2-20.module+el8.2.0+5433 will not use -S flag in "qemu-img convert" command to convert the guest, but virt-v2v-1.40.2-21.module+el8.2.0+5851 will use 

   (2)Compare conversion time with different virt-v2v build as below, we can see the conversion time of virt-v2v-1.40.2-21.module+el8.2.0+5851 is apparently shorter than virt-v2v-1.40.2-20.module+el8.2.0+5433 and write requests number of qemu-img 'convert of virt-v2v-1.40.2-21.module+el8.2.0+5851 is apparently less than virt-v2v-1.40.2-20.module+el8.2.0+5433. so the bug has been fixed, move the bug from ON_QA to VERIFIED


* virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64

                         Time of qemu-img 'convert        write requests of qemu-img 'convert    Total time of conversion

Convert 1 guest            25m31s                                45176                                27m03s 

Convert 5 guests           58m25s                                40035                                1h02s


* virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64

Summary verify result as below:

                         Time of qemu-img 'convert        write requests of qemu-img 'convert    Total time of conversion

Convert 1 guest             13m57s                                35942                               15m43s 
 
Convert 5 guests            43m46s                                36379                               46m42s
Comment 8 mxie@redhat.com 2020-03-05 08:55:04 UTC 
Created attachment 1667683 [details]
virt-v2v-1.40.2-20.module+el8.2.0+5433-reproduce.log
Comment 9 mxie@redhat.com 2020-03-05 08:55:50 UTC 
Created attachment 1667684 [details]
virt-v2v-1.40.2-21.module+el8.2.0+5851-verify.log
Comment 11 errata-xmlrpc 2020-05-05 09:51:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017