Description of problem: Copy from patch https://github.com/libguestfs/virt-v2v/commit/2aa78ade2d48e926b7b04050338ebd8a0c5e3f05: " qemu-img convert" detects zeroes in allocated areas and punch holes in the destination image. This may save space on the destination image, but slows down conversion when using outputs such as rhv-upload, which have very large overhead per requests. Using the -S flag, we can treat small areas filled with zeroes as data, limiting the number of requests, and speeding the operation. " We tested in QE this patch, on RHV-4.3.7/ and these patched packages on RHEL-7.7 RHV host: virt-v2v-1.40.2-6.el7_7.3.x86_64 libguestfs-1.40.2-6.el7_7.3.x86_64 For a VM with 100 GB/66 GB data, v2v disk conversion, using vddk method took 18 minutes (CFME UI time), and without the patch it took 20 minutes. For 10 VMS, with 100GB/66 GB data, v2v disk conversion, using vddk method took 31 minutes (CFME UI time), and without the patch it took 33 minutes. For 20 VMS, with 100GB/66 GB data, v2v disk conversion, using vddk method took 55 minutes (CFME UI time), and without the patch it took 55 minutes. Based on the above improvement, and as we didn't experience any regression, we recommend to add it this patch. It can be back ported as well to RHEL-7.7. Version-Release number of selected component (if applicable): RHEL-8.1, RHEL-7.7
Created attachment 1641059 [details] Patch for RHEL 7.7 tested by QE
> It can be back ported as well to RHEL-7.7. This is hardly happening.
ACKed for RHEL AV 8.2.0. As for RHEL 7, I also think that is tricky. We are trying not to make any feature changes in RHEL 7 (7.7 itself has long been released, so that is definitely impossible, 7.8 is in testing so the development deadline for that has passed too, 7.9 won't be released until late 2020).
Reproduce the bug with builds: virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libguestfs-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 nbdkit-1.16.2-2.module+el8.2.0+5664+dd92f997.x86_64 Steps to reproduce: Scenario1: 1.1 Convert one guest from VMware to rhv by virt-v2v # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -b ovirtmgmt --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data esx6.7-rhel8.2-x86_64 -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -v -x |& ts > virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log 1.2 Check v2v debug log to calculate how long "qemu-img 'convert" command will be finished #cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log |grep "(0.00/100%)" -B 1 Mar 05 13:03:15 qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovlf26ff0.qcow2' 'json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.03Idtl/nbdkit0.sock", "file.export": "/" }' nbdkit: debug: accepted connection #cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log |grep "(100.00/100%)" -A 2 -- nbdkit: python[1]: debug: python: flush Mar 05 13:28:46 nbdkit: python[1]: debug: python: flush Mar 05 13:28:46 nbdkit: python[1]: debug: client sent NBD_CMD_DISC, closing connection Result1.2 Can see there is no -S flag in "qemu-img 'convert'" command and this process takes 25m31s without -S flag to finish. 1.3 Calculate how many write requests happens during executing "qemu-img 'convert'" command # sed -n '/Mar 05 13:03:15/,/Mar 05 13:28:46/p' virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log | wc -l 45176 1.4 Check v2v debug log to calculate how long v2v will take to convert a guest from VMware to rhv #cat virt-v2v-1.40.2-20.module+el8.2.0+5433-1guest-reproduce-time.log ..... Mar 05 13:01:56 virt-v2v: libguestfs 1.40.2rhel=8,release=20.module+el8.2.0+5433+9e1420c8,libvirt (x86_64) .... Mar 05 13:28:59 nbdkit: debug: VixDiskLibVim: VixDiskLibVim_Exit: Clean up. Result1.4 v2v takes 27m03s to finish conversion Scenario2: 2.1 Convert five different guests from VMware to rhv by virt-v2v at the same time, the steps are same with scenario1, details info please refer to virt-v2v-1.40.2-20.module+el8.2.0+5433-reproduce.log Summary test result of scenario2 as below: Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Guest1 57m19s 45167 58m44s Guest2 46m25s 38163 48m38s Guest3 1h03m06s 28245 1h03m54s Guest4 1h05m34s 41868 1h07m57s Guest5 59m39s 46734 1h02m29s Average value 58m25s 40035 1h02s ***************************** Summary reproduced result as below: Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Convert 1 guest 25m31s 45176 27m03s Convert 5 guests 58m25s 40035 1h02s ----------------------------------------------------------------------------------------------------------------- Verify the bug with builds: virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64 libguestfs-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64 libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 nbdkit-1.16.2-2.module+el8.2.0+5664+dd92f997.x86_64 Steps: Scenario1: 1.1 Convert one guest from VMware to rhv by virt-v2v # virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -b ovirtmgmt --password-file /home/passwd -of raw -oo rhv-cluster=Default -os nfs_data esx6.7-rhel8.2-x86_64 -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -v -x |& ts > virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log 1.2 Check v2v debug log to calculate how long "qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k'....." will be finished # cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log |grep "(0.00/100%)" -B 1 Mar 04 14:46:43 qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '-S' '64k' '/var/tmp/v2vovl5f98d3.qcow2' 'json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.dHCJk7/nbdkit0.sock", "file.export": "/" }' nbdkit: debug: accepted connection # cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log |grep "(100.00/100%)" -A 2 -- nbdkit: python[1]: debug: python: flush Mar 04 15:00:40 nbdkit: python[1]: debug: python: flush Mar 04 15:00:40 nbdkit: python[1]: debug: client sent NBD_CMD_DISC, closing connection Result1.2 "qemu-img 'convert'" command takes 13m57s with -S flag and value '64k' to finish. 1.3 Calculate how many write requests happens during executing "qemu-img 'convert'" command # sed -n '/Mar 04 14:46:43/,/Mar 04 15:00:40/p' virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log | wc -l 35942 1.4 Check v2v debug log to calculate how long v2v will take to convert a guest from VMware to rhv # cat virt-v2v-1.40.2-21.module+el8.2.0+5851-1guest-time.log ..... Mar 04 14:45:11 virt-v2v: libguestfs 1.40.2rhel=8,release=21.module+el8.2.0+5851+8d6a931b,libvirt (x86_64) .... Mar 04 15:00:54 nbdkit: debug: VixDiskLibVim: VixDiskLibVim_Exit: Clean up. Result1.4 v2v takes 15m43s to finish conversion Scenario2: 2.1 Convert five different guests from VMware to rhv by virt-v2v at the same time, the steps are same with scenario1, details info please refer to virt-v2v-1.40.2-21.module+el8.2.0+5851-verify.log Summary test result scenario2 as below: Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Guest1 41m39s 35952 43m27s Guest2 44m41s 35365 49m15s Guest3 37m57s 28245 38m50s Guest4 49m16s 41260 51m43s Guest5 48m35s 41074 52m43s Average value 43m46s 36379 46m42s *************************************** Summary verify result as below: Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Convert 1 guest 13m57s 35942 15m43s Convert 5 guests 43m46s 36379 46m42s ---------------------------------------------------------------------------------------------------------- Final Result: (1) virt-v2v-1.40.2-20.module+el8.2.0+5433 will not use -S flag in "qemu-img convert" command to convert the guest, but virt-v2v-1.40.2-21.module+el8.2.0+5851 will use (2)Compare conversion time with different virt-v2v build as below, we can see the conversion time of virt-v2v-1.40.2-21.module+el8.2.0+5851 is apparently shorter than virt-v2v-1.40.2-20.module+el8.2.0+5433 and write requests number of qemu-img 'convert of virt-v2v-1.40.2-21.module+el8.2.0+5851 is apparently less than virt-v2v-1.40.2-20.module+el8.2.0+5433. so the bug has been fixed, move the bug from ON_QA to VERIFIED * virt-v2v-1.40.2-20.module+el8.2.0+5433+9e1420c8.x86_64 Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Convert 1 guest 25m31s 45176 27m03s Convert 5 guests 58m25s 40035 1h02s * virt-v2v-1.40.2-21.module+el8.2.0+5851+8d6a931b.x86_64 Summary verify result as below: Time of qemu-img 'convert write requests of qemu-img 'convert Total time of conversion Convert 1 guest 13m57s 35942 15m43s Convert 5 guests 43m46s 36379 46m42s
Created attachment 1667683 [details] virt-v2v-1.40.2-20.module+el8.2.0+5433-reproduce.log
Created attachment 1667684 [details] virt-v2v-1.40.2-21.module+el8.2.0+5851-verify.log
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017