RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1466563 - Libguestfs should pass copyonread flag through to the libvirt XML
Summary: Libguestfs should pass copyonread flag through to the libvirt XML
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.4
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1472272
Blocks: 1469902 1473046
TreeView+ depends on / blocked
 
Reported: 2017-06-30 02:33 UTC by kuwei@redhat.com
Modified: 2020-12-14 08:59 UTC (History)
13 users (show)

Fixed In Version: libguestfs-1.36.5-1.el7
Doc Type: Bug Fix
Doc Text:
Previously, when the libvirt back end to libguestfs was used, the "copyonread" flag was not passed through, which caused performance problems for libguestfs utilities. Notably, the virt-v2v utility was slowed significantly when converting guests from the VMware hypervisor. With this update, "copyonread" is correctly respected when running libguestfs utilities using the libvirt back end. As a result, the described problem no longer occurs.
Clone Of:
: 1469902 (view as bug list)
Environment:
Last Closed: 2018-04-10 09:15:08 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0677 0 normal SHIPPED_LIVE libguestfs new packages, bug fix and enhancement update 2018-04-10 13:13:49 UTC

Description kuwei@redhat.com 2017-06-30 02:33:22 UTC 
Description of problem:
This resulted in significant performance degradation (2x-3x
slower) when running virt-v2v convert guest from  VMware servers to local kvm.We should dropping the add_drive copyonread flag when using the libvirt
backend.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-14.el7.x86_64
kernel-3.10.0-681.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
virt-v2v-1.36.3-6.el7.x86_64
libvirt-2.0.0-10.el7.x86_64

How reproducible:


Steps to Reproduce:

Scenario 1:
Packages with below:
qemu-kvm-rhev-2.9.0-14.el7.x86_64
kernel-3.10.0-681.el7.x86_64
virt-v2v-1.36.3-6.el7.x86_64
libvirt-2.0.0-10.el7.x86_64

1.Convert a rhel7.3 guestfs from vmware to local kvm 
# virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64 --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64
[   1.3] Creating an overlay to protect the source from being modified
[   2.7] Initializing the target -o libvirt -os default
[   2.7] Opening the overlay
[  41.7] Inspecting the overlay
[ 128.1] Checking for sufficient free disk space in the guest
[ 128.1] Estimating space required on target for each disk
[ 128.1] Converting Red Hat Enterprise Linux Server 7.3 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 992.7] Mapping filesystem data to avoid copying unused and blank areas
[ 995.2] Closing the overlay
[ 995.5] Checking if the guest needs BIOS or UEFI to boot
[ 995.5] Assigning disks to buses
[ 995.5] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel7.3-x86_64-sda (raw)
    (100.00/100%)
[1648.2] Creating output metadata
Pool default refreshed

Domain esx5.5-rhel7.3-x86_64 defined from /tmp/v2vlibvirt8d734f.xml

[1648.6] Finishing off

Scenario 2:
Packages with below:
qemu-kvm-rhev-2.9.0-14.el7.x86_64
kernel-3.10.0-681.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
virt-v2v-1.36.3-6.el7.x86_64

1.Convert a rhel7.3 guestfs from vmware to local kvm 
# virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64 --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64
[   1.2] Creating an overlay to protect the source from being modified
[   1.7] Initializing the target -o libvirt -os default
[   1.7] Opening the overlay
[  56.1] Inspecting the overlay
[ 211.6] Checking for sufficient free disk space in the guest
[ 211.6] Estimating space required on target for each disk
[ 211.6] Converting Red Hat Enterprise Linux Server 7.3 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1792.8] Mapping filesystem data to avoid copying unused and blank areas
[1820.8] Closing the overlay
[1821.3] Checking if the guest needs BIOS or UEFI to boot
[1821.3] Assigning disks to buses
[1821.3] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel7.3-x86_64-sda (raw)
    (100.00/100%)
[2461.3] Creating output metadata
Pool default refreshed

Domain esx5.5-rhel7.3-x86_64 defined from /tmp/v2vlibvirt16025c.xml

[2461.9] Finishing off.

2.Compared scenario 1 and scenario 2,we can find that significant performance degradation when running virt-v2v convert guest from  VMware servers to local kvm with package libvirt-3.2.0-14.el7.x86_64


Actual results:
As above

Expected results:
Virt-v2v convert guest from  VMware server should have a better performance.

Additional info:
Comment 2 Richard W.M. Jones 2017-06-30 08:12:29 UTC 
Upstream fix:
https://www.redhat.com/archives/libguestfs/2017-June/msg00308.html
Comment 4 Richard W.M. Jones 2017-07-01 10:41:38 UTC 
Upstream commit:
https://github.com/libguestfs/libguestfs/commit/d96209ef07fef3db0e411648348f1b1dcf384e70
Comment 10 mxie@redhat.com 2017-09-21 10:55:28 UTC 
Try to reproduce this bug with builds:
virt-v2v-1.36.3-6.el7.x86_64
libguestfs-1.36.3-6.el7.x86_64
libvirt-client-3.7.0-2.el7.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64
virtio-win-1.9.3-1.el7.noarch

Steps:
1.virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-i386 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
> Use time [1425.7]
2.# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
> Use time [3871.0]
3.# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-win2016-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time  [1984.6] 
4.#virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-rhel7.4-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time  [3120.2]
5.#  virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008r2-x86_64 --password-file /tmp/passwd -o local -os /var/tmp
>Use time  [ 939.6]
6.# virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time [2641.2]
7.#virt-v2v -ic vpx://root.75.182/data/10.73.72.75/?no_verify=1 esx5.1-win7-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time [1196.5]



Verify the bug with below builds on same v2v conversion server with reproduced builds:
virt-v2v-1.36.6-1.el7.x86_64
libguestfs-1.36.6-1.el7.x86_64
libvirt-client-3.7.0-2.el7.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64
virtio-win-1.9.3-1.el7.noarch

Steps:
Convert the same guests with the ones which were converted by reproduced steps:
1.#virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-i386 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
> Use time [1616.5] 
2.# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
> Use time [1936.6]
3.# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-win2016-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time  [1740.9]
4.#virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-rhel7.4-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time [2364.1]
5.#  virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008r2-x86_64 --password-file /tmp/passwd -o local -os /var/tmp
>Use time  [ 656.4]
6.# virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.3-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time [1843.0]
7.#virt-v2v -ic vpx://root.75.182/data/10.73.72.75/?no_verify=1 esx5.1-win7-x86_64 --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
>Use time [1837.3]


Result:
   Using virt-v2v-1.36.6-1.el7.x86_64 to convert the guest could save more time than virt-v2v-1.36.3-6.el7.x86_64 sometimes, besides,could find  <driver name="qemu".... copy_on_read="on"/> in details v2v log with virt-v2v-1.36.6-1.el7.x86_64 build


Hi rjones,

  According above test result, Using virt-v2v-1.36.6-1.el7.x86_64 to convert the guest doesn't save more time than virt-v2v-1.36.3-6.el7.x86_64 at 100%, do you think the result could be accepted?
Comment 11 Pino Toscano 2017-09-22 08:22:54 UTC 
(In reply to mxie from comment #10)
>    Using virt-v2v-1.36.6-1.el7.x86_64 to convert the guest could save more
> time than virt-v2v-1.36.3-6.el7.x86_64 sometimes, besides,could find 
> <driver name="qemu".... copy_on_read="on"/> in details v2v log with
> virt-v2v-1.36.6-1.el7.x86_64 build

The right attribute in the libvirt XML is the important detail here, so it is good it is found as expected.

Regarding the conversion times: there is a general reduction of the times, which is expected; note also the actual times depends on various factors, such as:
- I/O on the source ESX, on vCenter, and on the destination
- network latency ESX<->vCenter, vCenter<->v2v, and v2v<->NFS
So if the times increasings only happen few times, I would say it is a local glitch.
Comment 12 mxie@redhat.com 2017-09-22 10:19:29 UTC 
Thanks Pino's reply, so according to comment10 and comment11, move this bug from ON_QA to VERIFIED
Comment 15 errata-xmlrpc 2018-04-10 09:15:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0677
Note You need to log in before you can comment on or make changes to this bug.