RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1026136 - Volume download speed is slow
Summary: Volume download speed is slow
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Martin Kletzander
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1026137
TreeView+ depends on / blocked
 
Reported: 2013-11-04 02:39 UTC by Luwen Su
Modified: 2016-11-03 18:06 UTC (History)
17 users (show)

Fixed In Version: libvirt-1.3.1-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of: 950416
: 1026137 (view as bug list)
Environment:
Last Closed: 2016-11-03 18:06:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 0 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Description Luwen Su 2013-11-04 02:39:42 UTC 
+++ This bug was initially created as a clone of Bug #950416 +++

Remove comments that not related this bug , make up the needinfo

--- Additional comment from time.su on 2013-10-25 03:46:34 EDT ---

Test with:
libvirt-1.1.1-10.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64


The comments 10's steps are okay for me , also test the raw img.
But , the vol-download's speed will down to too slow when the image large enough (1G in my environment)

#time virsh vol-download --vol vol-test.img --pool default --file /home/test1/a 

real	1m56.311s
user	1m24.267s
sys	0m2.743s

And the 500M's volumes time:
real  0m19.033s

virsh # vol-dumpxml vol-test.img --pool default
<volume>
  <name>vol-test.img</name>
  <key>/var/lib/libvirt/images/vol-test.img</key>
  <source>
  </source>
  <capacity unit='bytes'>1048576000</capacity>
  <allocation unit='bytes'>1048580096</allocation>
  <target>
    <path>/var/lib/libvirt/images/vol-test.img</path>
    <format type='raw'/>
    <permissions>
      <mode>0600</mode>
      <owner>0</owner>
      <group>0</group>
      <label>system_u:object_r:virt_image_t:s0</label>
    </permissions>
    <timestamps>
      <atime>1382683904.221608140</atime>
      <mtime>1382683904.144609566</mtime>
      <ctime>1382683904.145609548</ctime>
    </timestamps>
  </target>
</volume>


ll /home/test1/
total 63484
-rw-r--r--. 1 root root 65005760 Oct 25 14:53 a
# ll /home/test1/a 
-rw-r--r--. 1 root root 133943320 Oct 25 14:53 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 152029600 Oct 25 14:53 /home/test1/a
.........
-rw-r--r--. 1 root root 853136008 Oct 25 14:54 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 853987976 Oct 25 14:54 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 854643336 Oct 25 14:54 /home/test1/a



Is it related libvirt ?

--- Additional comment from time.su on 2013-10-30 04:25:22 EDT ---

BTW , also reproduced it on rhel6.
1.
And in the second time , the speed can be accepted if the target is same .

# time virsh vol-download --vol test.img --pool default --file /home/a 
real	3m38.059s
user	3m32.044s
sys	0m2.489s

# time virsh vol-download --vol test.img --pool default --file /home/a 
real	0m10.085s
user	0m1.719s
sys	0m2.323s

# time virsh vol-download --vol test.img --pool default --file /home/b  ;;cancel
^C
real	1m49.476s
user	1m45.474s
sys	0m1.921s



2.
During the process , virsh will use almost 100% CPU , watched via top

KiB Mem:   7364840 total,  4982532 used,  2382308 free,   126120 buffers

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND       28663 root      20   0  628020 312468   4784 R  98.3  4.2   0:48.99 virsh 
                                               /*HERE*/

--- Additional comment from Osier Yang on 2013-10-30 09:58:56 EDT ---

(In reply to time.su from comment #19)
> Test with:
> libvirt-1.1.1-10.el7.x86_64
> qemu-kvm-1.5.3-10.el7.x86_64
> 
> 
> The comments 10's steps are okay for me , also test the raw img.
> But , the vol-download's speed will down to too slow when the image large
> enough (1G in my environment)
> 
> #time virsh vol-download --vol vol-test.img --pool default --file
> /home/test1/a 
> 
> real	1m56.311s
> user	1m24.267s
> sys	0m2.743s
> 
> And the 500M's volumes time:
> real  0m19.033s
> 
> virsh # vol-dumpxml vol-test.img --pool default
> <volume>
>   <name>vol-test.img</name>
>   <key>/var/lib/libvirt/images/vol-test.img</key>
>   <source>
>   </source>
>   <capacity unit='bytes'>1048576000</capacity>
>   <allocation unit='bytes'>1048580096</allocation>
>   <target>
>     <path>/var/lib/libvirt/images/vol-test.img</path>
>     <format type='raw'/>
>     <permissions>
>       <mode>0600</mode>
>       <owner>0</owner>
>       <group>0</group>
>       <label>system_u:object_r:virt_image_t:s0</label>
>     </permissions>
>     <timestamps>
>       <atime>1382683904.221608140</atime>
>       <mtime>1382683904.144609566</mtime>
>       <ctime>1382683904.145609548</ctime>
>     </timestamps>
>   </target>
> </volume>
> 
> 
> ll /home/test1/
> total 63484
> -rw-r--r--. 1 root root 65005760 Oct 25 14:53 a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 133943320 Oct 25 14:53 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 152029600 Oct 25 14:53 /home/test1/a
> .........
> -rw-r--r--. 1 root root 853136008 Oct 25 14:54 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 853987976 Oct 25 14:54 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 854643336 Oct 25 14:54 /home/test1/a
> 
> 
> 
> Is it related libvirt ?

I think it's not related with libvirt, but I also have no idea why it becomes slow in the end of the transfering (when we talked f2f, time said that for a 1G volume, it becomes slow about at the point of 800M, before that it just looks fine), Dan, I think you are the more appropriate person for this question?
Comment 1 Luwen Su 2013-11-04 02:42:10 UTC 
Needinfo per osier's comment 

"I think it's not related with libvirt, but I also have no idea why it becomes slow in the end of the transfering (when we talked f2f, time said that for a 1G volume, it becomes slow about at the point of 800M, before that it just looks fine), Dan, I think you are the more appropriate person for this question?"
Comment 3 Dave Allan 2013-11-04 18:44:46 UTC 
(In reply to time.su from comment #0)
> And in the second time , the speed can be accepted if the target is same .
> 
> # time virsh vol-download --vol test.img --pool default --file /home/a 
> real	3m38.059s
> user	3m32.044s
> sys	0m2.489s
> 
> # time virsh vol-download --vol test.img --pool default --file /home/a 
> real	0m10.085s
> user	0m1.719s
> sys	0m2.323s
> 
> # time virsh vol-download --vol test.img --pool default --file /home/b 
> ;;cancel
> ^C
> real	1m49.476s
> user	1m45.474s
> sys	0m1.921s

If the cache effects make the problem go away, this is not a libvirt bug.
Comment 4 Daniel Berrangé 2015-06-03 17:05:24 UTC 
Re-opening based on info added to https://bugzilla.redhat.com/show_bug.cgi?id=1026137#c5
Comment 6 Martin Kletzander 2015-08-03 11:21:49 UTC 
Fixed upstream with v1.2.18-5-gd9c9e138f22c:
commit d9c9e138f22c48626f719f880920e04c639e0177
Author: Ossi Herrala <oherrala>
Date:   Mon Jul 20 12:44:32 2015 +0000

    rpc: Fix slow volume download (virsh vol-download)
Comment 9 yisun 2016-03-11 10:10:49 UTC 
Verified with libvirt libvirt-1.3.2-1.el7.x86_64
compared with libvirt libvirt-1.2.17-13.el7.x86_64

Result is obviously improved, so mark as PASSED.
 

=============== raw ==================

# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64


# virsh vol-list --details default
 Name                         Path                                                 Type      Capacity  Allocation
------------------------------------------------------------------------------------------------------------------
...
 vol-download-test.img        /var/lib/libvirt/images/vol-download-test.img        file      1.46 GiB    1.46 GiB
...

# time virsh vol-download --vol vol-download-test.img --pool default --file /tmp/img 
real	6m2.767s
user	5m57.464s
sys	0m3.326s


#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64


# time virsh vol-download --vol vol-download-test.img --pool default --file /tmp/img_fixed 
real	0m7.229s
user	0m0.777s
sys	0m3.172s

# diff /tmp/img /tmp/img_fixed 
# echo $?
0





=============== qcow2 =============== 
# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64

# virsh vol-list --details default
 Name                   Path                                           Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------
...
 rhel7.qcow2            /var/lib/libvirt/images/rhel7.qcow2            file  10.00 GiB   10.00 GiB
...

# time virsh vol-download --vol rhel7.qcow2 --pool default --file /tmp/qcow2 
^C   <==== too long, just terminate it...

real	10m29.729s
user	10m21.927s
sys	0m3.562s


#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64

# time virsh vol-download --vol rhel7.qcow2 --pool default --file /tmp/qcow2_fixed 
real	3m1.307s
user	0m6.381s
sys	0m21.614s


=============== iscsi =============== 
# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64

# virsh vol-list --details iscsi
 Name        Path                                                                                    Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------------------------------------
...
 unit:0:0:3  /dev/disk/by-path/ip-10.66.5.88:3260-iscsi-iqn.2014-12.com.redhat:libvirt-manual-lun-3  block  1.00 GiB    1.00 GiB

# time virsh vol-download --vol unit:0:0:3 --pool iscsi --file /tmp/iscsi 
real	2m11.461s
user	2m1.718s
sys	0m2.731s



#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64

# time virsh vol-download --vol unit:0:0:3 --pool iscsi --file /tmp/iscsi_fixed 
real	0m14.522s
user	0m0.653s
sys	0m2.350s

# diff /tmp/iscsi /tmp/iscsi_fixed 
# echo $?
0
Comment 11 errata-xmlrpc 2016-11-03 18:06:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html
Note You need to log in before you can comment on or make changes to this bug.