Bug 1026136

Summary: Volume download speed is slow
Product: Red Hat Enterprise Linux 7 Reporter: Luwen Su <lsu>
Component: libvirtAssignee: Martin Kletzander <mkletzan>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, berrange, chhu, cross, cwei, dallan, david.pravec, dyuan, gsun, jyang, mjenner, mzhan, vbudikov, xuzhang, yanyang, ydu, yisun
Target Milestone: rcKeywords: Reopened, Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.3.1-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 950416
: 1026137 (view as bug list) Environment:
Last Closed: 2016-11-03 18:06:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1026137    

Description Luwen Su 2013-11-04 02:39:42 UTC 
+++ This bug was initially created as a clone of Bug #950416 +++

Remove comments that not related this bug , make up the needinfo

--- Additional comment from time.su on 2013-10-25 03:46:34 EDT ---

Test with:
libvirt-1.1.1-10.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64


The comments 10's steps are okay for me , also test the raw img.
But , the vol-download's speed will down to too slow when the image large enough (1G in my environment)

#time virsh vol-download --vol vol-test.img --pool default --file /home/test1/a 

real	1m56.311s
user	1m24.267s
sys	0m2.743s

And the 500M's volumes time:
real  0m19.033s

virsh # vol-dumpxml vol-test.img --pool default
<volume>
  <name>vol-test.img</name>
  <key>/var/lib/libvirt/images/vol-test.img</key>
  <source>
  </source>
  <capacity unit='bytes'>1048576000</capacity>
  <allocation unit='bytes'>1048580096</allocation>
  <target>
    <path>/var/lib/libvirt/images/vol-test.img</path>
    <format type='raw'/>
    <permissions>
      <mode>0600</mode>
      <owner>0</owner>
      <group>0</group>
      <label>system_u:object_r:virt_image_t:s0</label>
    </permissions>
    <timestamps>
      <atime>1382683904.221608140</atime>
      <mtime>1382683904.144609566</mtime>
      <ctime>1382683904.145609548</ctime>
    </timestamps>
  </target>
</volume>


ll /home/test1/
total 63484
-rw-r--r--. 1 root root 65005760 Oct 25 14:53 a
# ll /home/test1/a 
-rw-r--r--. 1 root root 133943320 Oct 25 14:53 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 152029600 Oct 25 14:53 /home/test1/a
.........
-rw-r--r--. 1 root root 853136008 Oct 25 14:54 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 853987976 Oct 25 14:54 /home/test1/a
# ll /home/test1/a 
-rw-r--r--. 1 root root 854643336 Oct 25 14:54 /home/test1/a



Is it related libvirt ?

--- Additional comment from time.su on 2013-10-30 04:25:22 EDT ---

BTW , also reproduced it on rhel6.
1.
And in the second time , the speed can be accepted if the target is same .

# time virsh vol-download --vol test.img --pool default --file /home/a 
real	3m38.059s
user	3m32.044s
sys	0m2.489s

# time virsh vol-download --vol test.img --pool default --file /home/a 
real	0m10.085s
user	0m1.719s
sys	0m2.323s

# time virsh vol-download --vol test.img --pool default --file /home/b  ;;cancel
^C
real	1m49.476s
user	1m45.474s
sys	0m1.921s



2.
During the process , virsh will use almost 100% CPU , watched via top

KiB Mem:   7364840 total,  4982532 used,  2382308 free,   126120 buffers

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND       28663 root      20   0  628020 312468   4784 R  98.3  4.2   0:48.99 virsh 
                                               /*HERE*/

--- Additional comment from Osier Yang on 2013-10-30 09:58:56 EDT ---

(In reply to time.su from comment #19)
> Test with:
> libvirt-1.1.1-10.el7.x86_64
> qemu-kvm-1.5.3-10.el7.x86_64
> 
> 
> The comments 10's steps are okay for me , also test the raw img.
> But , the vol-download's speed will down to too slow when the image large
> enough (1G in my environment)
> 
> #time virsh vol-download --vol vol-test.img --pool default --file
> /home/test1/a 
> 
> real	1m56.311s
> user	1m24.267s
> sys	0m2.743s
> 
> And the 500M's volumes time:
> real  0m19.033s
> 
> virsh # vol-dumpxml vol-test.img --pool default
> <volume>
>   <name>vol-test.img</name>
>   <key>/var/lib/libvirt/images/vol-test.img</key>
>   <source>
>   </source>
>   <capacity unit='bytes'>1048576000</capacity>
>   <allocation unit='bytes'>1048580096</allocation>
>   <target>
>     <path>/var/lib/libvirt/images/vol-test.img</path>
>     <format type='raw'/>
>     <permissions>
>       <mode>0600</mode>
>       <owner>0</owner>
>       <group>0</group>
>       <label>system_u:object_r:virt_image_t:s0</label>
>     </permissions>
>     <timestamps>
>       <atime>1382683904.221608140</atime>
>       <mtime>1382683904.144609566</mtime>
>       <ctime>1382683904.145609548</ctime>
>     </timestamps>
>   </target>
> </volume>
> 
> 
> ll /home/test1/
> total 63484
> -rw-r--r--. 1 root root 65005760 Oct 25 14:53 a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 133943320 Oct 25 14:53 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 152029600 Oct 25 14:53 /home/test1/a
> .........
> -rw-r--r--. 1 root root 853136008 Oct 25 14:54 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 853987976 Oct 25 14:54 /home/test1/a
> # ll /home/test1/a 
> -rw-r--r--. 1 root root 854643336 Oct 25 14:54 /home/test1/a
> 
> 
> 
> Is it related libvirt ?

I think it's not related with libvirt, but I also have no idea why it becomes slow in the end of the transfering (when we talked f2f, time said that for a 1G volume, it becomes slow about at the point of 800M, before that it just looks fine), Dan, I think you are the more appropriate person for this question?
Comment 1 Luwen Su 2013-11-04 02:42:10 UTC 
Needinfo per osier's comment 

"I think it's not related with libvirt, but I also have no idea why it becomes slow in the end of the transfering (when we talked f2f, time said that for a 1G volume, it becomes slow about at the point of 800M, before that it just looks fine), Dan, I think you are the more appropriate person for this question?"
Comment 3 Dave Allan 2013-11-04 18:44:46 UTC 
(In reply to time.su from comment #0)
> And in the second time , the speed can be accepted if the target is same .
> 
> # time virsh vol-download --vol test.img --pool default --file /home/a 
> real	3m38.059s
> user	3m32.044s
> sys	0m2.489s
> 
> # time virsh vol-download --vol test.img --pool default --file /home/a 
> real	0m10.085s
> user	0m1.719s
> sys	0m2.323s
> 
> # time virsh vol-download --vol test.img --pool default --file /home/b 
> ;;cancel
> ^C
> real	1m49.476s
> user	1m45.474s
> sys	0m1.921s

If the cache effects make the problem go away, this is not a libvirt bug.
Comment 4 Daniel Berrangé 2015-06-03 17:05:24 UTC 
Re-opening based on info added to https://bugzilla.redhat.com/show_bug.cgi?id=1026137#c5
Comment 6 Martin Kletzander 2015-08-03 11:21:49 UTC 
Fixed upstream with v1.2.18-5-gd9c9e138f22c:
commit d9c9e138f22c48626f719f880920e04c639e0177
Author: Ossi Herrala <oherrala>
Date:   Mon Jul 20 12:44:32 2015 +0000

    rpc: Fix slow volume download (virsh vol-download)
Comment 9 yisun 2016-03-11 10:10:49 UTC 
Verified with libvirt libvirt-1.3.2-1.el7.x86_64
compared with libvirt libvirt-1.2.17-13.el7.x86_64

Result is obviously improved, so mark as PASSED.
 

=============== raw ==================

# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64


# virsh vol-list --details default
 Name                         Path                                                 Type      Capacity  Allocation
------------------------------------------------------------------------------------------------------------------
...
 vol-download-test.img        /var/lib/libvirt/images/vol-download-test.img        file      1.46 GiB    1.46 GiB
...

# time virsh vol-download --vol vol-download-test.img --pool default --file /tmp/img 
real	6m2.767s
user	5m57.464s
sys	0m3.326s


#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64


# time virsh vol-download --vol vol-download-test.img --pool default --file /tmp/img_fixed 
real	0m7.229s
user	0m0.777s
sys	0m3.172s

# diff /tmp/img /tmp/img_fixed 
# echo $?
0





=============== qcow2 =============== 
# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64

# virsh vol-list --details default
 Name                   Path                                           Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------
...
 rhel7.qcow2            /var/lib/libvirt/images/rhel7.qcow2            file  10.00 GiB   10.00 GiB
...

# time virsh vol-download --vol rhel7.qcow2 --pool default --file /tmp/qcow2 
^C   <==== too long, just terminate it...

real	10m29.729s
user	10m21.927s
sys	0m3.562s


#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64

# time virsh vol-download --vol rhel7.qcow2 --pool default --file /tmp/qcow2_fixed 
real	3m1.307s
user	0m6.381s
sys	0m21.614s


=============== iscsi =============== 
# rpm -qa | grep libvirt-1
libvirt-1.2.17-13.el7.x86_64

# virsh vol-list --details iscsi
 Name        Path                                                                                    Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------------------------------------
...
 unit:0:0:3  /dev/disk/by-path/ip-10.66.5.88:3260-iscsi-iqn.2014-12.com.redhat:libvirt-manual-lun-3  block  1.00 GiB    1.00 GiB

# time virsh vol-download --vol unit:0:0:3 --pool iscsi --file /tmp/iscsi 
real	2m11.461s
user	2m1.718s
sys	0m2.731s



#yum update libvirt -y 
# service libvirtd restart

# rpm -qa | grep libvirt-1
libvirt-1.3.2-1.el7.x86_64

# time virsh vol-download --vol unit:0:0:3 --pool iscsi --file /tmp/iscsi_fixed 
real	0m14.522s
user	0m0.653s
sys	0m2.350s

# diff /tmp/iscsi /tmp/iscsi_fixed 
# echo $?
0
Comment 11 errata-xmlrpc 2016-11-03 18:06:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html