Bug 635527
Summary: | KVM:qemu-img re-base poor performance(on local storage) when snapshot to a new disk | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | juzhang <juzhang> |
Component: | qemu-kvm | Assignee: | Kevin Wolf <kwolf> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 6.0 | CC: | ehabkost, lcapitulino, lihuang, mhusnain, michen, mkenneth, shuang, szhou, tburke, virt-maint, wquan |
Target Milestone: | beta | ||
Target Release: | 6.1 | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | qemu-kvm-0.12.1.2-2.136.el6 | Doc Type: | Bug Fix |
Doc Text: |
Previously, qemu-kvm did not include metadata caching, which resulted in poor performance for the the qemu-img rebase process. This is now fixed and the QCOW2 code now includes metadata caching, which results in improved performance for the qemu-img rebase process.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2011-05-19 11:27:05 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 580951 |
Description
juzhang
2010-09-20 05:00:09 UTC
(In reply to comment #0) > 2.Do rebase on qemu-kvm-0.12.1.2-2.113.el6_0.1.x86_64 compare to > qemu-kvm-0.12.1.2-2.53.el6.x86_64. there are a lot of fdatasync() calls.I am > not sure this is key reasons. Yes, it is the reason. The fdatasync is required to guarantee image integrity in case of a crash/power failure, so we can't just remove it again. There is a prototype for an optimization approach to get much of the performance back, but it's an intrusive change and not ready yet. Reproduce this bug with qemu-kvm-0.12.1.2-2.135.el6.x86_64 on local storage and iscsi storage as following: For local storage: 1. RHEL6.img is previous installed rhel6 guest image # qemu-img info RHEL6.img image: RHEL6.img file format: qcow2 virtual size: 20G (20971520000 bytes) disk size: 6.6G cluster_size: 65536 2. create snapshot chain RHEL6.img -> snA -> snB -> snC # qemu-img create -f qcow2 -F qcow2 -b RHEL6.img snA Formatting 'snA', fmt=qcow2 size=20971520000 backing_file='RHEL6.img' backing_fmt='qcow2' encryption=off cluster_size=0 # qemu-img create -f qcow2 -F qcow2 -b snA snB Formatting 'snB', fmt=qcow2 size=20971520000 backing_file='snA' backing_fmt='qcow2' encryption=off cluster_size=0 # qemu-img create -f qcow2 -F qcow2 -b snB snC Formatting 'snC', fmt=qcow2 size=20971520000 backing_file='snB' backing_fmt='qcow2' encryption=off cluster_size=0 3. create new base image with same size as base # qemu-img create -f qcow2 newbase.img 20G Formatting 'newbase.img', fmt=qcow2 size=21474836480 encryption=off cluster_size=0 4. record time of rebase snapshot c to new base. # time strace -c qemu-img rebase -b newbase.img -f qcow2 -F qcow2 snC % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 61.18 8.712965 104 83632 select 35.52 5.058238 149 33997 fdatasync 2.36 0.335972 1 227702 1639 futex 0.91 0.130272 1 100271 pwrite 0.01 0.001760 0 83637 read 0.01 0.001385 12 117 pread 0.01 0.000869 46 19 munmap 0.00 0.000011 1 22 close 0.00 0.000000 0 22 open 0.00 0.000000 0 8 stat 0.00 0.000000 0 6 fstat 0.00 0.000000 0 8 lseek 0.00 0.000000 0 37 mmap 0.00 0.000000 0 10 mprotect 0.00 0.000000 0 14 brk 0.00 0.000000 0 2 rt_sigaction 0.00 0.000000 0 4 rt_sigprocmask 0.00 0.000000 0 8 8 ioctl 0.00 0.000000 0 1 1 access 0.00 0.000000 0 1 clone 0.00 0.000000 0 1 execve 0.00 0.000000 0 3 fcntl 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 1 signalfd ------ ----------- ----------- --------- --------- ---------------- 100.00 14.241472 529527 1648 total real 36m21.471s user 0m30.643s sys 0m35.155s Last about 36 mins. For iscsi storage: 1.previous installed rhel6 guest # qemu-img info /dev/vgtest/lvtest image: /dev/vgtest/lvtest file format: qcow2 virtual size: 50G (53687091200 bytes) disk size: 0 cluster_size: 65536 2. create snapshot chain. RHEL6.img -> snA -> snB -> snC # qemu-img create -f qcow2 -F qcow2 -b /dev/vgtest/lvtest /dev/vgtest/snA Formatting '/dev/vgtest/snA', fmt=qcow2 size=53687091200 backing_file='/dev/vgtest/lvtest' backing_fmt='qcow2' encryption=off cluster_size=0 # qemu-img create -f qcow2 -F qcow2 -b /dev/vgtest/snA /dev/vgtest/snB Formatting '/dev/vgtest/snB', fmt=qcow2 size=53687091200 backing_file='/dev/vgtest/snA' backing_fmt='qcow2' encryption=off cluster_size=0 # qemu-img create -f qcow2 -F qcow2 -b /dev/vgtest/snB /dev/vgtest/snC Formatting '/dev/vgtest/snC', fmt=qcow2 size=53687091200 backing_file='/dev/vgtest/snB' backing_fmt='qcow2' encryption=off cluster_size=0 3. create new base image with same size as base image # qemu-img create -f qcow2 /dev/vgtest/newbase 50G Formatting '/dev/vgtest/newbase', fmt=qcow2 size=53687091200 encryption=off cluster_size=0 4. record time to rebase snapshot to base new. # time strace -c qemu-img rebase -b /dev/vgtest/newbase -f qcow2 -F qcow2 /dev/vgtest/snC % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 60.89 5.644797 160 35378 select 24.05 2.229636 162 13728 fdatasync 12.65 1.172540 53297 22 close 2.10 0.194862 2 91233 178 futex 0.24 0.022066 1 39913 pwrite 0.03 0.003153 27 116 pread 0.02 0.001873 0 35383 read 0.01 0.000969 51 19 munmap 0.00 0.000032 1 37 mmap 0.00 0.000000 0 22 open 0.00 0.000000 0 8 stat 0.00 0.000000 0 6 fstat 0.00 0.000000 0 8 lseek 0.00 0.000000 0 10 mprotect 0.00 0.000000 0 14 brk 0.00 0.000000 0 2 rt_sigaction 0.00 0.000000 0 4 rt_sigprocmask 0.00 0.000000 0 8 8 ioctl 0.00 0.000000 0 1 1 access 0.00 0.000000 0 1 clone 0.00 0.000000 0 1 execve 0.00 0.000000 0 3 fcntl 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 1 signalfd ------ ----------- ----------- --------- --------- ---------------- 100.00 9.269928 215922 187 total real 11m51.648s user 1m33.070s sys 0m18.393s Last about 11 mins. Verify this bug with qemu-kvm-0.12.1.2-2.145.el6.x86_64 as reproduce steps, time spend as following: Local storage: # time strace -c qemu-img rebase -b newbase.img -f qcow2 -F qcow2 snC % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 65.49 17.133394 237964 72 fdatasync 32.66 8.545440 111 76793 select 1.41 0.367682 2 225227 516 futex 0.42 0.109451 3 32383 pwrite 0.02 0.004100 35 117 pread 0.01 0.001906 0 76798 read 0.00 0.000342 31 11 munmap 0.00 0.000017 1 22 open 0.00 0.000009 0 73 brk 0.00 0.000000 0 22 close 0.00 0.000000 0 8 stat 0.00 0.000000 0 6 fstat 0.00 0.000000 0 8 lseek 0.00 0.000000 0 29 mmap 0.00 0.000000 0 10 mprotect 0.00 0.000000 0 2 rt_sigaction 0.00 0.000000 0 4 rt_sigprocmask 0.00 0.000000 0 8 8 ioctl 0.00 0.000000 0 1 1 access 0.00 0.000000 0 1 clone 0.00 0.000000 0 1 execve 0.00 0.000000 0 3 fcntl 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 1 signalfd ------ ----------- ----------- --------- --------- ---------------- 100.00 26.162341 411604 525 total real 2m45.269s user 0m29.700s sys 0m43.185s Last about 3 mins. For iscsi storage: # time strace -c qemu-img rebase -b /dev/vgtest/newbase -f qcow2 -F qcow2 /dev/vgtest/snC % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 76.30 6.349827 160 39803 select 11.34 0.943855 18151 52 fdatasync 10.39 0.864869 39312 22 close 1.58 0.131680 1 91823 279 futex 0.33 0.027392 2 12534 pwrite 0.03 0.002116 18 116 pread 0.02 0.001531 0 39808 read 0.01 0.001058 96 11 munmap 0.00 0.000325 5 72 brk 0.00 0.000000 0 22 open 0.00 0.000000 0 8 stat 0.00 0.000000 0 6 fstat 0.00 0.000000 0 8 lseek 0.00 0.000000 0 29 mmap 0.00 0.000000 0 10 mprotect 0.00 0.000000 0 2 rt_sigaction 0.00 0.000000 0 4 rt_sigprocmask 0.00 0.000000 0 8 8 ioctl 0.00 0.000000 0 1 1 access 0.00 0.000000 0 1 clone 0.00 0.000000 0 1 execve 0.00 0.000000 0 3 fcntl 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 1 signalfd ------ ----------- ----------- --------- --------- ---------------- 100.00 8.322653 184349 288 total real 3m19.102s user 1m32.898s sys 0m16.125s Last about 3 mins. So compare above test result, we have performance improvement as 36 mins --> 3 mins (local storage) 11 mins --> 3 mins ( iscsi storage) So this bug has been resolved. *** Bug 676565 has been marked as a duplicate of this bug. *** move to verified based on comment#14 Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: no metadata caching in the qemu-kvm qcow2 code. Consequence: poor performance of 'qemu-img rebase' Fix: a metadata cache was introduced into the qcow2 code. Result: improved performance of 'qemu-img rebase'. Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,4 +1 @@ -Cause: no metadata caching in the qemu-kvm qcow2 code. +Previously, qemu-kvm did not include metadata caching, which resulted in poor performance for the the qemu-img rebase process. This is now fixed and the QCOW2 code now includes metadata caching, which results in improved performance for the qemu-img rebase process.-Consequence: poor performance of 'qemu-img rebase' -Fix: a metadata cache was introduced into the qcow2 code. -Result: improved performance of 'qemu-img rebase'. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |