Bug 656851 - KVM: KVM IO Poor performance ( COW Sparse disk on NFS )
KVM: KVM IO Poor performance ( COW Sparse disk on NFS )
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm (Show other bugs)
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Kevin Wolf
Virtualization Bugs
Depends On:
Blocks: Rhel5KvmTier1
  Show dependency treegraph
Reported: 2010-11-24 05:59 EST by Oded Ramraz
Modified: 2013-01-09 18:22 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2011-01-16 17:40:46 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
IOMeter benchmark results (2.92 KB, application/zip)
2010-11-28 17:05 EST, Oded Ramraz
no flags Details

  None (edit)
Description Oded Ramraz 2010-11-24 05:59:13 EST
Description of problem:

I'm running IOMeter Benchmark on XP guest. Benchmarks results are very bad ( about 1 MB sec for both read/write )
I'm using NFS storage with COW sparse disk . 
When i perform IO operation manually on my host using dd / cp I'm reaching about 170-200 MB sec write rate ( No storage issues ).

My hosts are set to work with deadline scheduler 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Copy big file to the guest and measure IO performance 
Actual results:

Expected results:

Additional info:

Kernel info 

Linux blue-vdsc.qa.lab.tlv.redhat.com 2.6.18-194.26.1.el5 #1 SMP Fri Oct 29 14:21:16 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

KVM version:


kvm command line:

vdsm      4386  4377  4 12:17 ?        00:01:27 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet -rtc-td-hack -startdate 2010-11-24T10:17:44 -name kvmIOBenchmark1 -smp 1,cores=1 -k en-us -m 1024 -boot c -net nic,vlan=1,macaddr=00:1a:4a:16:95:7a,model=virtio -net tap,vlan=1,ifname=virtio_10_1,script=no -drive file=/rhev/data-center/00000000-0000-0000-0000-000000000002/00000000-0000-0000-0000-000000000011/images/ee2485d1-8132-4772-9700-b0ecc513f942/f56fc196-eac0-4cd8-bcf3-2f0b409d3828,media=disk,if=ide,cache=off,index=0,serial=72-9700-b0ecc513f942,boot=off,format=qcow2,werror=stop -drive file=/rhev/data-center/00000000-0000-0000-0000-000000000002/00000000-0000-0000-0000-000000000012/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_2.2_51181.iso,media=cdrom,index=2,if=ide -fda fat:floppy:/tmp/8e204d74-4b59-4754-9a7a-ed6200eeb776v4zryv.floppy -pidfile /var/vdsm/8e204d74-4b59-4754-9a7a-ed6200eeb776.pid -soundhw ac97 -spice sslpassword=,sslciphersuite=DEFAULT,sslcert=/var/vdsm/ts/certs/vdsmcert.pem,sslkey=/var/vdsm/ts/keys/vdsmkey.pem,ssldhfile=/var/vdsm/ts/keys/dh.pem,sslcafile=/var/vdsm/ts/certs/cacert.pem,host=0,secure-channels=main+inputs,ic=on,sport=5890,port=5910 -qxl 1 -cpu qemu64,+sse2 -M rhel5.5.0 -notify all -balloon none -smbios type=1,manufacturer=Red Hat,product=RHEL,version=5Server-,serial=AEFA9F8F-A3EB-3AA5-A112-12FA3AEF464F_00:1a:64:12:34:99,uuid=8e204d74-4b59-4754-9a7a-ed6200eeb776 -vmchannel di:0200,unix:/var/vdsm/8e204d74-4b59-4754-9a7a-ed6200eeb776.guest.socket,server -monitor unix:/var/vdsm/8e204d74-4b59-4754-9a7a-ed6200eeb776.monitor.socket,server
Comment 1 Oded Ramraz 2010-11-24 06:00:30 EST
I'm using IDE disk and using Virio network adapter ( Installed with latest 2.2 RHEV-tools )
Comment 2 Dor Laor 2010-11-24 07:11:35 EST
Needed info to isolate the problem

 - Try Linux guest to isolate winXp and ide/virtio driver issues
 - Don't copy files but use standard benchmark test or DD
 - Does the host test uses NFS as well?
 - When you use DD, try it with odirect mode (the dd command, in addition to 
Comment 3 Kevin Wolf 2010-11-24 08:07:53 EST
If it's qcow2 only, it's most likely due to the metadata flushes. I'm working on getting the impact of them small in upstream by batching requests, but block-queue is even there a very intrusive patch and I don't see any chance to backport it to RHEL 5. There are a few patches that can reduce the flushes a bit and should be possible to backport to RHEL 5, but they won't be able to compensate for the whole impact.

In any case, the performance should be better as soon as you start working on already allocated clusters (or if you preallocate metadata), just the initial growth is slow. Can you confirm you see this behaviour?
Comment 4 Oded Ramraz 2010-11-24 11:20:01 EST
I try to run my tests again with RAW ( both preallocated and Sparse ) and i didn't saw the problem. 
It might happen only with qcow2.
Comment 5 Dor Laor 2010-11-28 05:08:45 EST
Oded, do you have the data from comment #2?
Comment 6 Oded Ramraz 2010-11-28 05:44:47 EST
(In reply to comment #2)
> Needed info to isolate the problem
>  - Try Linux guest to isolate winXp and ide/virtio driver issues
>  - Don't copy files but use standard benchmark test or DD
>  - Does the host test uses NFS as well?   
>  - When you use DD, try it with odirect mode (the dd command, in addition to 
>    qemu)

I'm running IOMeter benchmark from a VM snapshot ( COW Sparse ) based on COW Sparse template . I'm using NFS storage.
I tried to reproduce this issue with RAW images on NFS ( Both Sparse / Preallocated ) without success.
I also tried to reproduce this issue on ISCSI: ( RAW Preallocated , COW Sparse  ) without success.
I assume that the problem is qcow2 related when using NFS storage.

If we need to test other kind of guests / DD odirect modes  in order to isolate the problem please ask KVM QE guys to perform these tests.
Comment 7 Dor Laor 2010-11-28 06:16:01 EST
What do you mean there is no problem? What are the performance?
Comment 8 Oded Ramraz 2010-11-28 17:02:10 EST
I run IOMeter benchmark again with NFS setup:
32kb IO size 50% read 50% write sequential write.
For RAW / Sparse and RAW preallocated I've got about 4 MB/Sec (2 read and 2 write)
For COW Sparse i got about 2.5 MB/Sec ( 1.25 read and 1.25 Write) 
I'm not sure that this issue is relevant for qcow2 disk type only.
Comment 9 Oded Ramraz 2010-11-28 17:05:03 EST
Created attachment 463378 [details]
IOMeter benchmark results
Comment 10 Dor Laor 2010-11-28 17:19:53 EST
The above is pretty slow even for raw. We might want to test it on a faster server.

Note You need to log in before you can comment on or make changes to this bug.