Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 621834 - Poor write I/O performance of virtio-win driver
Poor write I/O performance of virtio-win driver
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: virtio-win (Show other bugs)
6.1
All Linux
low Severity medium
: beta
: ---
Assigned To: Vadim Rozenfeld
Virtualization Bugs
: RHELNAK
Depends On:
Blocks: 580954
  Show dependency treegraph
 
Reported: 2010-08-06 04:50 EDT by Quan Wenli
Modified: 2013-01-09 17:59 EST (History)
7 users (show)

See Also:
Fixed In Version: virtio-win-1.1.19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-04-21 04:12:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sar.xp.ide.out (5.67 MB, application/octet-stream)
2010-08-06 04:52 EDT, Quan Wenli
no flags Details
sar.xp.virtio.out (3.40 MB, application/octet-stream)
2010-08-06 04:53 EDT, Quan Wenli
no flags Details
win2k8-deadline.ide.sar.out (2.88 MB, application/octet-stream)
2010-08-11 01:31 EDT, Quan Wenli
no flags Details
win2k8-deadline.virtio.sar.out (Host) (3.19 MB, application/octet-stream)
2010-08-11 01:32 EDT, Quan Wenli
no flags Details
kernel-59-qemu-109-win2k8datacenter-virtio-vs-ide-virtio-win.1.1.10-compare-iometer.ods (18.05 KB, application/vnd.oasis.opendocument.spreadsheet)
2010-08-12 07:47 EDT, Quan Wenli
no flags Details
win2k8r2-deadline-virtio.1.1.10-qemu-109-kernel-59/kernel-59-qemu-109-win2k8datacenter-virtio-ide-win.1.1.10-compare-iometer.ods.rawdata.csv (290.23 KB, text/csv)
2010-08-12 07:50 EDT, Quan Wenli
no flags Details
/win2k8r2-deadline-virtio.1.1.10-qemu-109-kernel-59/kernel-59-qemu-109-win2k8datacenter-virtio-win.1.1.10-compare-iometer.ods.rawdata.csv (292.33 KB, text/csv)
2010-08-12 07:52 EDT, Quan Wenli
no flags Details
iometer icf file which I used for testing. (8.27 KB, application/vnd.commerce-battelle)
2010-08-15 22:34 EDT, Quan Wenli
no flags Details
Win2k8x64R2 & Win7x64 result for RC. (125.41 KB, application/x-gzip)
2010-10-19 03:20 EDT, Quan Wenli
no flags Details
post rhel6.1 iometer results (267.43 KB, application/x-bzip2)
2011-03-17 23:14 EDT, Quan Wenli
no flags Details
rhel6.1-kernel-2.6.32-125-qemu-img-0.12.1.2-2.152-deadline-win2k8rc2sp1-virtio-win-prewhql-0.1-8 (1.35 MB, application/x-bzip2)
2011-04-06 05:21 EDT, Quan Wenli
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0782 normal SHIPPED_LIVE virtio-win bug fix and enhancement update 2011-05-18 14:08:21 EDT

  None (edit)
Description Quan Wenli 2010-08-06 04:50:59 EDT
Description of problem:

The write  throughput of virtio is always lower than ide in the iozone
test(direct io).

Info:
----------
guest (2pus 2G mem)
Host (8cpus 24G mem) 
Block I/O elevator(Host,guest) : CFQ(default)
swap off
iozone verson : 3.344


Output is in Kytes/sec   |  Write |Rewrite | Read | Reread| 
-------------------------+-------- +-------+------+-------+-
ide                         59600   76877    58079  60030
virtio                      51214   89766    65934  65934


CPU uilization %  | Write | Rewrite | Read | Reread | Host ( top qemu-kvm average)

------------------+------ +---------+------+--------+--------------
ide                 55.18  58.8      30.28   29.67   76.44
virtio              47.77  70.18     29.19   29.07   74.34


sar-average-ide (collect on the host during iozone running on the guest)

03:35:14 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      6.57      0.00      3.30      8.54      0.00     81.59

sar-average-virtio (collect on the host during iozone running on the guest)

04:10:42 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      6.42      0.00      3.07     12.04      0.00     78.47




Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.104.el6.x86_64
kernel-2.6.32-54.el6.x86_64
virtio-win-1.1.10-0


How reproducible:

always 
Steps to Reproduce:
1.CIL
/usr/libexec/qemu-kvm -m 2G -smp 2 -drive file=/home/WindowXp-32-virtio,cache=none,format=raw,if=none,id=drive-ide0-0-0 -device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,mac=00:00:11:12:31:4B,bus=pci.0,addr=0x4 -boot c -uuid 1ed1bec6-fe5e-4947-84f4-dfc5c4e48557 -rtc base=utc -no-kvm-pit-reinjection -cpu qemu64,+sse2,+x2apic -balloon none -monitor stdio -vnc :1 -usbdevice tablet
2.iozone -az -i 0 -i 1 -n 10G -g 10G -r 64k -Rb ./XP-ide.xls -+u
3.use top to record cpu uilization of qemu-kvm durning running iozone.
then calculate the average of them. 
4.use sar to collect cpu detail uilization during the iozone running.
then calculate the average of them.
  
Actual results:
Compare wirte I/O performance of ide block device, virtio block device could be
better .


Expected results:


Additional info:
Comment 2 Quan Wenli 2010-08-06 04:52:36 EDT
Created attachment 437076 [details]
sar.xp.ide.out
Comment 3 Quan Wenli 2010-08-06 04:53:29 EDT
Created attachment 437077 [details]
sar.xp.virtio.out
Comment 4 RHEL Product and Program Management 2010-08-06 05:07:51 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 5 Yaniv Kaul 2010-08-06 06:10:36 EDT
Would be interesting to compare with a server OS as well (Win2008 and such).
Comment 6 Vadim Rozenfeld 2010-08-11 00:26:36 EDT
(In reply to comment #0)
> Description of problem:
> The write  throughput of virtio is always lower than ide in the iozone
> test(direct io).
> Info:
> ----------
> guest (2pus 2G mem)
> Host (8cpus 24G mem) 
> Block I/O elevator(Host,guest) : CFQ(default)
> swap off
> iozone verson : 3.344
> Output is in Kytes/sec   |  Write |Rewrite | Read | Reread| 
> -------------------------+-------- +-------+------+-------+-
> ide                         59600   76877    58079  60030
> virtio                      51214   89766    65934  65934
> CPU uilization %  | Write | Rewrite | Read | Reread | Host ( top qemu-kvm
> average)
> ------------------+------ +---------+------+--------+--------------
> ide                 55.18  58.8      30.28   29.67   76.44
> virtio              47.77  70.18     29.19   29.07   74.34
> sar-average-ide (collect on the host during iozone running on the guest)
> 03:35:14 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
> Average:        all      6.57      0.00      3.30      8.54      0.00     81.59
> sar-average-virtio (collect on the host during iozone running on the guest)
> 04:10:42 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
> Average:        all      6.42      0.00      3.07     12.04      0.00     78.47
> Version-Release number of selected component (if applicable):
> qemu-kvm-0.12.1.2-2.104.el6.x86_64
> kernel-2.6.32-54.el6.x86_64
> virtio-win-1.1.10-0
> How reproducible:
> always 
> Steps to Reproduce:
> 1.CIL
> /usr/libexec/qemu-kvm -m 2G -smp 2 -drive
> file=/home/WindowXp-32-virtio,cache=none,format=raw,if=none,id=drive-ide0-0-0
> -device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0
> -device rtl8139,netdev=hostnet0,mac=00:00:11:12:31:4B,bus=pci.0,addr=0x4 -boot
> c -uuid 1ed1bec6-fe5e-4947-84f4-dfc5c4e48557 -rtc base=utc
> -no-kvm-pit-reinjection -cpu qemu64,+sse2,+x2apic -balloon none -monitor stdio
> -vnc :1 -usbdevice tablet
> 2.iozone -az -i 0 -i 1 -n 10G -g 10G -r 64k -Rb ./XP-ide.xls -+u
> 3.use top to record cpu uilization of qemu-kvm durning running iozone.
> then calculate the average of them. 
> 4.use sar to collect cpu detail uilization during the iozone running.
> then calculate the average of them.
> Actual results:
> Compare wirte I/O performance of ide block device, virtio block device could be
> better .
> Expected results:
> Additional info:    

Hi, Quan.
Thank you for performance testing.
Could you please re-test with elevator deadline.
Could you also try IoMeter as another disk benchmark tool.
Thanks,
Vadim.
Comment 7 Quan Wenli 2010-08-11 01:28:21 EDT
Hi,Vadim

Here is the result with elevator deadline by using Win2k8 R2 data center 64bits.
I will try IoMeter later, do you have any special trasnfer request size & queue depth requirement ?

Info:
----------
guest (2pus 2G mem)
Host (8cpus 24G mem) 
Block I/O elevator(Host,guest) : deadline 
swap off
iozone verson : 3.344
qemu-kvm-0.12.1.2-2.109.el6.x86_64
kernel-2.6.32-59.el6.x86_64
virtio-win-1.1.10-0



Output is in MB/sec      |  Write |Rewrite | Read | Reread| 
-------------------------+-------- +-------+------+-------+-
ide                         115.43   125.41  111.32  114.21
virtio                      84.8     89.07   131.76  132.93
ide vs virtio in %          -26.54%  -28.97% 18.36%  16.40%



CPU uilization %  | Write | Rewrite | Read | Reread | Host-CPU( 8 cpus)

------------------+------ +---------+------+--------+--------------
ide                  6.6      7.3     6.73   7.23     42.77
virtio               4.9     70.18    4.74   8.46     43.45
ide vs virtio in % -25.77% -35.04%   25.84%  9.93%    1.59%
Comment 8 Quan Wenli 2010-08-11 01:31:23 EDT
Created attachment 438078 [details]
win2k8-deadline.ide.sar.out
Comment 9 Quan Wenli 2010-08-11 01:32:30 EDT
Created attachment 438079 [details]
win2k8-deadline.virtio.sar.out (Host)
Comment 10 Vadim Rozenfeld 2010-08-11 06:56:43 EDT
(In reply to comment #7)
> Hi,Vadim
> Here is the result with elevator deadline by using Win2k8 R2 data center
> 64bits.
> I will try IoMeter later, do you have any special trasnfer request size & queue
> depth requirement ?

Hi, Quan.
Usually we check 2K/4K/8K/16K/32K/64K block sizes with 1/2/4/8/16/32/64
requests in queue.

Thanks,
Vadim.
Comment 11 Quan Wenli 2010-08-12 07:42:38 EDT
Hi Vadim 

Attached organized & raw result about virtio and ide performance data.
We can see
1.read performance 

virtio drop -10%~17% only when 32K/64K block sizes with 64 requests in queue ,other scenario it boost +3~500% compare with ide.

2.write performance 

virtio drop - 70%~90% when 2K/4K/8K/16K/32K/64K block sizes with 16/32/64 requests in queue
Comment 12 Quan Wenli 2010-08-12 07:47:22 EDT
Created attachment 438410 [details]
kernel-59-qemu-109-win2k8datacenter-virtio-vs-ide-virtio-win.1.1.10-compare-iometer.ods
Comment 13 Quan Wenli 2010-08-12 07:50:43 EDT
Created attachment 438412 [details]
win2k8r2-deadline-virtio.1.1.10-qemu-109-kernel-59/kernel-59-qemu-109-win2k8datacenter-virtio-ide-win.1.1.10-compare-iometer.ods.rawdata.csv
Comment 14 Quan Wenli 2010-08-12 07:52:02 EDT
Created attachment 438413 [details]
/win2k8r2-deadline-virtio.1.1.10-qemu-109-kernel-59/kernel-59-qemu-109-win2k8datacenter-virtio-win.1.1.10-compare-iometer.ods.rawdata.csv
Comment 15 Vadim Rozenfeld 2010-08-15 16:36:33 EDT
Hi, Quan.

Can you please post the icf files?
I have rechecked performance with the configuration files I got from our performance team. The results are slightly different from yours, and it is the reason why I'm interested in rerunning IoMeter with yours configuration files.

Thanks,
Vadim.
Comment 16 Quan Wenli 2010-08-15 22:34:57 EDT
Created attachment 438881 [details]
iometer icf file which I used for testing.
Comment 17 Quan Wenli 2010-10-19 03:14:37 EDT
Retest virio-win-1.1.2 by using iometer tool with profile from comment#16.


Here is the total result as the sum  of 2K/4K/8K/16K/32K/64K block sizes with 1/2/4/8/16/32/64 requests in queue.
virtio-win write performance still drop about 27% comparing  with ide driver.


Win2k8x64rc2
Output is in MB/sec | Read  | Write |                                          
--------------------+------- +------+
ide                  1353.19	793.44
virtio               2926.1	575.38    
ide vs virtio in %   116.24%	-27.48%      


CPU uilization %    | Read  | Write |   
                                      
--------------------+--------+------+
ide                  170.87    11.64
virtio               507.36    170.39
ide vs virtio in %   196.93%	1363.83%


Win7x64
Output is in MB/sec |  Read   | Write |   
                                               
--------------------+-------- +------+
ide                  2108.39	1567.29
virtio               2082.7	1130.78
ide vs virtio in %   -1.22%	-27.85%
      

CPU uilization %    |Read     | Write |   
                                               
--------------------+-------- +------+
ide                   513.1	412.82
virtio                938.71	282.06
ide vs virtio in %   82.95%	-31.67%
Comment 18 Quan Wenli 2010-10-19 03:20:52 EDT
Created attachment 454285 [details]
Win2k8x64R2 & Win7x64 result for RC.
Comment 20 Vadim Rozenfeld 2011-02-12 13:36:07 EST
Some performance improvement are expected after switching to MSI-X mode (on Win2K8 and higher platforms) and turning on indirect buffers support (all platforms).
Comment 21 Vadim Rozenfeld 2011-02-15 10:02:21 EST
Indirect buffers support will be turned on in rhel6.2. 
Meanwhile, please try the latest drivers:

http://download.lab.bos.redhat.com/devel/RHEV/virtio-win/1.1.19/
Comment 27 Quan Wenli 2011-03-31 05:12:39 EDT
Hi,Vadim

any comments for this bug?
Comment 28 Vadim Rozenfeld 2011-03-31 05:58:02 EDT
(In reply to comment #27)
> Hi,Vadim
> 
> any comments for this bug?

Hi, Quan.

Frankly, viostor doesn't look very impressive on write. 
What is your storage HW?
Do you use deadline scheduler?
Best regards,
Vadim.
Comment 29 Quan Wenli 2011-03-31 06:42:10 EDT
(In reply to comment #28)
> (In reply to comment #27)
> > Hi,Vadim
> > 
> > any comments for this bug?
> 
> Hi, Quan.
> 
> Frankly, viostor doesn't look very impressive on write. 
> What is your storage HW?
> Do you use deadline scheduler?
> Best regards,
> Vadim.

I used the default scheduler cfq on the host,need I test it again under the deadline?
Comment 30 Vadim Rozenfeld 2011-03-31 07:07:16 EDT
(In reply to comment #29)
> (In reply to comment #28)
> > (In reply to comment #27)
> > > Hi,Vadim
> > > 
> > > any comments for this bug?
> > 
> > Hi, Quan.
> > 
> > Frankly, viostor doesn't look very impressive on write. 
> > What is your storage HW?
> > Do you use deadline scheduler?
> > Best regards,
> > Vadim.
> 
> I used the default scheduler cfq on the host,need I test it again under the
> deadline?

Yes, please.
deadline is strongly recommended, if not mandatory, for virtio.
Best regards,
Vadim.
Comment 31 Quan Wenli 2011-04-06 05:19:39 EDT
Hi,Vadim

Please help us to check the result for the bug.

env:

Host  :kernel-2.6.32-125 & qemu-0.12.1.2-2.152
Schedule: elevator=deadline on the host
guest :window2K8RC2 sp1 
virtio driver : virtio-win-prewhql-0.1-8
cil:/usr/libexec/qemu-kvm -name iometer -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic,family=0xf -vnc :1 -vga qxl -rtc base=utc,clock=host,driftfix=none -M rhel6.1.0 -boot order=cdn,once=c,menu=off   -usbdevice tablet -no-kvm-pit-reinjection -enable-kvm -drive file=/home/windows-2k8-r2-sp1-raw-virtio,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=raw,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/home/10G,index=3,if=none,id=drive-virtio-disk2,media=disk,cache=none,format=raw,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2  -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:13:01:d1,bus=pci.0,addr=0x3 -netdev tap,id=hostnet0,script=/etc/qemu-ifup

Overall result :

                   6.1_ide   6.1_virtio_0xe     6.1_virtio_0xf

Read-total(MBps)   4943.6        6802.64          6893.83
Write-total(MBps)  3573.42       3633.68          3617.09 
Readcpu-total      1454          1609.44          1616.54
Writecpu-total     887.81        756.01           733.6


For detail results and  configure file ,please check attachment rhel6.1-kernel-2.6.32-125-qemu-img-0.12.1.2-2.152-deadline-win2k8rc2sp1-virtio-win-prewhql-0.1-8-iometer.tar.bz2
Comment 32 Quan Wenli 2011-04-06 05:21:39 EDT
Created attachment 490221 [details]
rhel6.1-kernel-2.6.32-125-qemu-img-0.12.1.2-2.152-deadline-win2k8rc2sp1-virtio-win-prewhql-0.1-8
Comment 35 Qunfang Zhang 2011-04-21 01:14:28 EDT
Hi, Vadim
As this bug is included in Errata RHBA-2011:10808-01 which we need to push by Apr 27. So could you reply in this bz if it can be verified pass or something else?

Thanks.
Qunfang

Note You need to log in before you can comment on or make changes to this bug.