Bug 509383 - rotational mode is much faster for virtio-blk disks, but uses non-rotational mode by default
Summary: rotational mode is much faster for virtio-blk disks, but uses non-rotational ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: All
OS: Linux
high
medium
Target Milestone: ---
Assignee: Justin M. Forbes
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 504376 (view as bug list)
Depends On:
Blocks: F11VirtTarget 513462
TreeView+ depends on / blocked
 
Reported: 2009-07-02 14:40 UTC by Richard W.M. Jones
Modified: 2009-09-29 14:34 UTC (History)
13 users (show)

Fixed In Version: 2.6.30.8-64.fc11
Clone Of:
Environment:
Last Closed: 2009-09-17 13:36:05 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
mke2fs.sh (535 bytes, text/plain)
2009-07-02 14:45 UTC, Richard W.M. Jones
no flags Details
mke2fs.sh (557 bytes, text/plain)
2009-07-02 15:05 UTC, Richard W.M. Jones
no flags Details

Description Richard W.M. Jones 2009-07-02 14:40:32 UTC
Description of problem:

The mkfs.ext2 command inside a guest runs much slower if the disks
are backed by virtio, than if they are backed by emulated IDE.
The difference in speed is roughly 6-10 times.

I created this bug because it looks like the issue is different
(but maybe related) to bug 504376.

Version-Release number of selected component (if applicable):

qemu-kvm-0.10.50-6.kvm86.fc12

How reproducible:

Always.

Steps to Reproduce:

[I will attached a simple test script which reproduces the problem]
  
Actual results:

mkfs.ext2 really slow with if=virtio.

Expected results:

mkfs.ext2 should be fast.

Additional info:

Comment 1 Richard W.M. Jones 2009-07-02 14:45:23 UTC
Created attachment 350290 [details]
mke2fs.sh

This is a libguestfs / guestfish script which reproduces
the problem.

Total runtime goes from 30s (virtio) to 21s (IDE).

HOWEVER that includes the overhead of starting up the
libguestfs appliance.  The time taken for the mke2fs
command goes from 8.63s (virtio) to 0.92s (IDE).

That means IDE is ~ 9 times faster than virtio at
running the mkfs.ext2 command.

Comment 2 Richard W.M. Jones 2009-07-02 15:05:38 UTC
Created attachment 350293 [details]
mke2fs.sh

Improved test script.  This really does need libguestfs
built from source / libguestfs >= 1.0.55.

Output from my laptop:

Testing using virtio backing disk:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 24.7128 s, 21.2 MB/s
elapsed time: 10.89 seconds
Testing using IDE backing disk:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 21.7416 s, 24.1 MB/s
elapsed time: 1.31 seconds

Comment 3 Richard W.M. Jones 2009-07-02 20:31:00 UTC
Thanks for Michael Tokarev and Charles Duffy for the
following suggestion which has mostly fixed this problem:

for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done

(inside the guest obviously).

Now the times are down to:
  virtio: 1.39s
  IDE:    0.98s

So it's not perfect, but the situation is greatly improved
by this change.

Comment 4 Richard W.M. Jones 2009-07-02 21:00:33 UTC
Pushed a cset to libguestfs to implement this:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=373a9ad0abb09ee079da834d251a744cdbe9ae70

Comment 5 Fedora Update System 2009-07-02 23:26:04 UTC
libguestfs-1.0.55-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.55-1.fc11

Comment 6 Mark McLoughlin 2009-07-03 13:18:30 UTC
virtio_blk is defaulting to non-rotational mode since these commits in 2.6.29:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=88e740f165
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=7d116b626b

The discussion that lead to the changes:

  http://lkml.org/lkml/2008/10/27/84

hch: any thoughts on this?

Comment 7 Chuck Ebbert 2009-07-06 18:06:08 UTC
Umm, maybe try elevator=deadline ?

Comment 8 Richard W.M. Jones 2009-07-06 19:07:10 UTC
(In reply to comment #7)
> Umm, maybe try elevator=deadline ?  

On my local machine using the test described below, elevator=deadline
in the guest makes no difference.

http://rwmj.wordpress.com/2009/07/04/time-filesystem-operations-using-guestfish/

Comment 9 chellwig@redhat.com 2009-07-06 20:07:20 UTC
It looks like the current virtio code defintively needs the plugs that rotational=0 removes.  That fits quite well into the bigger pictures as qemu with virtio still introduces far too large latencies for small I/Os.

Comment 10 Fedora Update System 2009-07-07 20:17:05 UTC
libguestfs-1.0.56-1.el5 has been submitted as an update for Fedora EPEL 5.
http://admin.fedoraproject.org/updates/libguestfs-1.0.56-1.el5

Comment 11 Fedora Update System 2009-07-07 21:19:45 UTC
libguestfs-1.0.56-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.56-2.fc11

Comment 12 erikj 2009-07-09 01:29:11 UTC
*** Bug 504376 has been marked as a duplicate of this bug. ***

Comment 13 erikj 2009-07-09 01:33:03 UTC
See my comments in bug 504376 with test results on a mkfs.ext3 to a virtio
device with and without rotation enabled.  I was asked to dupe it here.

Comment 14 Fedora Update System 2009-07-09 05:42:36 UTC
libguestfs-1.0.56-1.el5 has been pushed to the Fedora EPEL 5 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/EL-5/FEDORA-EPEL-2009-0019

Comment 15 Mark McLoughlin 2009-07-09 08:00:46 UTC
rich: the libguestfs update does not resolve this bug, could you remove the bug # from the update, or clone this bug to libguestfs and use that bug number ?

Comment 16 Richard W.M. Jones 2009-07-09 08:38:14 UTC
OK, I edited the update to remove this bug number.

Comment 17 Fedora Update System 2009-07-11 09:27:01 UTC
libguestfs-1.0.58-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.58-2.fc11

Comment 18 Fedora Update System 2009-07-11 17:24:01 UTC
libguestfs-1.0.55-1.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7493

Comment 19 Fedora Update System 2009-07-16 06:52:45 UTC
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557

Comment 20 Mark McLoughlin 2009-07-16 08:52:16 UTC
(In reply to comment #16)
> OK, I edited the update to remove this bug number.  

Still there.

Comment 21 Richard W.M. Jones 2009-07-16 09:04:11 UTC
Bodhi seems to have copied the bug number into half a dozen
different updates.  I don't want to edit
http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557
yet because if I do that, it will set the status back to
pending, and I want it stable ASAP.

Instead I'll just keep an eye on this bug and undo any
changes that the update system makes.

Comment 22 Fedora Update System 2009-07-19 10:27:31 UTC
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 23 Richard W.M. Jones 2009-07-19 10:55:08 UTC
Damn.  Hopefully the updates of doom will leave us alone now.

Comment 24 Mark McLoughlin 2009-08-07 11:30:45 UTC
(In reply to comment #9)
> It looks like the current virtio code defintively needs the plugs that
> rotational=0 removes.  That fits quite well into the bigger pictures as qemu
> with virtio still introduces far too large latencies for small I/Os.  

Can we get the default changed back to non-rotational mode, then?

Comment 25 chellwig@redhat.com 2009-09-01 15:59:26 UTC
I've done a lot of benchmarking, and I can't actually see any difference between rotational=0 vs =1 at all.

This is all using raw images, sparse or not.  You aren't doing something braindead like using qcow2, are you?

Comment 26 Mark McLoughlin 2009-09-04 12:24:19 UTC
(In reply to comment #25)
> I've done a lot of benchmarking, and I can't actually see any difference
> between rotational=0 vs =1 at all.

Strange

> This is all using raw images, sparse or not.  You aren't doing something
> braindead like using qcow2, are you?  

Nope, looks like Rich was using a non-sparse raw image with cache=none

Comment 27 Mark McLoughlin 2009-09-06 11:11:15 UTC
Christoph has posted a patch to to switch back to rotational mode:

  http://patchwork.kernel.org/patch/45787/

Comment 28 Mark McLoughlin 2009-09-06 11:13:16 UTC
Justin: maybe you could apply the patch?

Comment 29 Justin M. Forbes 2009-09-16 15:08:08 UTC
This patch has been added to the F12 kernels.

Comment 30 Mark McLoughlin 2009-09-17 13:36:05 UTC
Fixed in F12 since kernel-2.6.31-0.138.rc5.git3.fc12 and F11 since 2.6.30.7-58

* Wed Sep 09 2009 Justin M. Forbes <jforbes>
- Revert virio_blk to rotational mode. (#509383)

Comment 31 Fedora Update System 2009-09-25 16:02:02 UTC
kernel-2.6.30.8-64.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/kernel-2.6.30.8-64.fc11

Comment 32 Fedora Update System 2009-09-29 14:33:17 UTC
kernel-2.6.30.8-64.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.