Bug 509383 - rotational mode is much faster for virtio-blk disks, but uses non-rotational mode by default
rotational mode is much faster for virtio-blk disks, but uses non-rotational ...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
All Linux
high Severity medium
: ---
: ---
Assigned To: Justin M. Forbes
Fedora Extras Quality Assurance
: Reopened
: 504376 (view as bug list)
Depends On:
Blocks: 513462 F11VirtTarget
  Show dependency treegraph
 
Reported: 2009-07-02 10:40 EDT by Richard W.M. Jones
Modified: 2009-09-29 10:34 EDT (History)
13 users (show)

See Also:
Fixed In Version: 2.6.30.8-64.fc11
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-17 09:36:05 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
mke2fs.sh (535 bytes, text/plain)
2009-07-02 10:45 EDT, Richard W.M. Jones
no flags Details
mke2fs.sh (557 bytes, text/plain)
2009-07-02 11:05 EDT, Richard W.M. Jones
no flags Details

  None (edit)
Description Richard W.M. Jones 2009-07-02 10:40:32 EDT
Description of problem:

The mkfs.ext2 command inside a guest runs much slower if the disks
are backed by virtio, than if they are backed by emulated IDE.
The difference in speed is roughly 6-10 times.

I created this bug because it looks like the issue is different
(but maybe related) to bug 504376.

Version-Release number of selected component (if applicable):

qemu-kvm-0.10.50-6.kvm86.fc12

How reproducible:

Always.

Steps to Reproduce:

[I will attached a simple test script which reproduces the problem]
  
Actual results:

mkfs.ext2 really slow with if=virtio.

Expected results:

mkfs.ext2 should be fast.

Additional info:
Comment 1 Richard W.M. Jones 2009-07-02 10:45:23 EDT
Created attachment 350290 [details]
mke2fs.sh

This is a libguestfs / guestfish script which reproduces
the problem.

Total runtime goes from 30s (virtio) to 21s (IDE).

HOWEVER that includes the overhead of starting up the
libguestfs appliance.  The time taken for the mke2fs
command goes from 8.63s (virtio) to 0.92s (IDE).

That means IDE is ~ 9 times faster than virtio at
running the mkfs.ext2 command.
Comment 2 Richard W.M. Jones 2009-07-02 11:05:38 EDT
Created attachment 350293 [details]
mke2fs.sh

Improved test script.  This really does need libguestfs
built from source / libguestfs >= 1.0.55.

Output from my laptop:

Testing using virtio backing disk:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 24.7128 s, 21.2 MB/s
elapsed time: 10.89 seconds
Testing using IDE backing disk:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 21.7416 s, 24.1 MB/s
elapsed time: 1.31 seconds
Comment 3 Richard W.M. Jones 2009-07-02 16:31:00 EDT
Thanks for Michael Tokarev and Charles Duffy for the
following suggestion which has mostly fixed this problem:

for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done

(inside the guest obviously).

Now the times are down to:
  virtio: 1.39s
  IDE:    0.98s

So it's not perfect, but the situation is greatly improved
by this change.
Comment 4 Richard W.M. Jones 2009-07-02 17:00:33 EDT
Pushed a cset to libguestfs to implement this:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=373a9ad0abb09ee079da834d251a744cdbe9ae70
Comment 5 Fedora Update System 2009-07-02 19:26:04 EDT
libguestfs-1.0.55-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.55-1.fc11
Comment 6 Mark McLoughlin 2009-07-03 09:18:30 EDT
virtio_blk is defaulting to non-rotational mode since these commits in 2.6.29:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=88e740f165
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=7d116b626b

The discussion that lead to the changes:

  http://lkml.org/lkml/2008/10/27/84

hch: any thoughts on this?
Comment 7 Chuck Ebbert 2009-07-06 14:06:08 EDT
Umm, maybe try elevator=deadline ?
Comment 8 Richard W.M. Jones 2009-07-06 15:07:10 EDT
(In reply to comment #7)
> Umm, maybe try elevator=deadline ?  

On my local machine using the test described below, elevator=deadline
in the guest makes no difference.

http://rwmj.wordpress.com/2009/07/04/time-filesystem-operations-using-guestfish/
Comment 9 chellwig@redhat.com 2009-07-06 16:07:20 EDT
It looks like the current virtio code defintively needs the plugs that rotational=0 removes.  That fits quite well into the bigger pictures as qemu with virtio still introduces far too large latencies for small I/Os.
Comment 10 Fedora Update System 2009-07-07 16:17:05 EDT
libguestfs-1.0.56-1.el5 has been submitted as an update for Fedora EPEL 5.
http://admin.fedoraproject.org/updates/libguestfs-1.0.56-1.el5
Comment 11 Fedora Update System 2009-07-07 17:19:45 EDT
libguestfs-1.0.56-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.56-2.fc11
Comment 12 Erik Jacobson 2009-07-08 21:29:11 EDT
*** Bug 504376 has been marked as a duplicate of this bug. ***
Comment 13 Erik Jacobson 2009-07-08 21:33:03 EDT
See my comments in bug 504376 with test results on a mkfs.ext3 to a virtio
device with and without rotation enabled.  I was asked to dupe it here.
Comment 14 Fedora Update System 2009-07-09 01:42:36 EDT
libguestfs-1.0.56-1.el5 has been pushed to the Fedora EPEL 5 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/EL-5/FEDORA-EPEL-2009-0019
Comment 15 Mark McLoughlin 2009-07-09 04:00:46 EDT
rich: the libguestfs update does not resolve this bug, could you remove the bug # from the update, or clone this bug to libguestfs and use that bug number ?
Comment 16 Richard W.M. Jones 2009-07-09 04:38:14 EDT
OK, I edited the update to remove this bug number.
Comment 17 Fedora Update System 2009-07-11 05:27:01 EDT
libguestfs-1.0.58-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.58-2.fc11
Comment 18 Fedora Update System 2009-07-11 13:24:01 EDT
libguestfs-1.0.55-1.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7493
Comment 19 Fedora Update System 2009-07-16 02:52:45 EDT
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libguestfs'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557
Comment 20 Mark McLoughlin 2009-07-16 04:52:16 EDT
(In reply to comment #16)
> OK, I edited the update to remove this bug number.  

Still there.
Comment 21 Richard W.M. Jones 2009-07-16 05:04:11 EDT
Bodhi seems to have copied the bug number into half a dozen
different updates.  I don't want to edit
http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557
yet because if I do that, it will set the status back to
pending, and I want it stable ASAP.

Instead I'll just keep an eye on this bug and undo any
changes that the update system makes.
Comment 22 Fedora Update System 2009-07-19 06:27:31 EDT
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 23 Richard W.M. Jones 2009-07-19 06:55:08 EDT
Damn.  Hopefully the updates of doom will leave us alone now.
Comment 24 Mark McLoughlin 2009-08-07 07:30:45 EDT
(In reply to comment #9)
> It looks like the current virtio code defintively needs the plugs that
> rotational=0 removes.  That fits quite well into the bigger pictures as qemu
> with virtio still introduces far too large latencies for small I/Os.  

Can we get the default changed back to non-rotational mode, then?
Comment 25 chellwig@redhat.com 2009-09-01 11:59:26 EDT
I've done a lot of benchmarking, and I can't actually see any difference between rotational=0 vs =1 at all.

This is all using raw images, sparse or not.  You aren't doing something braindead like using qcow2, are you?
Comment 26 Mark McLoughlin 2009-09-04 08:24:19 EDT
(In reply to comment #25)
> I've done a lot of benchmarking, and I can't actually see any difference
> between rotational=0 vs =1 at all.

Strange

> This is all using raw images, sparse or not.  You aren't doing something
> braindead like using qcow2, are you?  

Nope, looks like Rich was using a non-sparse raw image with cache=none
Comment 27 Mark McLoughlin 2009-09-06 07:11:15 EDT
Christoph has posted a patch to to switch back to rotational mode:

  http://patchwork.kernel.org/patch/45787/
Comment 28 Mark McLoughlin 2009-09-06 07:13:16 EDT
Justin: maybe you could apply the patch?
Comment 29 Justin M. Forbes 2009-09-16 11:08:08 EDT
This patch has been added to the F12 kernels.
Comment 30 Mark McLoughlin 2009-09-17 09:36:05 EDT
Fixed in F12 since kernel-2.6.31-0.138.rc5.git3.fc12 and F11 since 2.6.30.7-58

* Wed Sep 09 2009 Justin M. Forbes <jforbes@redhat.com>
- Revert virio_blk to rotational mode. (#509383)
Comment 31 Fedora Update System 2009-09-25 12:02:02 EDT
kernel-2.6.30.8-64.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/kernel-2.6.30.8-64.fc11
Comment 32 Fedora Update System 2009-09-29 10:33:17 EDT
kernel-2.6.30.8-64.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.