Red Hat Bugzilla – Bug 509383
rotational mode is much faster for virtio-blk disks, but uses non-rotational mode by default
Last modified: 2009-09-29 10:34:06 EDT
Description of problem: The mkfs.ext2 command inside a guest runs much slower if the disks are backed by virtio, than if they are backed by emulated IDE. The difference in speed is roughly 6-10 times. I created this bug because it looks like the issue is different (but maybe related) to bug 504376. Version-Release number of selected component (if applicable): qemu-kvm-0.10.50-6.kvm86.fc12 How reproducible: Always. Steps to Reproduce: [I will attached a simple test script which reproduces the problem] Actual results: mkfs.ext2 really slow with if=virtio. Expected results: mkfs.ext2 should be fast. Additional info:
Created attachment 350290 [details] mke2fs.sh This is a libguestfs / guestfish script which reproduces the problem. Total runtime goes from 30s (virtio) to 21s (IDE). HOWEVER that includes the overhead of starting up the libguestfs appliance. The time taken for the mke2fs command goes from 8.63s (virtio) to 0.92s (IDE). That means IDE is ~ 9 times faster than virtio at running the mkfs.ext2 command.
Created attachment 350293 [details] mke2fs.sh Improved test script. This really does need libguestfs built from source / libguestfs >= 1.0.55. Output from my laptop: Testing using virtio backing disk: 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 24.7128 s, 21.2 MB/s elapsed time: 10.89 seconds Testing using IDE backing disk: 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 21.7416 s, 24.1 MB/s elapsed time: 1.31 seconds
Thanks for Michael Tokarev and Charles Duffy for the following suggestion which has mostly fixed this problem: for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done (inside the guest obviously). Now the times are down to: virtio: 1.39s IDE: 0.98s So it's not perfect, but the situation is greatly improved by this change.
Pushed a cset to libguestfs to implement this: http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=373a9ad0abb09ee079da834d251a744cdbe9ae70
libguestfs-1.0.55-1.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/libguestfs-1.0.55-1.fc11
virtio_blk is defaulting to non-rotational mode since these commits in 2.6.29: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=88e740f165 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=7d116b626b The discussion that lead to the changes: http://lkml.org/lkml/2008/10/27/84 hch: any thoughts on this?
Umm, maybe try elevator=deadline ?
(In reply to comment #7) > Umm, maybe try elevator=deadline ? On my local machine using the test described below, elevator=deadline in the guest makes no difference. http://rwmj.wordpress.com/2009/07/04/time-filesystem-operations-using-guestfish/
It looks like the current virtio code defintively needs the plugs that rotational=0 removes. That fits quite well into the bigger pictures as qemu with virtio still introduces far too large latencies for small I/Os.
libguestfs-1.0.56-1.el5 has been submitted as an update for Fedora EPEL 5. http://admin.fedoraproject.org/updates/libguestfs-1.0.56-1.el5
libguestfs-1.0.56-2.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/libguestfs-1.0.56-2.fc11
*** Bug 504376 has been marked as a duplicate of this bug. ***
See my comments in bug 504376 with test results on a mkfs.ext3 to a virtio device with and without rotation enabled. I was asked to dupe it here.
libguestfs-1.0.56-1.el5 has been pushed to the Fedora EPEL 5 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update libguestfs'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/EL-5/FEDORA-EPEL-2009-0019
rich: the libguestfs update does not resolve this bug, could you remove the bug # from the update, or clone this bug to libguestfs and use that bug number ?
OK, I edited the update to remove this bug number.
libguestfs-1.0.58-2.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/libguestfs-1.0.58-2.fc11
libguestfs-1.0.55-1.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update libguestfs'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7493
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update libguestfs'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557
(In reply to comment #16) > OK, I edited the update to remove this bug number. Still there.
Bodhi seems to have copied the bug number into half a dozen different updates. I don't want to edit http://admin.fedoraproject.org/updates/F11/FEDORA-2009-7557 yet because if I do that, it will set the status back to pending, and I want it stable ASAP. Instead I'll just keep an eye on this bug and undo any changes that the update system makes.
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.
Damn. Hopefully the updates of doom will leave us alone now.
(In reply to comment #9) > It looks like the current virtio code defintively needs the plugs that > rotational=0 removes. That fits quite well into the bigger pictures as qemu > with virtio still introduces far too large latencies for small I/Os. Can we get the default changed back to non-rotational mode, then?
I've done a lot of benchmarking, and I can't actually see any difference between rotational=0 vs =1 at all. This is all using raw images, sparse or not. You aren't doing something braindead like using qcow2, are you?
(In reply to comment #25) > I've done a lot of benchmarking, and I can't actually see any difference > between rotational=0 vs =1 at all. Strange > This is all using raw images, sparse or not. You aren't doing something > braindead like using qcow2, are you? Nope, looks like Rich was using a non-sparse raw image with cache=none
Christoph has posted a patch to to switch back to rotational mode: http://patchwork.kernel.org/patch/45787/
Justin: maybe you could apply the patch?
This patch has been added to the F12 kernels.
Fixed in F12 since kernel-2.6.31-0.138.rc5.git3.fc12 and F11 since 2.6.30.7-58 * Wed Sep 09 2009 Justin M. Forbes <jforbes@redhat.com> - Revert virio_blk to rotational mode. (#509383)
kernel-2.6.30.8-64.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/kernel-2.6.30.8-64.fc11
kernel-2.6.30.8-64.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.