Bug 1243548 - AIO is processing only 1 request, even if >1 requests in virtio
Summary: AIO is processing only 1 request, even if >1 requests in virtio
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Stefan Hajnoczi
QA Contact: FuXiangChun
: 1285624 (view as bug list)
Depends On:
Blocks: 1285624 1288337 RHEV_IO_NATIVE_EVERYWHERE 1327613
TreeView+ depends on / blocked
Reported: 2015-07-15 18:49 UTC by Pradeep Kumar Surisetty
Modified: 2016-11-07 20:28 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-rhev-2.5.0-1.el7
Doc Type: Bug Fix
Doc Text:
When QEMU was configured with the aio=native parameter, KVM virtual machines were slowed significantly. With this update, asynchronous I/O (AIO) can correctly process more than one request at a time, and using aio=native no longer has a negative impact on guest performance.
Clone Of:
: 1285624 RHEV_IO_NATIVE_EVERYWHERE 1327613 (view as bug list)
Last Closed: 2016-11-07 20:28:09 UTC
Target Upstream Version:

Attachments (Terms of Use)
aio-nr-test-result (9.75 MB, text/plain)
2016-09-18 06:51 UTC, FuXiangChun
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 0 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Pradeep Kumar Surisetty 2015-07-15 18:49:44 UTC
Description of problem:

While running I/O workloads on RHEL 7.1 VM on RHEL 7.1 host using qemu-kvm-rhev: 
We realised aio=native is 50% lesser than aio=threads.   There are several requests to process. But AIO is proceesing one at a time.

Systemtap logs to monitor 
virtio_queue_notify: ts=1436461565480052,vdev=140628771509224,n=0,vq=140628770978384
virtio_blk_handle_read: ts=1436461565480071,req=140628771310288,sector=0,nsectors=8
virtio_blk_handle_read: ts=1436461565480134,req=140628771703808,sector=8,nsectors=8
virtio_blk_handle_read: ts=1436461565480150,req=140628771788928,sector=16,nsectors=8
virtio_blk_handle_read: ts=1436461565480165,req=140628771838192,sector=24,nsectors=8
virtio_blk_handle_read: ts=1436461565480179,req=140628818049344,sector=32,nsectors=8
virtio_blk_handle_read: ts=1436461565480193,req=140628818098608,sector=40,nsectors=8
virtio_blk_handle_read: ts=1436461565480207,req=140628818147872,sector=48,nsectors=8
virtio_blk_handle_read: ts=1436461565480221,req=140628818197136,sector=56,nsectors=8
virtio_blk_handle_read: ts=1436461565480271,req=140628818246400,sector=64,nsectors=8

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.  Start RHEL7.1 VM on RHEL7.1 host with qemu-kvm-rhev
2.  Trigger any IO workload on VM start reading from a disk in chunks. For ex: 4K sequential read. 
3.  Monitor systemtap traces for io_submit

Actual results:

AIO  processing only one request

Expected results:

AIO should batch IO reqeusts/
Additional info:

Comment 4 Pradeep Kumar Surisetty 2015-11-12 04:20:11 UTC
stefan has provided upstream fix for this. 

Commit id:  fc73548e444ae3239f6cef44a5200b5d2c3e85d1

    The raw-posix block driver implements Linux AIO batching so multiple
    requests can be submitted with a single io_submit(2) system call.
    Batching is currently only used by virtio-scsi and
    Enable batching for regular virtio-blk so the number of io_submit(2)
    system calls is reduced for workloads with queue depth > 1.

Comment 9 Stefan Hajnoczi 2016-01-15 12:37:58 UTC
Commit fc73548e444ae3239f6cef44a5200b5d2c3e85d1 is included in qemu-kvm-rhev-2.5.0-1.el7.  It was introduced as part of the QEMU 2.5 rebase.

Dropping RHEL 7.2.z flag.  Unless there is a specific justification for z-stream this improvement can become available in RHEL 7.3.

Comment 20 Miroslav Rezanina 2016-03-17 07:33:27 UTC
*** Bug 1285624 has been marked as a duplicate of this bug. ***

Comment 29 FuXiangChun 2016-09-18 06:51:31 UTC
Created attachment 1202103 [details]

Comment 33 errata-xmlrpc 2016-11-07 20:28:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.