Bug 1930286

Summary:	randread and randrw regression with virtio-blk multi-queue
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Stefan Hajnoczi <stefanha>
Component:	qemu-kvm	Assignee:	Stefano Garzarella <sgarzare>
qemu-kvm sub component:	virtio-blk,scsi	QA Contact:	Tingting Mao <timao>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	chayang, coli, djdumas, dshaks, jen, jinzhao, juzhang, kkiwi, ldoktor, mimehta, sgarzare, timao, virt-maint, yama
Version:	8.2	Keywords:	Regression, Triaged
Target Milestone:	rc
Target Release:	8.4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-16 07:51:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1827722
Bug Blocks:	1957194

Description Stefan Hajnoczi 2021-02-18 16:22:57 UTC

Description of problem:

virtio-blk multi-queue was enabled by default to improve scalability. randread and randrw on NVMe on an smp=4 x86 guest with iodepth=64,ioengine=libaio,bs=16kb and 16 threads regressed by 18% and 25%, respectively.

Benchmark details are available here:
http://kvm-perf.englab.nay.redhat.com/results/regression/multiqueue_rhel8.4.0/iothread-none-scheduler-native/raw.virtio_blk.*.x86_64.html

Version-Release number of selected component (if applicable):
qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a

How reproducible:


Steps to Reproduce:
See benchmark details above.

Actual results:
18% and 25% performance regression.

Expected results:
Comparable or better than single-queue.

Additional info:

Comment 13 Stefano Garzarella 2021-07-22 07:10:04 UTC

v2 posted: https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg05492.html

Patches merged upstream and released with QEMU v6.1.0-rc0:
  d7ddd0a161 linux-aio: limit the batch size using `aio-max-batch` parameter
  1793ad0247 iothread: add aio-max-batch parameter
  0445409d74 iothread: generalize iothread_set_param/iothread_get_param

Comment 17 Ademar Reis 2021-07-23 22:09:14 UTC

See also: Bug 1859048

Comment 20 Yanan Fu 2021-07-30 01:38:31 UTC

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 24 Tingting Mao 2021-08-02 12:30:26 UTC

Verified this bug as below, there is no regression now.


Tested with:
qemu-kvm-6.0.0-26.module+el8.5.0+12044+525f0ebc
kernel-modules-4.18.0-321.el8.x86_64


Results:
http://kvm-perf.englab.nay.redhat.com/results/regression/nvme8.5.0-qemu6.0.0-26-12044_kernel4.18.0-321_with_numqueues/raw.virtio_blk.*.x86_64.html

Comment 26 errata-xmlrpc 2021-11-16 07:51:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684