Bug 2116741

Summary:	Performance Degraded ~30% for luks+blk with write + 64k block size + 8 Iodepth
Product:	Red Hat Enterprise Linux 9	Reporter:	Tingting Mao <timao>
Component:	qemu-kvm	Assignee:	Stefan Hajnoczi <stefanha>
qemu-kvm sub component:	Storage	QA Contact:	Tingting Mao <timao>
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	high
Priority:	high	CC:	chayang, coli, jinzhao, juzhang, kkiwi, kwolf, stefanha, virt-maint
Version:	9.1	Keywords:	Regression, Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-11-02 09:29:20 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tingting Mao 2022-08-09 09:38:55 UTC

Description of problem:
The performance in latest rhel9.1 degraded ~30% for luks+blk with write + 64k block size + 8 Iodepth compared to rhel9.0.

While for raw or qcow2, there is no such degradation.

For details, please refer to:
http://kvm-perf.englab.nay.redhat.com/results/regression/xfs9.1.0_qemu7.0.0-9_kernel5.14.0-130/02/luks.virtio_blk.*.x86_64.html 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

Comment 3 Tingting Mao 2022-09-01 02:46:04 UTC

Hi Stefan,

After bisection test, the degradation is invovled firstly in qemu-kvm-core-7.0.0-1.el9. Could you please help to check in the code side? The details refer to below please.

http://kvm-perf.englab.nay.redhat.com/results/regression/xfs9.1.0_qemu7.0.0-9_kernel5.14.0-130/01/luks.virtio_blk.*.x86_64.html 

Thanks.

Comment 5 Stefan Hajnoczi 2022-10-13 17:49:54 UTC

I have started looking at this. There are lots of code changes between qemu-kvm-6.2.0-9.el9 and qemu-kvm-7.0.0-1.el9 but nothing immediately obvious for the crypto (luks) driver.

I'll try to reproduce this issue locally so that I can profile it and experiment with code changes.

Comment 6 Stefan Hajnoczi 2022-10-20 20:09:06 UTC

I attempted to reproduce the issue but don't measure a significant difference between qemu-kvm-6.2.0-9.el9 and qemu-kvm-7.0.0-1.el9 in the write bs=64k iodepth=8 numjobs=16 benchmark.

Maybe the bottleneck only occurs in certain environments.

Are you able to provide access to a host where the performance differences between qemu-kvm-6.2.0-9.el9 and qemu-kvm-7.0.0-1.el9 is visible? Thanks!

Comment 7 Tingting Mao 2022-11-02 09:29:20 UTC

I have also tried 2 machines to reproduce this bug(one of them is the right one which is used while reporting the bug), but I can not reproduce it, either.

So I will close this bug now. Thanks Stefan.

http://kvm-perf.englab.nay.redhat.com/results/regression/xfs9.1.0_qemu7.0.0-9_kernel5.14.0-130/newnew/luks.virtio_blk.*.x86_64.html