Bug 2080223

Summary:	[ark-eln] IO performance degradation on 5.18.0-0.rc4
Product:	[Fedora] Fedora	Reporter:	Bruno Goncalves <bgoncalv>
Component:	kernel	Assignee:	Kernel Maintainer List <kernel-maint>
Status:	NEW ---	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	acaringi, adscvr, airlied, alciregi, bskeggs, hdegoede, hpa, jarodwilson, jglisse, jhladky, jjurca, jonathan, josef, kernel-maint, lgoncalv, linville, lmusil, masami256, mchehab, ptalbert, spetrovi, steved
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Bruno Goncalves 2022-04-29 09:46:59 UTC

1. Please describe the problem:
When testing kernel 5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln there is big performance impact. This seems to affect all arches.

5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln.x86_64

1. Measure time of some IO
# time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done


dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 3.3157 s, 158 MB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 3.0184 s, 174 MB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 3.20987 s, 163 MB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 3.18907 s, 164 MB/s

real	0m16.229s
user	0m0.013s
sys	0m15.653s

########

Compare with 5.18.0-0.rc3.27.eln117.x86_64

# time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.416749 s, 1.3 GB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.306906 s, 1.7 GB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.271836 s, 1.9 GB/s

dd if=/dev/zero of=/tmp/test.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.272663 s, 1.9 GB/s

real	0m1.463s
user	0m0.003s
sys	0m1.430s

Comment 2 Bruno Goncalves 2022-04-29 11:09:05 UTC

this is reproducible with kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117

https://koji.fedoraproject.org/koji/buildinfo?buildID=1957354

Comment 3 Jan Jurca 2022-05-02 09:38:48 UTC

Hello 
I can confirm measured regression on most of our testcases. Regression is up to 90% in comparison with kernel-5.14.0-42.el9.x86_64 (RHEL-9.0.0-20220117.0).

Results of comparison can be found here:

http://cqe.perfqe.tpb.lab.eng.brq.redhat.com/reports/?area=fs&target_kernel=kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117.x86_64&submit=Search

Jan Jurča

Kernel Performance QE

Comment 4 Jiri Hladky 2022-05-02 11:04:30 UTC

Thank you for confirming Bruno's findings, Jan!

I think the performance drop is due to the DEBUG config options being set. I have checked the kernel config file for kernel-5.18.0-0.rc4.33.eln117 and DEBUG options are not set. Beaker jobs are scheduled, could you please check the performance of this kernel? I hope it will resolve the issue.