Fedora Account System
Red Hat Associate
Red Hat Customer
1. Please describe the problem: When testing kernel 5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln there is big performance impact. This seems to affect all arches. 5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln.x86_64 1. Measure time of some IO # time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.3157 s, 158 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.0184 s, 174 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.20987 s, 163 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.18907 s, 164 MB/s real 0m16.229s user 0m0.013s sys 0m15.653s ######## Compare with 5.18.0-0.rc3.27.eln117.x86_64 # time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.416749 s, 1.3 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.306906 s, 1.7 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.271836 s, 1.9 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.272663 s, 1.9 GB/s real 0m1.463s user 0m0.003s sys 0m1.430s
this is reproducible with kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117 https://koji.fedoraproject.org/koji/buildinfo?buildID=1957354
Hello I can confirm measured regression on most of our testcases. Regression is up to 90% in comparison with kernel-5.14.0-42.el9.x86_64 (RHEL-9.0.0-20220117.0). Results of comparison can be found here: http://cqe.perfqe.tpb.lab.eng.brq.redhat.com/reports/?area=fs&target_kernel=kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117.x86_64&submit=Search Jan Jurča Kernel Performance QE
Thank you for confirming Bruno's findings, Jan! I think the performance drop is due to the DEBUG config options being set. I have checked the kernel config file for kernel-5.18.0-0.rc4.33.eln117 and DEBUG options are not set. Beaker jobs are scheduled, could you please check the performance of this kernel? I hope it will resolve the issue.
This report has not received an update in over 90 days. If this issue is still relevant to you, please do not hesitate to update this report with any relevant details on recent investigative steps or changes and we will absolutely work to understand how to best move forward. Otherwise, this bug report will need to be closed in 30 days. No worries, however; should you need still assistance after the report is closed, please do not hesitate to create a new bug report referencing this report and with any new details on the matter. NOTE: This is an automated mass update.