1. Please describe the problem: When testing kernel 5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln there is big performance impact. This seems to affect all arches. 5.18.0-0.rc4.46cf2c613f4b10e.35.test.eln.x86_64 1. Measure time of some IO # time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.3157 s, 158 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.0184 s, 174 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.20987 s, 163 MB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 3.18907 s, 164 MB/s real 0m16.229s user 0m0.013s sys 0m15.653s ######## Compare with 5.18.0-0.rc3.27.eln117.x86_64 # time for _ in $(seq 4); do dd if=/dev/zero of=/tmp/test.img bs=1M count=500; done dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.416749 s, 1.3 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.306906 s, 1.7 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.271836 s, 1.9 GB/s dd if=/dev/zero of=/tmp/test.img bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 0.272663 s, 1.9 GB/s real 0m1.463s user 0m0.003s sys 0m1.430s
this is reproducible with kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117 https://koji.fedoraproject.org/koji/buildinfo?buildID=1957354
Hello I can confirm measured regression on most of our testcases. Regression is up to 90% in comparison with kernel-5.14.0-42.el9.x86_64 (RHEL-9.0.0-20220117.0). Results of comparison can be found here: http://cqe.perfqe.tpb.lab.eng.brq.redhat.com/reports/?area=fs&target_kernel=kernel-5.18.0-0.rc4.20220428git8f4dd16603ce834.36.eln117.x86_64&submit=Search Jan Jurča Kernel Performance QE
Thank you for confirming Bruno's findings, Jan! I think the performance drop is due to the DEBUG config options being set. I have checked the kernel config file for kernel-5.18.0-0.rc4.33.eln117 and DEBUG options are not set. Beaker jobs are scheduled, could you please check the performance of this kernel? I hope it will resolve the issue.