Bug 2271375

Summary: Performance tests fail
Product: [Fedora] Fedora Reporter: Stanislav Graf <sgraf>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 40CC: acaringi, adscvr, airlied, alciregi, bskeggs, gilbert.fernandes, hdegoede, hpa, itrymybest80, jarod, josef, kernel-maint, linville, masami256, mchehab, ptalbert, steved
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Stanislav Graf 2024-03-25 10:53:02 UTC
1. Please describe the problem:

Latest 6.8.1 kernel works great, no issues, except of performance test suite. I assume this is a bug within the suite.

$ sudo ./runtests.sh -t performance
Test suite called with performance
gmake[2]: Warning: File '../bin/aarch64-linux-gnu/lib_tcp.o' has modification time 307156 s in the future
lib_timing.c: In function ‘touch’:
lib_timing.c:1626:17: error: type defaults to ‘int’ in declaration of ‘psize’ [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/aarch64-linux-gnu/lib_timing.o] Error 1
make[1]: *** [Makefile:114: lmbench] Error 2
make: *** [Makefile:20: build] Error 2
gmake[2]: Warning: File '../bin/aarch64-linux-gnu/lib_tcp.o' has modification time 307156 s in the future
lib_timing.c: In function ‘touch’:
lib_timing.c:1626:17: error: type defaults to ‘int’ in declaration of ‘psize’ [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/aarch64-linux-gnu/lib_timing.o] Error 1
make[1]: *** [Makefile:114: lmbench] Error 2
make: *** [Makefile:23: results] Error 2
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
{"message":"Upload successful!"}
The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected


2. What is the Version-Release number of the kernel:
6.8.1-300.fc40.aarch64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
Don't know

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
Yes, 100% reproducibility with ./runtests.sh -t performance

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
yes, looks like the same error
6.9.0-0.rc0.20240322git8e938e398669.14.fc41.aarch64

6. Are you running any modules that not shipped with directly Fedora's kernel?:
No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
command generates empty file

Reproducible: Always

Comment 1 itrymybest80 2024-06-02 14:21:00 UTC
Test suite called with performance
lib_timing.c: I funktion ”touch”:
lib_timing.c:1626:17: fel: typen sätts skönsmässigt till ”int” i deklarationen av ”psize” [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/x86_64-linux-gnu/lib_timing.o] Fel 1
make[1]: *** [Makefile:114: lmbench] Fel 2
make: *** [Makefile:20: build] Fel 2
lib_timing.c: I funktion ”touch”:
lib_timing.c:1626:17: fel: typen sätts skönsmässigt till ”int” i deklarationen av ”psize” [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/x86_64-linux-gnu/lib_timing.o] Fel 1
make[1]: *** [Makefile:114: lmbench] Fel 2
make: *** [Makefile:23: results] Fel 2
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS

kernel-6.8.12-300.fc40.x86_64 [error]
kernel-6.9.3-200.fc40.x86_64 [error]
kernel-6.10.0-0.rc1.20240531git4a4be1ad3a6e.21.fc41.x86_64 [error]

Default performance test(s) pass, but the performance test(s) will always end prematurely.

Comment 2 Gilbert Fernandes 2024-06-18 18:42:36 UTC
I am sorry to report that I have been able to run that performance suite without error.
Fedora 40, up to date.
Also checked my test suite is up to date using git pull --rebase :

commit 6ca1b0d46fa3ece0a8b24bad49ef83da531576a0 (HEAD -> master, origin/master, origin/HEAD)
Author: Justin M. Forbes <jforbes>
Date:   Thu Mar 21 15:29:52 2024 -0500

    Update the anonymous commithook for the new kerneltest update
    
    Signed-off-by: Justin M. Forbes <jforbes>

I did run the test exactly like you did :

gf@aesir:~/kernel-test$ sudo ./runtests.sh -t performance
[sudo] password for gf: 
Test suite called with performance
Using config in CONFIG.aesir
Tue Jun 18 08:14:48 PM CEST 2024
Latency measurements
Tue Jun 18 08:16:39 PM CEST 2024
Calculating file system latency
Tue Jun 18 08:16:41 PM CEST 2024
Local networking
Tue Jun 18 08:20:23 PM CEST 2024
Bandwidth measurements
Tue Jun 18 08:38:28 PM CEST 2024
Calculating context switch overhead
Tue Jun 18 08:38:32 PM CEST 2024
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
fedora_submit.py -u <fasuser> [-p <password>] -l <logfile>
The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Vulnerable: Safe RET, no microcode
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

No error did occur.
I did saw the RAM used go up to almost 50 Gb (of the 64 Gb) and the CPUs did have to work a little with the fan to get until the end.

But no error.

Linux aesir 6.9.4-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jun 12 13:33:34 UTC 2024 x86_64 GNU/Linux

Comment 3 itrymybest80 2024-07-03 23:32:17 UTC
https://pagure.io/kernel-tests/issue/53 not sure if related or not, but it would be nice to complete your "performance test" again; apologies if it turns out to be off-topic.

Comment 4 Gilbert Fernandes 2024-07-04 08:20:19 UTC
To answer the "Did it work previously in Fedora?" part :

I did grab the changes to the git repository. And ran the standard then performance test again.
I do not see errors on the Fedora 41 + all updates available applied through dnf.

Linux aesir 6.9.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 27 18:11:45 UTC 2024 x86_64 GNU/Linux
NAME="Fedora Linux"
VERSION="40 (Workstation Edition)"

performance test :

gf@aesir:~/kernel-test$ sudo ./runtests.sh -t performance
[sudo] password for gf: 
Test suite called with performance
Using config in CONFIG.aesir
Thu Jul  4 09:49:56 AM CEST 2024
Latency measurements
Thu Jul  4 09:51:53 AM CEST 2024
Calculating file system latency
Thu Jul  4 09:51:55 AM CEST 2024
Local networking
Thu Jul  4 09:55:35 AM CEST 2024
Bandwidth measurements
Thu Jul  4 10:14:01 AM CEST 2024
Calculating context switch overhead
Thu Jul  4 10:14:05 AM CEST 2024
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
/home/gf/kernel-test/logs/kernel-test-1720079395.log.txt
Please visit https://id.fedoraproject.org/openidc/Authorization?scope=openid+email+profile+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Fgroups+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Fagreements+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Ffas-attributes&response_type=code&client_id=kerneltest-client&redirect_uri=[REDIRECT_URI]&response_mode=query to grant authorization
127.0.0.1 - - [04/Jul/2024 10:17:37] "GET /?code=[CODE] HTTP/1.1" 200 47
{"message":"Upload successful!"}

The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Vulnerable: Safe RET, no microcode
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

No error.
The OpenID part worked flawlessly too.

Comment 5 itrymybest80 2024-08-06 14:45:14 UTC
I'm able to complete the performance test with this patch: https://pagure.io/kernel-tests/pull-request/54

Comment 6 Aoife Moloney 2025-04-25 10:22:30 UTC
This message is a reminder that Fedora Linux 40 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 40 on 2025-05-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '40'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 40 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 7 Stanislav Graf 2025-04-25 12:47:00 UTC
I haven't seen this issue when I participated in kernel test week for 6.13 or 6.14. Performance tests are now passing for me.