Bug 2271375

Summary: Performance tests fail
Product: [Fedora] Fedora Reporter: Stanislav Graf <sgraf>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 40CC: acaringi, adscvr, airlied, alciregi, bskeggs, gilbert.fernandes, hdegoede, hpa, itrymybest80, jarod, josef, kernel-maint, linville, masami256, mchehab, ptalbert, steved
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Stanislav Graf 2024-03-25 10:53:02 UTC
1. Please describe the problem:

Latest 6.8.1 kernel works great, no issues, except of performance test suite. I assume this is a bug within the suite.

$ sudo ./runtests.sh -t performance
Test suite called with performance
gmake[2]: Warning: File '../bin/aarch64-linux-gnu/lib_tcp.o' has modification time 307156 s in the future
lib_timing.c: In function ‘touch’:
lib_timing.c:1626:17: error: type defaults to ‘int’ in declaration of ‘psize’ [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/aarch64-linux-gnu/lib_timing.o] Error 1
make[1]: *** [Makefile:114: lmbench] Error 2
make: *** [Makefile:20: build] Error 2
gmake[2]: Warning: File '../bin/aarch64-linux-gnu/lib_tcp.o' has modification time 307156 s in the future
lib_timing.c: In function ‘touch’:
lib_timing.c:1626:17: error: type defaults to ‘int’ in declaration of ‘psize’ [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/aarch64-linux-gnu/lib_timing.o] Error 1
make[1]: *** [Makefile:114: lmbench] Error 2
make: *** [Makefile:23: results] Error 2
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
{"message":"Upload successful!"}
The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected


2. What is the Version-Release number of the kernel:
6.8.1-300.fc40.aarch64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
Don't know

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
Yes, 100% reproducibility with ./runtests.sh -t performance

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
yes, looks like the same error
6.9.0-0.rc0.20240322git8e938e398669.14.fc41.aarch64

6. Are you running any modules that not shipped with directly Fedora's kernel?:
No

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.
command generates empty file

Reproducible: Always

Comment 1 itrymybest80 2024-06-02 14:21:00 UTC
Test suite called with performance
lib_timing.c: I funktion ”touch”:
lib_timing.c:1626:17: fel: typen sätts skönsmässigt till ”int” i deklarationen av ”psize” [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/x86_64-linux-gnu/lib_timing.o] Fel 1
make[1]: *** [Makefile:114: lmbench] Fel 2
make: *** [Makefile:20: build] Fel 2
lib_timing.c: I funktion ”touch”:
lib_timing.c:1626:17: fel: typen sätts skönsmässigt till ”int” i deklarationen av ”psize” [-Wimplicit-int]
 1626 |         static  psize;
      |                 ^~~~~
gmake[2]: *** [Makefile:240: ../bin/x86_64-linux-gnu/lib_timing.o] Fel 1
make[1]: *** [Makefile:114: lmbench] Fel 2
make: *** [Makefile:23: results] Fel 2
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS

kernel-6.8.12-300.fc40.x86_64 [error]
kernel-6.9.3-200.fc40.x86_64 [error]
kernel-6.10.0-0.rc1.20240531git4a4be1ad3a6e.21.fc41.x86_64 [error]

Default performance test(s) pass, but the performance test(s) will always end prematurely.

Comment 2 Gilbert Fernandes 2024-06-18 18:42:36 UTC
I am sorry to report that I have been able to run that performance suite without error.
Fedora 40, up to date.
Also checked my test suite is up to date using git pull --rebase :

commit 6ca1b0d46fa3ece0a8b24bad49ef83da531576a0 (HEAD -> master, origin/master, origin/HEAD)
Author: Justin M. Forbes <jforbes>
Date:   Thu Mar 21 15:29:52 2024 -0500

    Update the anonymous commithook for the new kerneltest update
    
    Signed-off-by: Justin M. Forbes <jforbes>

I did run the test exactly like you did :

gf@aesir:~/kernel-test$ sudo ./runtests.sh -t performance
[sudo] password for gf: 
Test suite called with performance
Using config in CONFIG.aesir
Tue Jun 18 08:14:48 PM CEST 2024
Latency measurements
Tue Jun 18 08:16:39 PM CEST 2024
Calculating file system latency
Tue Jun 18 08:16:41 PM CEST 2024
Local networking
Tue Jun 18 08:20:23 PM CEST 2024
Bandwidth measurements
Tue Jun 18 08:38:28 PM CEST 2024
Calculating context switch overhead
Tue Jun 18 08:38:32 PM CEST 2024
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
fedora_submit.py -u <fasuser> [-p <password>] -l <logfile>
The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Vulnerable: Safe RET, no microcode
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

No error did occur.
I did saw the RAM used go up to almost 50 Gb (of the 64 Gb) and the CPUs did have to work a little with the fan to get until the end.

But no error.

Linux aesir 6.9.4-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jun 12 13:33:34 UTC 2024 x86_64 GNU/Linux

Comment 3 itrymybest80 2024-07-03 23:32:17 UTC
https://pagure.io/kernel-tests/issue/53 not sure if related or not, but it would be nice to complete your "performance test" again; apologies if it turns out to be off-topic.

Comment 4 Gilbert Fernandes 2024-07-04 08:20:19 UTC
To answer the "Did it work previously in Fedora?" part :

I did grab the changes to the git repository. And ran the standard then performance test again.
I do not see errors on the Fedora 41 + all updates available applied through dnf.

Linux aesir 6.9.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 27 18:11:45 UTC 2024 x86_64 GNU/Linux
NAME="Fedora Linux"
VERSION="40 (Workstation Edition)"

performance test :

gf@aesir:~/kernel-test$ sudo ./runtests.sh -t performance
[sudo] password for gf: 
Test suite called with performance
Using config in CONFIG.aesir
Thu Jul  4 09:49:56 AM CEST 2024
Latency measurements
Thu Jul  4 09:51:53 AM CEST 2024
Calculating file system latency
Thu Jul  4 09:51:55 AM CEST 2024
Local networking
Thu Jul  4 09:55:35 AM CEST 2024
Bandwidth measurements
Thu Jul  4 10:14:01 AM CEST 2024
Calculating context switch overhead
Thu Jul  4 10:14:05 AM CEST 2024
./performance/lmbench3                                           PASS    

Test suite complete                                              PASS    

Your log file is being submitted...
/home/gf/kernel-test/logs/kernel-test-1720079395.log.txt
Please visit https://id.fedoraproject.org/openidc/Authorization?scope=openid+email+profile+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Fgroups+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Fagreements+https%3A%2F%2Fid.fedoraproject.org%2Fscope%2Ffas-attributes&response_type=code&client_id=kerneltest-client&redirect_uri=[REDIRECT_URI]&response_mode=query to grant authorization
127.0.0.1 - - [04/Jul/2024 10:17:37] "GET /?code=[CODE] HTTP/1.1" 200 47
{"message":"Upload successful!"}

The following information is not submitted with your log;
it is for informational purposes only.
Vulnerability status:
/sys/devices/system/cpu/vulnerabilities/gather_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/mmio_stale_data:Not affected
/sys/devices/system/cpu/vulnerabilities/reg_file_data_sampling:Not affected
/sys/devices/system/cpu/vulnerabilities/retbleed:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_rstack_overflow:Vulnerable: Safe RET, no microcode
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

No error.
The OpenID part worked flawlessly too.

Comment 5 itrymybest80 2024-08-06 14:45:14 UTC
I'm able to complete the performance test with this patch: https://pagure.io/kernel-tests/pull-request/54