Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 2154428 [details] dmesg log 1. Please describe the problem: sched_ext BPF schedulers (scx_bpfland, scx_lavd, etc.) fail to load with "func_proto incompatible with vmlinux" error. The kernel's vmlinux BTF has malformed sched_ext kfunc prototypes — KF_IMPLICIT_ARGS kfuncs still carry the implicit 'struct bpf_prog_aux *' (aux) argument in their public BTF prototype. ----------- 2. What is the Version-Release number of the kernel: This affects both Fedora 44 (kernel 7.1.8-200.fc44) and Fedora 45 Rawhide (kernel 7.2.0-0.rc6.260807gf9a2394a2348.52.fc45), confirmed by testing both. ----------- 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : Last time I checked I was on kernel 6.19 and it worked then. I checked now after a while and I am on 7.1.8. The KF_IMPLICIT_ARGS kfunc issue was introduced in Linux 7.0. ----------- 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: When running on 7.1.8 (stable Fedora 44 kernel) or when using the Linux 7.2.0-0.rc6.260807gf9a2394a2348.52.fc45.x86_64 kernel from Koji while testing. Terminal command: sudo scx_bpfland -m auto Ouput: 17:25:44 [INFO] NUMA nodes: 1 17:25:44 [INFO] Disabling NUMA optimizations 17:25:44 [INFO] scx_bpfland 1.1.2 x86_64-unknown-linux-gnu SMT on 17:25:44 [INFO] scheduler options: scx_bpfland -m auto 17:25:44 [INFO] scheduler flags: 0x36 Error: the running kernel's BTF has malformed scx kfunc prototype(s): __scx_bpf_dsq_insert_vtime, __scx_bpf_select_cpu_and, scx_bpf_cid_override, scx_bpf_cid_to_cpu, scx_bpf_cidperf_set, scx_bpf_cpu_curr, scx_bpf_cpuperf_cap, scx_bpf_cpuperf_cur, scx_bpf_dispatch_cancel, scx_bpf_dispatch_nr_slots, scx_bpf_dsq_insert___v2, scx_bpf_dsq_move_to_local___v2, scx_bpf_dsq_nr_queued, scx_bpf_dsq_peek, scx_bpf_error_bstr, scx_bpf_exit_bstr, scx_bpf_get_idle_cpumask, scx_bpf_kick_cid, scx_bpf_pick_any_cpu_node, scx_bpf_pick_idle_cpu, scx_bpf_select_cpu_dfl, scx_bpf_test_and_clear_cpu_idle. These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype still carries the implicit 'struct bpf_prog_aux *' argument, which makes BPF programs fail to load with 'func_proto incompatible with vmlinux'. This happens when the kernel was built with pahole < 1.26. Fix: boot a kernel whose BTF was generated with pahole >= 1.26. Affected distros include Ubuntu 24.04 LTS. See kernel commit 9edd04c4189e ("docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs"). Caused by: 0: Failed to load BPF program 1: Invalid argument (os error 22) ----------- 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Yes, see above. ----------- 6. Are you running any modules that not shipped with directly Fedora's kernel?: No out-of-tree modules. ----------- 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Attached.
*** This bug has been marked as a duplicate of bug 2514913 ***