Fedora Account System
Red Hat Associate
Red Hat Customer
1. Please describe the problem: 2. What is the Version-Release number of the kernel: 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: 6. Are you running any modules that not shipped with directly Fedora's kernel?: 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Reproducible: Always Steps to Reproduce: 1. Install scx-scheds (from COPR bieszczaders/kernel-cachyos-addons or any build using upstream scx) 2. Enable scx_loader.service / try to start scx_lavd (or any scx_* scheduler using the affected kfuncs) 3. Observe the "func_proto incompatible with vmlinux" error in journalctl -u scx_loader.service Actual Results: BPF program fails to load (Invalid argument / os error 22), /sys/kernel/sched_ext/state stays "disabled" Expected Results: sched_ext BPF scheduler loads successfully and /sys/kernel/sched_ext/state reports "enabled" Additional Information: uname -r: 7.1.9-200.fc44.x86_64 (also reproduced on 7.1.8-200.fc44) scx-scheds: 1.1.3-2.fc44 Kernel lockdown: none [integrity] confidentiality (Secure Boot enabled) — not related to this issue, as the failure is a BTF/BPF load error, not a permission (EPERM) error. The dwarves/pahole package currently shipped in Fedora repositories is already well above 1.26 (libdwarves1-1.30-3.fc44 in Rawhide), so this appears to be a kernel buildroot issue: the kernel RPM itself was likely compiled against an older pahole snapshot in the Koji buildroot.
Created attachment 2155266 [details] journalctl --no-hostname -k > ~/dmesg.txt
Adding the missing details from my original report, plus answers to the standard questions: 1. Please describe the problem: sched_ext BPF schedulers (e.g. scx_lavd from the scx-scheds package) fail to load with the following error: libbpf: extern (func ksym) 'scx_bpf_kick_cpu': func_proto [1026] incompatible with vmlinux [64055] libbpf: failed to load BPF skeleton 'bpf_bpf': -EINVAL Error: the running kernel's BTF has malformed scx kfunc prototype(s): scx_bpf_cpu_curr, scx_bpf_cpuperf_cur, scx_bpf_cpuperf_set, scx_bpf_dispatch_nr_slots, scx_bpf_dsq_move_to_local___v2, scx_bpf_dsq_peek, scx_bpf_dsq_reenq, scx_bpf_get_idle_cpumask_node, scx_bpf_kick_cpu, scx_bpf_locked_rq, scx_bpf_pick_any_cpu, scx_bpf_reenqueue_local, scx_bpf_task_cgroup. These kfuncs are KF_IMPLICIT_ARGS but their public BTF prototype still carries the implicit 'struct bpf_prog_aux *' argument, which makes BPF programs fail to load with 'func_proto incompatible with vmlinux'. This happens when the kernel was built with pahole < 1.26. This matches the upstream kernel documentation requirement (kernel.org commit 9edd04c4189e, "docs: Raise minimum pahole version to 1.26 for KF_IMPLICIT_ARGS kfuncs"), which states that since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26 or later. 2. What is the Version-Release number of the kernel: kernel-7.1.9-200.fc44.x86_64 (also reproduced on kernel-7.1.8-200.fc44.x86_64) 3. Did it work previously in Fedora? If so, what kernel version did the issue first appear? Not certain of the exact first affected version, but confirmed present on both 7.1.8-200.fc44 and 7.1.9-200.fc44. 4. Can you reproduce this issue? Yes, always. See Steps to Reproduce below. 5. Does this problem occur with the latest Rawhide kernel? Not tested yet — happy to test if needed, would appreciate guidance on whether the fix has already landed in Rawhide. 6. Are you running any modules that not shipped with directly Fedora's kernel? No out-of-tree kernel modules. scx-scheds itself (userspace BPF loader / schedulers) comes from COPR bieszczaders/kernel-cachyos-addons, but the kernel itself is the stock Fedora kernel package, not a custom build. 7. Kernel logs attached separately (see attachment). Additional note: the dwarves/pahole package currently shipped in Fedora repositories is already well above 1.26 (libdwarves1-1.30-3.fc44 in Rawhide), so this appears to be a kernel buildroot timing issue: the kernel RPM itself was likely compiled against an older pahole snapshot present in the Koji buildroot at build time, rather than a missing package on the user's system.
Additional test to rule out a scheduler-specific bug: I also tested scx_bpfland (a different scx scheduler binary) instead of scx_lavd, using `scxctl switch -s scx_bpfland`. It fails with the exact same error: Error: the running kernel's BTF has malformed scx kfunc prototype(s): scx_bpf_cpu_curr, scx_bpf_cpuperf_cur, scx_bpf_cpuperf_set, scx_bpf_dispatch_nr_slots, scx_bpf_dsq_move_to_local___v2, scx_bpf_dsq_peek, scx_bpf_dsq_reenq, scx_bpf_get_idle_cpumask_node, scx_bpf_kick_cpu, scx_bpf_locked_rq, scx_bpf_pick_any_cpu, scx_bpf_reenqueue_local, scx_bpf_task_cgroup. This confirms the issue is not specific to scx_lavd or a particular scheduler binary — it's a generic sched_ext/kernel BTF issue affecting any scx scheduler on this kernel build. === Additional timeline info: scx-scheds was first installed and successfully configured on 2026-07-03 (dnf history transaction 31), and it worked correctly at that time. Looking at dnf history, the kernel was upgraded from 6.19.10-300.fc44 to 7.1.3-200.fc44 on 2026-07-07 (transaction 33). This strongly suggests the regression was introduced somewhere in the Linux 7.x kernel series build for Fedora, consistent with the fact that the pahole >= 1.26 requirement for KF_IMPLICIT_ARGS kfuncs only applies "since Linux 7.0" per the upstream kernel documentation. sched_ext worked fine on 6.19.10-300.fc44 and has been broken on every 7.x kernel I've tested since (7.1.3 era onward through 7.1.8 and 7.1.9). I no longer have journal logs from that period to confirm the exact kernel active on boot at the time (journald retention has since rotated past early July), but the dnf transaction history clearly brackets the kernel version change around the same window scx-scheds was last known to work.
*** This bug has been marked as a duplicate of bug 2514913 ***