Fedora Account System
Red Hat Associate
Red Hat Customer
All sched_ext BPF schedulers fail to load on Fedora 44 with kernel 7.1.4-200.fc44.x86_64. The kernel's vmlinux BTF retains the implicit 'struct bpf_prog_aux *' argument on KF_IMPLICIT_ARGS kfuncs, so BPF programs declaring the 2-parameter form are rejected by libbpf with "func_proto incompatible with vmlinux". ENVIRONMENT Kernel: 7.1.4-200.fc44.x86_64 Config: CONFIG_PAHOLE_VERSION=130 (pahole 1.30) CONFIG_SCHED_CLASS_EXT=y Hardware: AMD Ryzen AI Max+ 395 (Strix Halo), 16C/32T EVIDENCE The malformed prototype is visible directly in the kernel's own BTF, independent of any scheduler package: $ sudo bpftool btf dump file /sys/kernel/btf/vmlinux format raw \ | grep -A3 '^\[63980\]' [63980] FUNC_PROTO '(anon)' ret_type_id=0 vlen=3 'cpu' type_id=138367 'flags' type_id=152537 'aux' type_id=7932 Type 63980 is the prototype for scx_bpf_kick_cpu. Upstream declares: void scx_bpf_kick_cpu(s32 cpu, u64 flags) i.e. 2 parameters. The kernel BTF has vlen=3, retaining 'aux'. AFFECTED KFUNCS (13) scx_bpf_cpu_curr, scx_bpf_cpuperf_cur, scx_bpf_cpuperf_set, scx_bpf_dispatch_nr_slots, scx_bpf_dsq_move_to_local___v2, scx_bpf_dsq_peek, scx_bpf_dsq_reenq, scx_bpf_get_idle_cpumask_node, scx_bpf_kick_cpu, scx_bpf_locked_rq, scx_bpf_pick_any_cpu, scx_bpf_reenqueue_local, scx_bpf_task_cgroup STEPS TO REPRODUCE Minimal (no third-party packages required): sudo dnf install bpftool sudo bpftool btf dump file /sys/kernel/btf/vmlinux format raw \ | grep -A3 '^\[63980\]' -> shows vlen=3 with an 'aux' parameter Full reproduction with a scheduler: 1. sudo dnf copr enable bieszczaders/kernel-cachyos-addons 2. sudo dnf install scx-scheds 3. sudo scx_lavd --autopilot ACTUAL RESULT libbpf: extern (func ksym) 'scx_bpf_kick_cpu': func_proto [1079] incompatible with vmlinux [63980] libbpf: failed to load BPF skeleton 'bpf_bpf': -EINVAL Error: Failed to load BPF program / Invalid argument (os error 22) $ cat /sys/kernel/sched_ext/state disabled $ cat /sys/kernel/sched_ext/root/ops cat: No such file or directory EXPECTED RESULT Scheduler attaches; /sys/kernel/sched_ext/state reads "enabled" and /sys/kernel/sched_ext/root/ops reads "lavd". NOTES - Reproduced with both packaged scx-scheds 1.1.2 and a fresh build of current upstream sched-ext/scx git main. Not a packaging issue. - The scx error message attributes this to pahole < 1.26, but this kernel reports CONFIG_PAHOLE_VERSION=130. The stated cause does not apply here, suggesting a different failure in the BTF post-processing (resolve_btfids / btf2btf implicit-arg stripping) during the kernel build. - Affects all sched_ext schedulers, not only scx_lavd, since the failure is in the shared kfunc binding layer. Reproducible: Always
Created attachment 2149336 [details] Kernel config: CONFIG_PAHOLE_VERSION=130, CONFIG_SCHED_CLASS_EXT=y, BTF options
Created attachment 2149337 [details] scx_lavd --autopilot full console output showing BPF load failure
Created attachment 2149338 [details] vmlinux BTF dump of scx_bpf_* kfunc prototypes showing retained 'aux' parameter
Does this work with the updated libbpf in rawhide? https://koji.fedoraproject.org/koji/buildinfo?buildID=2965412
tried it, doesn't fix it. installed libbpf/libbpf-devel/libbpf-static 1.7.0-1.fc45 from that build and rebuilt scx_lavd from current git main. same error: libbpf: extern (func ksym) 'scx_bpf_kick_cpu': func_proto [1079] incompatible with vmlinux [63980] libbpf: failed to load BPF skeleton 'bpf_bpf': -EINVAL scx statically links libbpf through libbpf-sys, and the build log says "Compiling libbpf-sys v1.7.0+v1.7.0", so git main was already on 1.7.0 before i installed anything. ldd on the binary shows no libbpf at all. so 1.6.3 (packaged scx-scheds 1.1.2) and 1.7.0 (git main) both fail. kernel BTF still shows 3 params: [63980] FUNC_PROTO '(anon)' ret_type_id=0 vlen=3 'cpu' type_id=138367 'flags' type_id=152537 'aux' type_id=7932 upstream is void scx_bpf_kick_cpu(s32 cpu, u64 flags), so 2 params. 13 kfuncs affected.
still broken on 7.1.5-200.fc44, CONFIG_PAHOLE_VERSION=130. this has now survived 7.1.3, 7.1.4, and 7.1.5 - every kernel build reproduces it. the affected kfunc list changed between 7.1.4 and 7.1.5 (now scx_bpf_create_dsq, scx_bpf_cpu_node, scx_bpf_task_set_dsq_vtime, etc.) and the type id moved from 63980 to 63984, so the BTF is being regenerated each build - the implicit-arg stripping just isn't being applied. This seems to be persistent at least across the last few kernel updates. pahole is 1.30, four versions past the 1.26 the error message blames. [63984] FUNC_PROTO '(anon)' ret_type_id=0 vlen=3 ...still carries the implicit 'aux' param
I am guessing it has nothing to do with pahole, and is instead a side effect of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ba0a42423335f76d3e0513df42416c69dc6b742
verified it's not in the f44 kernels - three ways. 1. no cid symbols in the running kernel: $ sudo grep -E 'scx_bpf_(kick_cid|nr_cids|nr_online_cids|this_cid)' /proc/kallsyms (empty) 2. rpm changelog for kernel-core-7.1.5-200.fc44 shows upstream stable rebases plus small named cherry-picks (kvm, drm/amd, net) - no sched_ext patches carried at all. 3. my 7.1.4 BTF dump from earlier also has none of the cid kfuncs - the name-sorted FUNC entries go straight from get_possible_cpumask to kick_cpu with no kick_cid between them. timing fits too: the commit is dated apr 29, three days after 7.1-rc1 closed the merge window, and it's purely additive - so it's 7.2 material (7.2-rc1 was jun 28). if you were looking at a rawhide kernel you'd see it there, but not in 7.1.x. so it can't be the direct cause here. worth noting though that the commit's unchanged context lines show scx_bpf_kick_cpu already had the explicit aux param in the source before it - so whatever introduced the explicit-aux + KF_IMPLICIT_ARGS pattern is earlier, and that's what 7.1.x has. the aux is in my vmlinux BTF (vlen=3), pahole is 1.30, libbpf 1.7.0 doesn't tolerate it - so wherever the stripping or matching is supposed to happen, it isn't, on stock f44. still reproduces on 7.1.5-200, and the failing kfunc list shifted between 7.1.4 and 7.1.5 with the same scx binary - fedora carries no sched_ext patches, so that's coming through upstream stable itself.
still reproduces on 7.1.5-200.fc44 as of aug 10. two updates - one on the commit theory, one big one. re the linked commit: i don't think it's in the f44 kernels at all. no cid symbols in the running kernel: $ sudo grep -E 'scx_bpf_(kick_cid|nr_cids|nr_online_cids|this_cid)' /proc/kallsyms (empty) rpm changelog for kernel-core-7.1.5-200 shows upstream stable rebases plus small named cherry-picks (kvm, drm/amd, net) - no sched_ext patches carried. and the BTF FUNC entries are name-sorted with none of the cid kfuncs where they'd sit. it's dated apr 29, after 7.1-rc1 closed the merge window, and it's purely additive - looks like 7.2 material. so it can't be the direct cause here. the big update: i got past the userspace error and got the kernel verifier itself to identify the missing piece. i patched the loader (vendored libbpf in an scx build) to tolerate a kernel FUNC_PROTO with exactly one extra trailing 'struct bpf_prog_aux *' param. that got ksym resolution past libbpf for the first time, and the kernel then failed with: cannot find function scx_bpf_task_set_dsq_vtime_impl in BTF kernel function btf_id 139514 does not have a valid func_proto so the KF_IMPLICIT_ARGS design wants two BTF entries per kfunc: the base name with aux stripped (for programs to match) and a <name>_impl entry keeping the full proto (the verifier rewrites calls to it). on this kernel that generation is PARTIAL - 27 *_impl entries exist, but not for all kfuncs that need them. and the correlation is exact: the set of failing kfuncs and the set missing a *_impl twin are the same set. every kfunc that stopped failing between 7.1.4 and 7.1.5 (kick_cpu, cpuperf_cur, dispatch_nr_slots, dsq_peek, dsq_move_to_local, task_cgroup) has its _impl entry in 7.1.5's BTF. every currently-failing one lacks it. and it's not a tidy category - get_idle_cpumask_impl exists but get_idle_cpumask_node fails, get_idle_smtmask_node_impl exists but plain get_idle_smtmask fails, cpuperf_cap/cur have twins but cpuperf_set doesn't. it even splits within one kfunc family: dsq_insert and dsq_move_to_local have both base and ___v2 _impl twins, while reenqueue_local has only ___v2_impl and its unprocessed base entry is exactly the one failing. [139514] FUNC 'scx_bpf_task_set_dsq_vtime' type_id=64015 <- no _impl [139503] FUNC 'scx_bpf_reenqueue_local___v2_impl' <- has one 139514 is exactly the btf_id the verifier named. so one per-kfunc BTF post-processing step is partially succeeding, and which kfuncs it covers changed between two stable builds of the same series. CONFIG_PAHOLE_VERSION=130, 272 DECL_TAG 'bpf_kfunc' tags present. full _impl list on 7.1.5-200.fc44: scx_bpf_cpuperf_cap_impl scx_bpf_cpuperf_cur_impl scx_bpf_cpu_rq_impl scx_bpf_destroy_dsq_impl scx_bpf_dispatch_cancel_impl scx_bpf_dispatch_nr_slots_impl scx_bpf_dsq_insert_impl scx_bpf_dsq_insert___v2_impl scx_bpf_dsq_insert_vtime_impl scx_bpf_dsq_move_to_local_impl scx_bpf_dsq_move_to_local___v2_impl scx_bpf_dsq_nr_queued_impl scx_bpf_dsq_peek_impl scx_bpf_error_bstr_impl scx_bpf_exit_bstr_impl scx_bpf_get_idle_cpumask_impl scx_bpf_get_idle_smtmask_node_impl scx_bpf_kick_cpu_impl scx_bpf_pick_idle_cpu_impl scx_bpf_pick_idle_cpu_node_impl scx_bpf_reenqueue_local___v2_impl scx_bpf_select_cpu_and_impl scx_bpf_select_cpu_dfl_impl scx_bpf_sub_dispatch_impl scx_bpf_task_cgroup_impl scx_bpf_task_set_slice_impl scx_bpf_test_and_clear_cpu_idle_impl
*** This bug has been marked as a duplicate of bug 2514913 ***