Originally reported in https://bodhi.fedoraproject.org/updates/FEDORA-2022-c4057cabb4: systemd reports during boot: systemd[1]: bpf-lsm: Failed to load BPF object: No such process The error ultimately comes from libbpf. It is easier to reproduce with a test program that loads the same code: $ sudo /usr/lib/systemd/tests/test-bpf-lsm Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy libbpf: failed to find valid kernel BTF libbpf: Error loading vmlinux BTF: -3 libbpf: failed to load object 'restrict_fs_bpf' libbpf: failed to load BPF skeleton 'restrict_fs_bpf': -3 bpf-lsm: Failed to load BPF object: No such process test-bpf-lsm: LSM BPF hooks are not supported, skipping tests. The problem apparently started appearing with kernel 6.0. That part of the in systemd hasn't changed in a while, so I don't think exact systemd version matters. Package versions: systemd-252~rc3 from git libbpf-0.8.0-2.fc37.x86_64 kernel-core-6.0.5-300.fc37.x86_64 How reproducible: Seems deterministic. Steps to Reproduce: Just install kernel-6.0, reboot, in a VM or real hardware. Additional info: $ sudo strace -e openat,read,access,close,writev -y /usr/lib/systemd/tests/test-bpf-lsm ... openat(AT_FDCWD</home/zbyszek/src/systemd-work>, "/lib64/libbpf.so.0", O_RDONLY|O_CLOEXEC) = 3</usr/lib64/libbpf.so.0.8.0> read(3</usr/lib64/libbpf.so.0.8.0>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 close(3</usr/lib64/libbpf.so.0.8.0>) = 0 openat(AT_FDCWD</home/zbyszek/src/systemd-work>, "/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3</usr/lib64/libelf-0.187.so> read(3</usr/lib64/libelf-0.187.so>, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 close(3</usr/lib64/libelf-0.187.so>) = 0 openat(AT_FDCWD</home/zbyszek/src/systemd-work>, "/sys/kernel/security/lsm", O_RDONLY|O_CLOEXEC) = 3</sys/kernel/security/lsm> read(3</sys/kernel/security/lsm>, "lockdown,capability,yama,bpf,lan"..., 4096) = 37 read(3</sys/kernel/security/lsm>, "", 4096) = 0 close(3</sys/kernel/security/lsm>) = 0 access("/proc/version_signature", R_OK) = -1 ENOENT (No such file or directory) close(4<anon_inode:bpf-prog>) = 0 access("/sys/kernel/btf/vmlinux", R_OK) = 0 openat(AT_FDCWD</home/zbyszek/src/systemd-work>, "/sys/kernel/btf/vmlinux", O_RDONLY) = 4</sys/kernel/btf/vmlinux> read(4</sys/kernel/btf/vmlinux>, "\237\353\1\0\30\0\0\0\0\0\0\0\30T1\0\30T1\0\323\27\"\0\1\0\0\0\0\0\0\1"..., 4096) = 4096 read(4</sys/kernel/btf/vmlinux>, "g_support\0max_data_retry_count\0s"..., 4096) = 3075 read(4</sys/kernel/btf/vmlinux>, "\237\353\1\0\30\0\0\0\0\0\0\0\30T1\0\30T1\0\323\27\"\0\1\0\0\0\0\0\0\1"..., 5464064) = 4096 ... read(4</sys/kernel/btf/vmlinux>, "_ATTR_BAND_PREF\0NL80211_BSS_SELE"..., 4096) = 4096 read(4</sys/kernel/btf/vmlinux>, "g_support\0max_data_retry_count\0s"..., 4096) = 3075 close(4</sys/kernel/btf/vmlinux>) = 0 access("/boot/vmlinux-6.0.5-300.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/6.0.5-300.fc37.x86_64/vmlinux-6.0.5-300.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/6.0.5-300.fc37.x86_64/build/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/modules/6.0.5-300.fc37.x86_64/kernel/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-6.0.5-300.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-6.0.5-300.fc37.x86_64.debug", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/lib/modules/6.0.5-300.fc37.x86_64/vmlinux", R_OK) = -1 ENOENT (No such file or directory) writev(2</dev/pts/0>, [{iov_base="libbpf: failed to find valid ker"..., iov_len=39}, {iov_base="\n", iov_len=1}], 2libbpf: failed to find valid kernel BTF ) = 40 ...
#2084955 is somewhat similar, but it's not the same bug, the error message is different. But I see that "systemd[1]: Failed to load BPF object: No such process" was also reported in https://bugzilla.redhat.com/show_bug.cgi?id=2084955#c22. That comment is from 2022-06-03, so it's an older kernel version.
this seems to be related to the enum64 support added in kernel v6.0, we need the libbpf 1.0 update, which will still take some time, because all dependent packages need to support that meanwhile I tried disabling enum64 in kernel BTF build and it seems to fix the systemd tests.. could you please test with this scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=93592781 will try to push it to fedora rawhide as temporary workaround thanks, jirka
No error with kernel from scratch bulid. LSM BPF program attached instead of error.
FYI got message from fedora maintainer that workaround will be in 6.0.x kernels for F35 - 37.
I am not seeing the error in my logs now on this FC37 box with the latest update - kernel-6.0.6-300.fc37.x86_64
Same here, with kernel-6.0.6-300.fc37.x86_64 the error is gone from dmesg logs
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
In Fedora 37, this error has appeared after upgrade to kernel-6.2.7-200.fc37.x86_64. It was not seen in kernel-6.0.12-300.fc37.x86_64 dmesg --level=err,warn | grep -i bpf daemon:err : 2023-03-23T11:22:39,583603+05:30 systemd[1]: bpf-lsm: Failed to load BPF object: No such process daemon:err : 2023-03-23T11:22:42,493464+05:30 systemd[1]: bpf-lsm: Failed to load BPF object: No such process Regards
(In reply to olsajiri from comment #4) > FYI got message from fedora maintainer that workaround will be in 6.0.x > kernels for F35 - 37. > this error has appeared after upgrade to kernel-6.2.7-200.fc37.x86_64 What is the plan here? F37 still has libbpf-0.8.0-2.fc37.
*** Bug 2175496 has been marked as a duplicate of this bug. ***
The problem is actually in F37. F38 has newer libbpf so the issue doesn't occur there.
I have scratch build if you want to test it: https://koji.fedoraproject.org/koji/taskinfo?taskID=100098872 I'll try to check if the bulk update is possible in fedora 37, there's about 10 packages that depend on libbpf that would need to get updated as well (at least there were in rawhide)
This message is a reminder that Fedora Linux 37 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '37'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 37 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 37 entered end-of-life (EOL) status on None. Fedora Linux 37 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.