Bug 2084955 - libbpf failed to load object
Summary: libbpf failed to load object
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 36
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-12 17:19 UTC by Ralf Corsepius
Modified: 2022-06-06 02:11 UTC (History)
19 users (show)

Fixed In Version: systemd-250.6-1.fc36 systemd-250.7-1.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-06 02:11:19 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github systemd systemd pull 23596 0 None Draft Silence messages from libbpf 2022-06-02 06:00:09 UTC

Description Ralf Corsepius 2022-05-12 17:19:44 UTC
Description of problem:

Upon booting up f36 systems I am finding these messages:

May 12 17:30:58 beck systemd[1360]: Failed to load BPF object: Operation not permitted
May 12 17:30:58 beck systemd[1360]: libbpf: failed to load BPF skeleton 'restrict_ifaces_bpf': -1
May 12 17:30:58 beck systemd[1360]: libbpf: failed to load object 'restrict_ifaces_bpf'
May 12 17:30:58 beck systemd[1360]: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONF>
May 12 17:30:58 beck systemd[1360]: libbpf: failed to load BPF skeleton 'socket_bind_bpf': -1
May 12 17:30:58 beck systemd[1360]: libbpf: failed to load object 'socket_bind_bpf'
May 12 17:30:58 beck systemd[1360]: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONF>
Ma

Version-Release number of selected component (if applicable):
libbpf-0.5.0-2.fc36.x86_64
kernel-5.17.6-300.fc36.x86_64
systemd-250.3-8.fc36.x86_64

How reproducible:
Always. So far I am observing this issue on all systems I have upgraded from f35 to f36.

Steps to Reproduce:
1. reboot
2. Check 
journalctl -r -b

Comment 1 Alexey Rochev 2022-05-13 13:19:51 UTC
I have the same error messages in log after upgrading.

Comment 2 olsajiri 2022-05-13 14:16:00 UTC
AFAICS similar bug should be already fixed in 0.5.0 ... but there's already 0.7.0,
which is not yet in fedora 36, I'll push the update

Comment 3 olsajiri 2022-05-15 18:17:00 UTC
could you please try with following build:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1966501

thanks,
jirka

Comment 4 Ralf Corsepius 2022-05-16 07:44:54 UTC
Tested on 2 fc36 systems.

libbpf-0.7.0-3.fc36.x86_64.rpm seems to silence the
"libbpf: failed to load BPF skeleton" messages.

However, one error message remains (Also present with libbpf-0.5.0-2.fc36.x86_64):
"Failed to open libbpf, LSM BPF is not supported: Operation not supported"

Comment 5 olsajiri 2022-05-16 08:33:53 UTC
(In reply to Ralf Corsepius from comment #4)
> Tested on 2 fc36 systems.
> 
> libbpf-0.7.0-3.fc36.x86_64.rpm seems to silence the
> "libbpf: failed to load BPF skeleton" messages.
> 
> However, one error message remains (Also present with
> libbpf-0.5.0-2.fc36.x86_64):
> "Failed to open libbpf, LSM BPF is not supported: Operation not supported"

that message does not seem to come from libbpf,
can you describe the workflow you do in more details?

thanks

Comment 6 Ralf Corsepius 2022-05-16 14:19:03 UTC
(In reply to olsajiri from comment #5)
> that message does not seem to come from libbpf,
> can you describe the workflow you do in more details?

It's quite simple 
1. boot
2. log-in as root
3. Check the output of "journalctl -r -b"

AFAICT, the message above originates from systemd itself.

A somewhat more verbose log (Produced by: journalctl -r -b | grep -i bpf | grep -v audit)
...
May 16 15:58:40 barnaby systemd[1291]: Failed to create BPF map: Operation not permitted
May 16 15:51:15 barnaby systemd[1]: LSM BPF program attached
May 16 15:51:15 barnaby systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 15:51:13 barnaby systemd[1]: Failed to open libbpf, LSM BPF is not supported: Operation not supported
May 16 15:51:13 barnaby systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 15:51:13 barnaby kernel: LSM support for eBPF active
...

Note: The "Failed to open libbpf, LSM BPF is not supported"-message stems from a process with pid=1 (== systemd).
No idea, why systemd is thinking "LSM BPF is not supported" on this machine. 

On a another machine with a very similar processor the log looks like this:
... 
May 16 12:46:18 troy systemd[1]: LSM BPF program attached
May 16 12:46:18 troy systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 14:46:11 troy systemd[1]: LSM BPF program attached
May 16 14:46:11 troy systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 14:46:11 troy kernel: LSM support for eBPF active
...

Note: No "Failed to open libbpf" warning, here. No idea, why systemd is thinking "LSM BPF" is supported", on this machine.

Comment 7 Stan King 2022-05-16 23:47:27 UTC
I only see this on one of the four systems that I've upgraded to Fedora 36.  It's not the oldest processor, either.

If someone could point me to instructions for how to test with the Koji build referenced in comment 3, I'd be willing to try that.

Comment 8 olsajiri 2022-05-17 07:10:21 UTC
(In reply to Ralf Corsepius from comment #6)
> (In reply to olsajiri from comment #5)
> > that message does not seem to come from libbpf,
> > can you describe the workflow you do in more details?
> 
> It's quite simple 
> 1. boot
> 2. log-in as root
> 3. Check the output of "journalctl -r -b"
> 
> AFAICT, the message above originates from systemd itself.
> 
> A somewhat more verbose log (Produced by: journalctl -r -b | grep -i bpf |
> grep -v audit)
> ...
> May 16 15:58:40 barnaby systemd[1291]: Failed to create BPF map: Operation
> not permitted
> May 16 15:51:15 barnaby systemd[1]: LSM BPF program attached
> May 16 15:51:15 barnaby systemd[1]: systemd v250.3-8.fc36 running in system
> mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
> +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD
> +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ
> +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT
> default-hierarchy=unified)
> May 16 15:51:13 barnaby systemd[1]: Failed to open libbpf, LSM BPF is not
> supported: Operation not supported
> May 16 15:51:13 barnaby systemd[1]: systemd v250.3-8.fc36 running in system
> mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
> +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD
> +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ
> +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT
> default-hierarchy=unified)
> May 16 15:51:13 barnaby kernel: LSM support for eBPF active
> ...
> 
> Note: The "Failed to open libbpf, LSM BPF is not supported"-message stems
> from a process with pid=1 (== systemd).
> No idea, why systemd is thinking "LSM BPF is not supported" on this machine. 
> 
> On a another machine with a very similar processor the log looks like this:
> ... 
> May 16 12:46:18 troy systemd[1]: LSM BPF program attached
> May 16 12:46:18 troy systemd[1]: systemd v250.3-8.fc36 running in system
> mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
> +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD
> +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ
> +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT
> default-hierarchy=unified)
> May 16 14:46:11 troy systemd[1]: LSM BPF program attached
> May 16 14:46:11 troy systemd[1]: systemd v250.3-8.fc36 running in system
> mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS
> +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD
> +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ
> +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT
> default-hierarchy=unified)
> May 16 14:46:11 troy kernel: LSM support for eBPF active
> ...
> 
> Note: No "Failed to open libbpf" warning, here. No idea, why systemd is
> thinking "LSM BPF" is supported", on this machine.

I  see this one also on latest rawhide, not sure who to cc from
systemd side.. trying from git log, zbyszek. any idea? ;-)

thanks,
jirka

Comment 9 olsajiri 2022-05-17 07:11:51 UTC
(In reply to Stan King from comment #7)
> I only see this on one of the four systems that I've upgraded to Fedora 36. 
> It's not the oldest processor, either.
> 
> If someone could point me to instructions for how to test with the Koji
> build referenced in comment 3, I'd be willing to try that.

install new libbpf from the link in Comment 3, reboot and check journal
as described in Comment 6 for that libbpf related error

jirka

Comment 10 Zbigniew Jędrzejewski-Szmek 2022-05-17 08:20:37 UTC
Oh man, I wrote such a beautiful message with links to sources on github considering all the
possible reasons, but it turns out it was a simple logic error in systemd cleanup logic ;]
tl;dr: the message is harmless.

This is a bit confusing because it's in reverse order:

May 16 15:58:40 barnaby systemd[1291]: Failed to create BPF map: Operation not permitted
  ↑ this is the real error.

May 16 15:51:15 barnaby systemd[1]: LSM BPF program attached
  ↑ OK, on the second try, we loaded libbpf and are able to attach a program.

May 16 15:51:15 barnaby systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 15:51:13 barnaby systemd[1]: Failed to open libbpf, LSM BPF is not supported: Operation not supported
  ↑ this is from the initrd. systemd tries to dlopen() libbpf, and fails.
    This is OK, libbpf doesn't have to be installed in the initrd.

May 16 15:51:13 barnaby systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 16 15:51:13 barnaby kernel: LSM support for eBPF active

The error is from the user manager instance (PID 1291).
When initializing, we only do BPF setup if we are privileged (PID 1).
But we tried to do cleanup unconditionally, which triggered the initialization
logic to detect if we have support, which generated the error about permissions.

https://github.com/systemd/systemd/pull/23407 should fix this.

Comment 11 Fedora Update System 2022-05-25 20:11:03 UTC
FEDORA-2022-3ca356cd2e has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3ca356cd2e

Comment 12 Fedora Update System 2022-05-26 02:32:54 UTC
FEDORA-2022-3ca356cd2e has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3ca356cd2e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3ca356cd2e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2022-05-27 01:10:08 UTC
FEDORA-2022-3ca356cd2e has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Phil O 2022-05-27 15:07:14 UTC
Still seeing this error, even after updating to latest systemd.

$ rpm -q systemd
systemd-250.6-1.fc36.x86_64

$ sudo journalctl -b | egrep -i '(bpf|lsm)'
May 27 07:57:55 kernel: LSM: Security Framework initializing
May 27 07:57:55 kernel: LSM support for eBPF active
May 27 07:57:55 systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 07:57:55 systemd[1]: Failed to open libbpf, LSM BPF is not supported: Operation not supported
May 27 07:58:00 systemd[1]: systemd v250.6-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 07:58:00 systemd[1]: LSM BPF program attached
May 27 07:58:29 systemd[1770]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 27 07:58:29 systemd[1770]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 27 07:58:50 systemd[1913]: libbpf: Error freezing map(restrict.rodata) as read-only: Operation not permitted
May 27 07:58:50 systemd[1913]: libbpf: map 'restrict.rodata': failed to create: Operation not permitted(-1)
May 27 07:58:50 systemd[1913]: libbpf: failed to load object 'restrict_ifaces_bpf'
May 27 07:58:50 systemd[1913]: libbpf: failed to load BPF skeleton 'restrict_ifaces_bpf': -1
May 27 07:58:50 systemd[1913]: Failed to load BPF object: Operation not permitted
May 27 08:04:01 systemd[2229]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 27 08:04:01 systemd[2229]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor

Comment 15 Zbigniew Jędrzejewski-Szmek 2022-05-27 15:30:32 UTC
I see it, but it's a different error.
Before the upgrade, you have "Failed to open libbpf, LSM BPF is not supported: Operation not supported",
but after the upgrade, the spew that starts with "libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor".
It seems that the fix created an issue in some other place.

Comment 16 Ralf Corsepius 2022-05-27 17:11:29 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #15)
> I see it, but it's a different error.
> Before the upgrade, you have "Failed to open libbpf, LSM BPF is not
> supported: Operation not supported",

Well, with systemd v250.6-1.fc36, for me, the error messages on the machines from #6, now look as follows:

# journalctl -r -b | grep -i bpf | grep -v audit
May 27 18:37:02 barnaby systemd[1]: LSM BPF program attached
May 27 18:37:02 barnaby systemd[1]: systemd v250.6-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 18:37:00 barnaby systemd[1]: Failed to open libbpf, LSM BPF is not supported: Operation not supported
May 27 18:37:00 barnaby systemd[1]: systemd v250.6-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 18:37:00 barnaby kernel: LSM support for eBPF active

I.e. what you called "the real error" in Comment#10, is gone, the rest seems unchanged. The "Failed to open libbpf, LSM BPF is not supported: Operation not supported" is still there.


# journalctl -r -b | grep -i bpf | grep -v audit
May 27 19:04:06 troy systemd[1]: LSM BPF program attached
May 27 19:04:06 troy systemd[1]: systemd v250.6-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 21:03:59 troy systemd[1]: LSM BPF program attached
May 27 21:03:59 troy systemd[1]: systemd v250.3-8.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 27 21:03:58 troy kernel: LSM support for eBPF active

Still no idea, why systemd doesn't complain on this machine.

Comment 17 Phil O 2022-05-27 20:03:30 UTC
Ralf - is barnaby using a different kernel which does not have CONFIG_SECURITYFS enabled?  Does /sys/kernel/security/lsm exist, and is "bpf" in it?

Comment 18 Ralf Corsepius 2022-05-28 05:58:15 UTC
(In reply to Phil O from comment #17)
> Ralf - is barnaby using a different kernel which does not have
> CONFIG_SECURITYFS enabled?

All machines are running Fedora's stock kernels. 
I.e. currently kernel-5.17.9-300.fc36.x86_64.

# grep CONFIG_SECURITYFS /boot/config-5.17.9-300.fc36.x86_64 
CONFIG_SECURITYFS=y

>  Does /sys/kernel/security/lsm exist, and is
> "bpf" in it?

On all machines, /sys/kernel/security/lsm exists and contains this:
# cat /sys/kernel/security/lsm
lockdown,capability,yama,selinux,bpf,landlock

FWIW: I am seeing the same behavior as on barnaby on all other machines, but on "troy".
Troy's "special feature" is it being a notebook and my only Windows/Fedora dual boot system. All others are Fedora-only desktops or servers.

Comment 19 Ralf Corsepius 2022-05-29 19:15:24 UTC
Now, I am also experiencing the errors Phil O. has reported:

# journalctl -r -b | grep -i bpf | grep -v audit
May 29 21:12:02 barnaby systemd[5080]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 29 21:12:02 barnaby systemd[5080]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 29 21:11:01 barnaby systemd[5056]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 29 21:11:01 barnaby systemd[5056]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 29 21:10:01 barnaby systemd[5033]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 29 21:10:01 barnaby systemd[5033]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 29 21:09:01 barnaby systemd[5009]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 29 21:09:01 barnaby systemd[5009]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
May 29 21:08:01 barnaby systemd[4987]: libbpf: prog 'sd_restrictif_i': failed to attach to cgroup: Bad file descriptor
May 29 21:08:01 barnaby systemd[4987]: libbpf: prog 'sd_bind4': failed to attach to cgroup: Bad file descriptor
...

Note: One message per minute!!!!

Comment 20 Damian Wrobel 2022-06-01 06:10:14 UTC
With the latest available libbpf and systemd:

$ rpm -qv libbpf systemd
libbpf-0.7.0-3.fc36.aarch64
systemd-250.6-1.fc36.aarch64


I still see errors as follows:

$ journalctl --no-hostname -b | grep -i bpf | grep -v audit
May 25 02:00:03 kernel: LSM support for eBPF active
May 25 02:00:03 systemd[1]: systemd v250.6-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
May 25 02:00:03 systemd[1]: Failed to load BPF object
Jun 01 07:37:00 systemd[572]: Failed to load BPF object: Operation not permitted
Jun 01 07:37:00 systemd[572]: libbpf: Error freezing map(restrict.rodata) as read-only: Operation not permitted
Jun 01 07:37:00 systemd[572]: libbpf: map 'restrict.rodata': failed to create: Operation not permitted(-1)
Jun 01 07:37:00 systemd[572]: libbpf: failed to load object 'restrict_ifaces_bpf'
Jun 01 07:37:00 systemd[572]: libbpf: failed to load BPF skeleton 'restrict_ifaces_bpf': -1

Note: Please don't be confused with timestamps, it's on RPi which doesn't have RTC module.

Comment 21 Fedora Update System 2022-06-03 08:41:02 UTC
FEDORA-2022-303f99728c has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-303f99728c

Comment 22 Damian Wrobel 2022-06-03 10:18:59 UTC
On systemd v250.7-1.fc36 it looks much better:

$ journalctl --no-hostname -b | grep -i bpf | grep -v audit
Jun 02 02:00:03 kernel: LSM support for eBPF active
Jun 02 02:00:03 systemd[1]: systemd v250.7-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jun 02 02:00:03 systemd[1]: Failed to load BPF object: No such process

Comment 23 Ralf Corsepius 2022-06-03 11:39:33 UTC
I am not seeing any difference between systemd v250.7-1.fc36 and systemd v250.6-1.fc36.

The outputs are still as in comment #16.

Comment 24 Zbigniew Jędrzejewski-Szmek 2022-06-03 13:06:01 UTC
> The outputs are still as in comment #16.

The outputs in #16 seem OK. "Failed to open libbpf, LSM BPF is not supported: Operation not supported"
is most likely from the initrd, where libbpf is not available.

Comment 25 Ralf Corsepius 2022-06-03 14:02:58 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #24)
> > The outputs are still as in comment #16.
> 
> The outputs in #16 seem OK. "Failed to open libbpf, LSM BPF is not
> supported: Operation not supported"
> is most likely from the initrd, where libbpf is not available.

OK, but do you have any explanation, why I am seeing this warning only on one of these two machines?

Comment 26 Zbigniew Jędrzejewski-Szmek 2022-06-03 14:49:15 UTC
> why I am seeing this warning only on one of these two machines?

I don't think it's worth figuring out. https://github.com/systemd/systemd/pull/23617
should get rid of the warning altogether.

Comment 27 Ralf Corsepius 2022-06-03 14:59:35 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #26)
> > why I am seeing this warning only on one of these two machines?
> 
> I don't think it's worth figuring out.
Why? AFAIU, it means systemd is behaving non-deterministically.
Something which should not be take light-heartedly, IMHO.

Comment 28 Zbigniew Jędrzejewski-Szmek 2022-06-03 15:05:12 UTC
If you think that this is important and you want to figure this out, start by checking that the message
is indeed from the initrd, and unpacking the initrds and checking if they both have libbpf...

Comment 29 Stan King 2022-06-03 19:59:22 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #28)
> If you think that this is important and you want to figure this out, start
> by checking that the message
> is indeed from the initrd, and unpacking the initrds and checking if they
> both have libbpf...

On my affected system, booting from the Fedora 36 Xfce Live image does not have this BPF problem.

Does knowing that help narrow down the problem?  Perhaps the initrd is being incorrectly generated on certain CPU models?

Comment 30 Phil O 2022-06-04 01:50:18 UTC
Stan/Ralf: did you regenerate your initrd?  Installing an updated kernel will do so, but if you haven't done that then the initrd has the prior version.  You can force a regen with dracut.

Comment 31 Fedora Update System 2022-06-04 02:08:07 UTC
FEDORA-2022-303f99728c has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-303f99728c`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-303f99728c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 32 Stan King 2022-06-04 02:13:37 UTC
(In reply to Phil O from comment #30)
> Stan/Ralf: did you regenerate your initrd?  Installing an updated kernel
> will do so, but if you haven't done that then the initrd has the prior
> version.  You can force a regen with dracut.

I've done just the usual updates, and yes, the system installed on the hard drive has undergone some kernel updates since installation from the live image.  Today, I verified that the live image showed normal BPF messages upon boot.

This one particular system shows the BPF problems discussed in this bugzilla.  Three other F36 systems do not have the problem.  Those other systems have different CPUs, motherboards, etc.

Comment 33 Ralf Corsepius 2022-06-04 06:22:48 UTC
I think, I've found something:

troy's (The machine not complaining about missing libbpf) initrd contains libbpf:
[root@troy tmp]# lsinitrd /boot/initramfs-5.17.11-300.fc36.x86_64.img | grep bpf
-rwxr-xr-x   1 root     root       364200 Mar 30 10:53 usr/lib64/libbpf.so.0.7.0
lrwxrwxrwx   1 root     root           27 Mar 30 10:53 usr/lib64/libbpf.so.0 -> ../../lib64/libbpf.so.0.7.0

barnaby's (The machine complaining about missing libbpf) initrd doesn't:
[root@barnaby tmp]# lsinitrd /boot/initramfs-5.17.11-300.fc36.x86_64.img | grep bpf

After some time comparing troy's and barnaby's rpms, I found barnaby didn't have dracut-network installed, so I decided to installed it and rebuild initrd:
[root@barnaby tmp]# dnf install dracut-network
..
[root@barnaby tmp]# dracut -f
..

E voilà, now, libbpf is in initrd:
[root@barnaby tmp]# lsinitrd /boot/initramfs-5.17.11-300.fc36.x86_64.img | grep bpf
-rwxr-xr-x   1 root     root       364200 Mar 30 10:53 usr/lib64/libbpf.so.0.7.0
lrwxrwxrwx   1 root     root           27 Mar 30 10:53 usr/lib64/libbpf.so.0 -> ../../lib64/libbpf.so.0.7.0


Reboot, checking the journal:
[root@barnaby barnaby]# journalctl -r -b | grep -i bpf | grep -v audit
Jun 04 08:14:29 barnaby systemd[1]: LSM BPF program attached
Jun 04 08:14:29 barnaby systemd[1]: systemd v250.7-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jun 04 08:14:26 barnaby systemd[1]: LSM BPF program attached
Jun 04 08:14:26 barnaby systemd[1]: systemd v250.7-1.fc36 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Jun 04 08:14:26 barnaby kernel: LSM support for eBPF active
 
=> The "Failed to open libbpf, ..." message is gone!

=> Yes, this error seems to originate from an initrd. 

Apparently, current dracut doesn't match well with current systemd's demands.

Comment 34 Ralf Corsepius 2022-06-04 08:19:40 UTC
Wild guess: 

AFAIU, dracut seems to apply some dep-tracking of dynamically linked libraries to put libraries into initrd. 

Due to the fact systemd dlopens libbpf, dracut fails to add systemd's dependency on libbpf in initrd, but dracut-network adds it to initrd, because it pulls in /usr/sbin/ip, which is linked against libbpf.so.0.

Comment 35 Stan King 2022-06-04 19:17:55 UTC
(In reply to Ralf Corsepius from comment #34)
> Wild guess: 
> 
> AFAIU, dracut seems to apply some dep-tracking of dynamically linked
> libraries to put libraries into initrd. 
> 
> Due to the fact systemd dlopens libbpf, dracut fails to add systemd's
> dependency on libbpf in initrd, but dracut-network adds it to initrd,
> because it pulls in /usr/sbin/ip, which is linked against libbpf.so.0.

In my affected system, I do have dracut-network, but "dracut -f" still produces an initrd without libbpf.

My affected system is the only one that is a fresh install of F36, rather than an upgrade from F35.  Would that make a difference?

Comment 36 Zbigniew Jędrzejewski-Szmek 2022-06-05 08:26:02 UTC
Let me repeat: this is not a problem. The warning is harmless.

Comment 37 Ralf Corsepius 2022-06-05 10:57:35 UTC
Let me repeat: dlopen is harmful, attempts to silence warnings on bugs, even if they are harmless, is non-professional

Comment 38 Fedora Update System 2022-06-06 02:11:19 UTC
FEDORA-2022-303f99728c has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.