Bug 1955585 - selinux prevents tracing containers with BPF
Summary: selinux prevents tracing containers with BPF
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ondrej Mosnacek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1958025 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-30 12:53 UTC by Jakub Hrozek
Modified: 2021-06-15 01:21 UTC (History)
30 users (show)

Fixed In Version: kernel-5.12.10-300.fc34 kernel-5.12.10-200.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-15 01:05:17 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jakub Hrozek 2021-04-30 12:53:48 UTC
Description of problem:
I starting seeing this with F-34. When I run a container that is traced with eBPF to record the syscalls it is doing, auditd is flooded with messages like:

type=AVC msg=audit(1619784520.593:282387): avc:  denied  { confidentiality } for  pid=476 comm="auditd" lockdown_reason="use of bpf to read kernel RAM" scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:system_r:auditd_t:s0 tclass=lockdown permissive=0

This seems to be leading to auditd running out of space in the backlog buffer and eventually OOMs the machine.

Version-Release number of selected component (if applicable):
podman-3.1.2-1.fc34.x86_64
selinux-policy-34-1.fc34.noarch
container-selinux-2.160.0-2.fc34.noarch
oci-seccomp-bpf-hook-1.2.2-0.1.git4e42394.fc34.x86_64
kernel-core-5.11.12-300.fc34.x86_64

How reproducible:
always

Steps to Reproduce:
1. sudo podman run --name=seccomp-test --privileged --annotation io.containers.trace-syscall="of:/tmp/selinuxd-seccomp.json" fedora sh

(the container must be started as root for the bpf hook to work)

Actual results:
auditd running at 99% CPU presumably processing all the messages, eventually I get:
Apr 30 12:20:42 fedora kernel: audit: backlog limit exceeded
Apr 30 12:20:42 fedora kernel: audit: backlog limit exceeded
Apr 30 12:20:42 fedora kernel: audit: audit_backlog=2152579 > audit_backlog_limit=64
Apr 30 12:20:42 fedora kernel: audit: audit_backlog=2152626 > audit_backlog_limit=64
Apr 30 12:20:42 fedora kernel: audit: audit_backlog=2152694 > audit_backlog_limit=64
Apr 30 12:20:42 fedora kernel: audit: audit_lost=6878426 audit_rate_limit=0 audit_backlog_limit=64
Apr 30 12:20:45 fedora kernel: oci-seccomp-bpf invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-1000
Apr 30 12:20:45 fedora kernel: CPU: 0 PID: 13284 Comm: oci-seccomp-bpf Not tainted 5.11.12-300.fc34.x86_64 #1
Apr 30 12:20:45 fedora kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-2.fc32 04/01/2014


Expected results:
the BPF hook should work as it did in F-33

Additional info:
I don't know if the issue is in selinux-policy, the kernel, the eBPF hook or somewhere else. Feel free to reassign the bug as appropriate or ask for any other information.

Comment 1 Ondrej Mosnacek 2021-05-05 11:38:39 UTC
Actually, this seems to be a kernel bug. Apparently there is a security_locked_down() call in a few BPF helper functions, so it gets called when the BPF program is run, leading to SELinux looking up the permission for the domain of the process that performs the given syscall and not the one that loaded the program.

I'm currently working on a patch to fix this and a few related issues, so reassigning this to myself.

Anyway, I don't understand why this particular program is triggering the calls, since it doesn't call any of the problematic helpers explicitly... There might be some BPF compiler weirdness behind this, too...

Comment 2 Ondrej Mosnacek 2021-05-07 08:49:39 UTC
*** Bug 1958025 has been marked as a duplicate of this bug. ***

Comment 3 Chris Murphy 2021-05-10 20:41:07 UTC
Also with 5.12.2-300.fc34.x86_64+debug. And it's rampant enough that the audit.log is being rotated every 5 seconds.

Comment 6 Ondrej Mosnacek 2021-05-26 13:26:24 UTC
FYI, in the meantime I posted a patch to fix this issue upstream:

[v1] https://lore.kernel.org/selinux/20210507114048.138933-1-omosnace@redhat.com/T/
[v2] https://lore.kernel.org/selinux/20210517092006.803332-1-omosnace@redhat.com/T/

Comment 7 Justin M. Forbes 2021-05-26 15:44:56 UTC
@Ondrej do you mind updating here when this patch has been accepted upstream.  It doesn't have to be in linus tree yet, just accepted upstream. I can pull it back to 5.12 and newer releases at that point.

Comment 8 Ondrej Mosnacek 2021-06-07 11:13:53 UTC
So, this particular issue has now been addressed on the BPF side with this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/kernel/bpf?id=ff40e51043af63715ab413995ff46996ecf9583f

There is still an an ongoing discussion on how to address the problem better from security perspective, but that's beyond the scope of this bug.

Comment 9 Graham White 2021-06-09 08:11:02 UTC
@Justin, it would be good to know which kernel release this lands in for Fedora.  If you're able to update here when you've pulled it into 5.12 or other releases. Thanks!

Comment 10 Ondrej Mosnacek 2021-06-09 08:31:54 UTC
BTW, the patch has been queued for the 5.12.10 stable release yesterday [1], so it may be best to just let Fedora inherit it naturally from upstream at this point.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/?h=linux-5.12.y&id=de9dc6a1cc8d21cae0efa923dc370683ac842308

Comment 11 Graham White 2021-06-09 08:45:55 UTC
OK, great.  Thanks very much.

Comment 12 Justin M. Forbes 2021-06-09 13:02:58 UTC
Yes, 5.12.10 is in rc phase now, expected to release Thursday sometime. Depending on that time, the build will be done for Fedora either Thursday or Friday. This bug will be included in the update, so bodhi should auto post here as it is filed and works through to stable.

Comment 13 Fedora Update System 2021-06-10 23:15:42 UTC
FEDORA-2021-bc2a819bc5 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc2a819bc5

Comment 14 Fedora Update System 2021-06-10 23:15:59 UTC
FEDORA-2021-db2bb87f35 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-db2bb87f35

Comment 15 Fedora Update System 2021-06-11 01:42:47 UTC
FEDORA-2021-db2bb87f35 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-db2bb87f35`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-db2bb87f35

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2021-06-11 02:08:09 UTC
FEDORA-2021-bc2a819bc5 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-bc2a819bc5`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc2a819bc5

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-06-15 01:05:17 UTC
FEDORA-2021-bc2a819bc5 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 Fedora Update System 2021-06-15 01:21:28 UTC
FEDORA-2021-db2bb87f35 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.