Bug 1768075 - Runs SELinux operations in toolbox, causes package install failures
Summary: Runs SELinux operations in toolbox, causes package install failures
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-02 09:06 UTC by Martin Pitt
Modified: 2019-11-29 02:46 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-29 00:53:54 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github containers libpod issues 4452 'None' 'closed' '/sys/fs/selinux gets mounted in rootless mode when both ''--network host'' and ''--privileged'' are used' 2019-11-26 04:51:37 UTC

Description Martin Pitt 2019-11-02 09:06:12 UTC
Description of problem: I have a script [1] to build my development toolbox about once a week, which I've used for quite some time now since early Fedora 30. During Fedora 31's alpha and beta periods this also worked reasonably well, but shortly before F31 release it now fails.

In particular, it seems that package installs in an F31 now trigger SELinux operations such as restorecon, which break the toolbox in multiple ways.

I reduced this quite a bit to get a small reproducer (see below). As soon as selinux-policy{,-targeted} gets installed into the toolbox, things start to go haywire. I don't explicitly want/need this package in my toolbox, but it's an inevitable dependency of e. g. `dnf builddep cockpit`. My understanding is that in containers it should be pretty much inert.


[1] https://piware.de/gitweb/?p=bin.git;a=blob;f=build-devtoolbox

Version-Release number of selected component (if applicable):

selinux-policy-targeted-3.14.4-39.fc31.noarch
podman-1.6.2-2.fc31.x86_64
toolbox-0.0.15-1.fc31.noarch
kernel-5.3.7-301.fc31.x86_64

How reproducible: Always


Steps to Reproduce:
1. toolbox create -c test
2. toolbox run -c test sh -exc 'sudo ln -sfn /run/host/monitor/resolv.conf /etc/resolv.conf; sudo dnf install -y selinux-policy selinux-policy-targeted'
3. toolbox run -c test dnf install -y systemd-udev

Actual results:

- Step 2 takes a long time as that triggers restorecon¸ which runs at 100% CPU for a minute or two.

- Step 3 fails. First, during unpack, there are tons of errors like

    error: lsetfilecon: (/usr/share/man/man8/kbdrate.8.gz;5dbd4232, system_u:object_r:man_t:s0) Operation not supported

  and finally

  Running scriptlet: systemd-udev-243-4.gitef67743.fc31.x86_64                                        10/14
error: failed to exec scriptlet interpreter /bin/sh: Permission denied
error: %prein(systemd-udev-243-4.gitef67743.fc31.x86_64) scriptlet failed, exit status 127

Error in PREIN scriptlet in rpm package systemd-udev
  Cleanup          : systemd-rpm-macros-243-1.fc31.noarch                                             11/14
error: systemd-udev-243-4.gitef67743.fc31.x86_64: install failed
error: systemd-243-1.fc31.x86_64: erase skipped

  Cleanup          : systemd-libs-243-1.fc31.x86_64                                                   12/14
  Cleanup          : systemd-pam-243-1.fc31.x86_64                                                    13/14
  Running scriptlet: systemd-pam-243-1.fc31.x86_64                                                    13/14
error: failed to exec scriptlet interpreter /bin/sh: Permission denied
warning: %triggerin(glibc-common-2.30-1.fc31.x86_64) scriptlet failed, exit status 127

Failed:
  systemd-udev-243-4.gitef67743.fc31.x86_64 systemd-243-4.gitef67743.fc31.x86_64 systemd-243-1.fc31.x86_64



Expected results: Package install works, as before.


Additional info:

I tried this with `toolbox create --release 30 -c test` and it fails the same way -- i. e. I'm pretty sure that this is a bug on the host side, not the guest. 

This is *not* a direct regression from cgroupsv2 -- after the initial hiccups with that switch, my script worked fine for a while.

This must be related to some of the additional magic in toolbox. Doing this with podman directly works fine:

    podman run -it --rm -u root fedora:31 bash -exc 'dnf install -y selinux-policy selinux-policy-targeted; dnf install -y systemd-udev'

I. e. there is no hang on installing selinux-policy (restorecon isn't being called), no lsetfilecon errors, and package install works fine.

Comment 1 Martin Pitt 2019-11-02 09:08:39 UTC
> 2. toolbox run -c test sh -exc 'sudo ln -sfn /run/host/monitor/resolv.conf /etc/resolv.conf; sudo dnf install -y selinux-policy selinux-policy-targeted'

Sorry, here my local workaround for "toolbox does not work with systemd-resolved on the host" slipped in. If you use the standard "static /etc/resolv.conf" setup, then it's of course just 

    toolbox run -c test sudo dnf install -y selinux-policy selinux-policy-targeted

Comment 2 Martin Pitt 2019-11-02 09:43:59 UTC
CC'ing Andreas and Garrett, two colleagues of mine who also use OSTree and toolbox. FYI, you likely run into the same trouble.

I tried to work around this with a simple 

    sudo ln -sfn /dev/null /etc/selinux/config

This actually goes get preserved after `dnf install selinux-policy`, but doesn't help to prevent the bug. But removing all files in selinux-policy does, so I added that as a blunt hack to my build script: https://piware.de/gitweb/?p=bin.git;a=commitdiff;h=7e476e5f7c7f4fe

Comment 3 Debarshi Ray 2019-11-05 17:04:42 UTC
Thanks for the detailed report, Martin!

I can reproduce that installing selinux-policy-targeted does take a while inside a toolbox container, but doesn't inside a throwaway 'podman run ...' session.

I also got these:
  Running scriptlet: selinux-policy-targeted-3.14.4-40.fc31.noarch                                                              9/15 
SELinux:  Could not load policy file /etc/selinux/targeted/policy/policy.31:  Permission denied
load_policy:  Can't load policy:  Permission denied

Comment 4 Debarshi Ray 2019-11-05 17:41:37 UTC
This is the 'restorecon' invocation that's getting "stuck":
restorecon -R /root /var/log /run /etc/passwd /etc/passwd- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /etc/shadow /etc/shadow-

When run inside a 'podman run ...' session, it cleanly and immediately exits, but when run inside a toolbox container it doesn't and complains a lot about things inside the container's /run/host. For example:
/sbin/restorecon: Could not set context for /run/host/usr/share/icons/HighContrast/48x48/actions/view-more.png:  Operation not permitted

These directories under /run/host that it complains about are bind mounts from the host. /run/host/usr is the host's /usr.

Comment 5 Debarshi Ray 2019-11-05 18:07:30 UTC
Quickly grepping through the code of restorecon [1], particularly init_selinuxmnt, I see one difference. /sys/fs/selinux is empty inside the 'podman run ...' session, but not inside a toolbox container. I wonder why.

[1] https://github.com/SELinuxProject/selinux

Comment 6 Debarshi Ray 2019-11-05 18:28:46 UTC
https://github.com/containers/toolbox/issues/47 has raised its head again.

Compare these two:
$ podman run --interactive --tty --network host --privileged --rm registry.fedoraproject.org/f31/fedora-toolbox:31 ls /sys/fs/selinux
...
$ podman run --interactive --tty --rm registry.fedoraproject.org/f31/fedora-toolbox:31 ls /sys/fs/selinux

Comment 7 Debarshi Ray 2019-11-05 18:50:56 UTC
Filed: https://github.com/containers/libpod/issues/4452

Comment 8 Fedora Update System 2019-11-20 18:11:46 UTC
FEDORA-2019-78653c0d0b has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-78653c0d0b

Comment 9 Fedora Update System 2019-11-20 18:12:43 UTC
FEDORA-2019-70b5213b27 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-70b5213b27

Comment 10 Fedora Update System 2019-11-20 18:14:21 UTC
FEDORA-2019-c786a43afe has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c786a43afe

Comment 11 Fedora Update System 2019-11-21 02:27:29 UTC
toolbox-0.0.17-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-78653c0d0b

Comment 12 Fedora Update System 2019-11-21 04:18:12 UTC
toolbox-0.0.17-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-70b5213b27

Comment 13 Fedora Update System 2019-11-21 04:22:16 UTC
toolbox-0.0.17-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c786a43afe

Comment 14 Fedora Update System 2019-11-29 00:53:54 UTC
toolbox-0.0.17-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2019-11-29 02:46:23 UTC
toolbox-0.0.17-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.