Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2074090

Summary: Podman 4.0.0+ fails to SELinux-relabel symlinks in volumes when using the ":Z" option, resulting in "permission denied" errors
Product: Red Hat Enterprise Linux 8 Reporter: Javi Polo <jpolo>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.6CC: asreddy, bbaude, dornelas, dwalsh, jligon, jnovy, lsm5, mheon, miabbott, mirollin, pthomas, rmanes, tsweeney, umohnani, walters
Target Milestone: rcKeywords: Regression, Reopened
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-17 06:13:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2072072    

Description Javi Polo 2022-04-11 14:09:42 UTC
Description of problem:
When mounting a volume containing symlinks using Podman 4.0.0+ in a SELinux-enabled system (e.g. a clean Fedora 36 Workstation beta install), specifying the :Z flag in order to relabel the files in the volume, the symlink is not relabeled (but regular files are), resulting in "permission denied" errors

Version-Release number of selected component (if applicable): 4.0.0+


How reproducible:
Always

Steps to Reproduce:
1.podman run --rm -v .:/stuff:Z busybox ls -lah /stuff

Actual results:
$ podman run --rm -v .:/stuff:Z busybox ls -lah /stuff
ls: /stuff/link: Permission denied
total 4K     
drwxr-xr-x    1 root     root          16 Mar 30 20:38 .
dr-xr-xr-x    1 root     root          36 Mar 30 20:39 ..
-rw-r--r--    1 root     root           6 Mar 30 20:38 file


Expected results:
$ podman run --rm -v .:/stuff:Z busybox ls -lah /stuff
total 4K     
drwxr-xr-x    1 root     root          16 Mar 30 20:38 .
dr-xr-xr-x    1 root     root          36 Mar 30 20:39 ..
-rw-r--r--    1 root     root           6 Mar 30 20:38 file


Additional info:
This bug is breaking Openshift installations, because podman gets run in a loop during startup until the container exits properly, and due to this bug it's always failing:

until /usr/bin/podman run --rm --authfile /var/lib/kubelet/config.json    --net=host --volume /etc/systemd/system:/etc/systemd/system:z SOMECONTAINER:SOMEVERSION node-ip set --retry-on-failure SOMEIP; do sleep 5; done

The bug has been identified and fixed upstream already:
https://github.com/containers/podman/pull/13689

We need this in RHEL8.6 before it goes GA

Comment 1 Daniel Walsh 2022-04-11 19:24:47 UTC
Fixed in podman 4.1

This was fixed in the opencontainers/selinux package which is now vendored into podman upstream.  4.1 RC1 is being built next tuesday.

Comment 3 Matthew Heon 2022-04-12 15:58:10 UTC
Can I ask why you are attempting to use `--volume /etc/systemd/system:/etc/systemd/system:z`? I would not consider this a safe command to run - you are altering SELinux labeling of a very important directory on the host, and that could easily break systemd?

The proper answer here is likely to either use Udica to generate an SELinux policy that allows access to that directory, or disable labeling on the container with `--security-opt label=disable`

Comment 4 Daniel Walsh 2022-04-12 16:01:53 UTC
Good Catch Matt, this should not be done.  Relabling system content from the host within a container will break the system functionality.  Any confined domain that needs to use /etc/sytemd/system will be broken by this container.

You should disable SELinux separation on this container by running it with spc_t type.

Not relabeling the content.

The actually relabling content issue will be fixed in podman 4.1 but it should not be required for this bugzilla since what  you are trying to do will break the system.

Comment 5 Micah Abbott 2022-04-12 16:04:05 UTC
I think the failing `podman` commands are being called from this MCO template - https://github.com/openshift/machine-config-operator/blob/c415ce6aed25604bc1d2478951db16759dac31f6/templates/common/on-prem/units/nodeip-configuration.service.yaml#L19-L31

Comment 6 Colin Walters 2022-04-12 16:18:47 UTC
Yes, this is a MCO bug.    Filed https://bugzilla.redhat.com/show_bug.cgi?id=2074613

Comment 7 Javi Polo 2022-04-12 16:20:51 UTC
I really dont know since I just found the issue while debugging why cluster won't install using rhel8.6 based rhcos

As Micah says, this code is part of Machine Config Operator ... AFAIK it's at least in 2 different templates:
https://github.com/openshift/machine-config-operator/blob/5a571cd23751abb39708cd5707fef57683d45817/templates/common/_base/units/nodeip-configuration.service.yaml#L18-L29
https://github.com/openshift/machine-config-operator/blob/c415ce6aed25604bc1d2478951db16759dac31f6/templates/common/on-prem/units/nodeip-configuration.service.yaml#L21-L28

Comment 8 Colin Walters 2022-04-12 16:21:58 UTC
Direct xref https://github.com/openshift/machine-config-operator/pull/3079

Comment 9 Daniel Walsh 2022-04-12 17:52:08 UTC
You should never relabel system files period.  If you need to use System Files within a container you need to disable SELinux separation.

Comment 12 Tom Sweeney 2022-06-10 19:06:26 UTC
Per Dan Walsh, fixed in Podman v4.1.1 which will be in RHEL 8.6.0.2.  Setting to Post and assigning to Jindrich for any further BZ/Packaging needs.