RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1481617 - Regression issue mounting a single file
Summary: Regression issue mounting a single file
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.6
Hardware: All
OS: Linux
medium
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
: 1475137 (view as bug list)
Depends On:
Blocks: 1186913 1537321
TreeView+ depends on / blocked
 
Reported: 2017-08-15 08:05 UTC by Babak Mozaffari
Modified: 2021-03-11 15:35 UTC (History)
19 users (show)

Fixed In Version: docker-1.12.6-61.git85d7426.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-29 22:08:07 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Template to reproduce mount file issue (1014 bytes, text/plain)
2017-08-15 08:05 UTC, Babak Mozaffari 		no flags Details
View All

Description Babak Mozaffari 2017-08-15 08:05:49 UTC 
Created attachment 1313499 [details]
Template to reproduce mount file issue

Description of problem:
The combination of mountPath and subPath should allow injecting a single file without overwriting the entire parent directory. This worked in OCP 3.5 but is broken in 3.6

Version-Release number of selected component (if applicable):

How reproducible:
Solution described no longer works in OCP 3.6: https://access.redhat.com/solutions/3100761

Steps to Reproduce:
1. Run the attached template: oc new-app -f mount-file.yml

2. Once the pod is on, hit the httpd homepage and see if the index.html file has been successfully replaced:
oc describe pod `oc get pods | grep Running | grep -v deploy | awk {'print $1'}`| grep IP | awk '{print $2}' | xargs -I @ curl -i @:8080

3. Once the pod is running, this curl command will return a value

Actual results:
HTTP/1.1 403 Forbidden

Expected results:
Success!

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
If you rsh into the pod, for the two versions of OCP respectively:

*** OCP 3.6:
ls -l /opt/rh/httpd24/root/usr/share/httpd/noindex/ 
ls: cannot access /opt/rh/httpd24/root/usr/share/httpd/noindex/index.html: Permission denied
total 0
-????????? ? ? ? ?            ? index.html


*** OCP 3.5:
ls -l /opt/rh/httpd24/root/usr/share/httpd/noindex/ 
total 4
-rw-r--r--. 1 root 1000620000 9 Aug 15 06:27 index.html
Comment 1 Matthew Wong 2017-08-16 21:44:04 UTC 
It has something to do with SELinux and the configMap index.html being a symlink (to ..data/index.html) under the hood. Docker will relabel index.html to svirt_sandbox_file_t, but the label of index.html doesn't actually matter, it's ..data/index.html which must be relabeled. The template will work if altered slightly to have `subPath: ..data/index.html`, but of course since this exploits an implementation detail of openshift this is not recommended as anything more than a temporary workaround...

I'm tempted to blame Docker for not following symlinks when relabelling volume files, but more investigation is needed to determine why this issue arose in 3.6.
Comment 2 Matthew Wong 2017-08-17 20:15:07 UTC 
IMO it is a bug that Docker does not follow symlinks when changing SELinux contexts via :Z.

The reason we're only seeing this now in 3.6 is https://github.com/openshift/origin/pull/12942. Prior to 3.6 we were insulated from the bug because every file under openshift.local.volumes was getting created with svirt_sandbox_file_t anyway, so even though index.html was chcon'd and ..data/index.html wasn't, the data could still be read from the container. (It's perhaps a separate security issue that the actual file ..data/index.html wasn't getting chcon'd because that means it wasn't getting the SELinux categories applied to it.)

We should fix the bug in Docker.
Comment 3 Glenn West 2017-08-18 01:25:19 UTC 
This appears to be blocking me as well.
When I deploy ocp 3.6 on azure, I need to update the registry config file,
and uses similar concept. This is now broken, resulting in registry not using the proper storage provider.
Comment 4 Daniel Walsh 2017-08-25 10:57:00 UTC 
What AVC are you getting?

I take it this is the equivalent of doing


mkdir /var/foo
touch /var/foo/dan
ln -s /var/foo /var/bar

docker run -ti -v /var/bar:/var/bar:Z IMAGE sh

And you want SELinux to relabel /var/foo

sh-4.4# mkdir /var/foo
sh-4.4# touch /var/foo/dan
sh-4.4# ln -s /var/foo /var/bar
sh-4.4# docker run -ti -v /var/bar:/var/bar:Z fedora sh
/usr/bin/docker-current: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See '/usr/bin/docker-current run --help'.
sh-4.4# systemctl start docker
sh-4.4# docker run -ti -v /var/bar:/var/bar:Z fedora sh
sh-4.3# ls -lZ /var/bar/
ls: cannot open directory '/var/bar/': Permission denied
Comment 5 Daniel Walsh 2017-08-25 11:02:32 UTC 
Antonio does docker translate the volume mount of a symbolic link into its realpath before mounting?  If so we should be passing this to SELinux not the symbolic link.  If the kernel is doing this under the covers then we have a different issue.
Comment 6 Daniel Walsh 2017-08-25 11:05:14 UTC 
I think kernel is doing this for us.  So we should probably change docker to pass realpath to SELinux for relabeling.

realpath /var/bar
/var/foo
Comment 7 Bradley Childs 2017-09-07 19:50:31 UTC 
Moving to RHEL/Docker
Comment 9 Seth Jennings 2017-09-08 18:38:25 UTC 
*** Bug 1475137 has been marked as a duplicate of this bug. ***
Comment 10 Antonio Murdaca 2017-09-08 23:34:37 UTC 
I've opened an upstream PR: https://github.com/moby/moby/pull/34792

Meanwhile, back ported to 1.12.6 (https://github.com/projectatomic/docker/commit/1a89821641e5527d303ad88ac769316a3ed796e9) and 1.13.1 (https://github.com/projectatomic/docker/commit/05021bc5b87b0a2648f74508ccb8214b763274ed)

Dan, PTAL at the upstream PR.

Moving to POST.
Comment 11 Jan Pazdziora 2017-10-09 14:10:57 UTC 
The upstream PR https://github.com/moby/moby/pull/34792 seems to have been merged on 2017-09-27.
Comment 12 Daniel Walsh 2017-10-09 15:05:48 UTC 
Looks like we already have back port.
Comment 14 Daniel Walsh 2017-10-10 14:48:15 UTC 
Lokesh can we get this into next RHEL Release.
Note You need to log in before you can comment on or make changes to this bug.