Bug 1481617

Summary: Regression issue mounting a single file
Product: Red Hat Enterprise Linux 7 Reporter: Babak Mozaffari <bmozaffa>
Component: dockerAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 7.6CC: amurdaca, aos-bugs, aos-storage-staff, bchilds, dornelas, dwalsh, gwest, jamills, jgodoy, jnordell, jpazdziora, jrosenta, jswensso, lsm5, mawong, nschuetz, sbunciak, tatanaka, tparsons
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: docker-1.12.6-61.git85d7426.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-29 22:08:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913, 1537321    
Attachments:
Description Flags
Template to reproduce mount file issue none

Description Babak Mozaffari 2017-08-15 08:05:49 UTC
Created attachment 1313499 [details]
Template to reproduce mount file issue

Description of problem:
The combination of mountPath and subPath should allow injecting a single file without overwriting the entire parent directory. This worked in OCP 3.5 but is broken in 3.6

Version-Release number of selected component (if applicable):

How reproducible:
Solution described no longer works in OCP 3.6: https://access.redhat.com/solutions/3100761

Steps to Reproduce:
1. Run the attached template: oc new-app -f mount-file.yml

2. Once the pod is on, hit the httpd homepage and see if the index.html file has been successfully replaced:
oc describe pod `oc get pods | grep Running | grep -v deploy | awk {'print $1'}`| grep IP | awk '{print $2}' | xargs -I @ curl -i @:8080

3. Once the pod is running, this curl command will return a value

Actual results:
HTTP/1.1 403 Forbidden

Expected results:
Success!

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
If you rsh into the pod, for the two versions of OCP respectively:

*** OCP 3.6:
ls -l /opt/rh/httpd24/root/usr/share/httpd/noindex/ 
ls: cannot access /opt/rh/httpd24/root/usr/share/httpd/noindex/index.html: Permission denied
total 0
-????????? ? ? ? ?            ? index.html


*** OCP 3.5:
ls -l /opt/rh/httpd24/root/usr/share/httpd/noindex/ 
total 4
-rw-r--r--. 1 root 1000620000 9 Aug 15 06:27 index.html

Comment 1 Matthew Wong 2017-08-16 21:44:04 UTC
It has something to do with SELinux and the configMap index.html being a symlink (to ..data/index.html) under the hood. Docker will relabel index.html to svirt_sandbox_file_t, but the label of index.html doesn't actually matter, it's ..data/index.html which must be relabeled. The template will work if altered slightly to have `subPath: ..data/index.html`, but of course since this exploits an implementation detail of openshift this is not recommended as anything more than a temporary workaround...

I'm tempted to blame Docker for not following symlinks when relabelling volume files, but more investigation is needed to determine why this issue arose in 3.6.

Comment 2 Matthew Wong 2017-08-17 20:15:07 UTC
IMO it is a bug that Docker does not follow symlinks when changing SELinux contexts via :Z.

The reason we're only seeing this now in 3.6 is https://github.com/openshift/origin/pull/12942. Prior to 3.6 we were insulated from the bug because every file under openshift.local.volumes was getting created with svirt_sandbox_file_t anyway, so even though index.html was chcon'd and ..data/index.html wasn't, the data could still be read from the container. (It's perhaps a separate security issue that the actual file ..data/index.html wasn't getting chcon'd because that means it wasn't getting the SELinux categories applied to it.)

We should fix the bug in Docker.

Comment 3 Glenn West 2017-08-18 01:25:19 UTC
This appears to be blocking me as well.
When I deploy ocp 3.6 on azure, I need to update the registry config file,
and uses similar concept. This is now broken, resulting in registry not using the proper storage provider.

Comment 4 Daniel Walsh 2017-08-25 10:57:00 UTC
What AVC are you getting?

I take it this is the equivalent of doing


mkdir /var/foo
touch /var/foo/dan
ln -s /var/foo /var/bar

docker run -ti -v /var/bar:/var/bar:Z IMAGE sh

And you want SELinux to relabel /var/foo

sh-4.4# mkdir /var/foo
sh-4.4# touch /var/foo/dan
sh-4.4# ln -s /var/foo /var/bar
sh-4.4# docker run -ti -v /var/bar:/var/bar:Z fedora sh
/usr/bin/docker-current: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See '/usr/bin/docker-current run --help'.
sh-4.4# systemctl start docker
sh-4.4# docker run -ti -v /var/bar:/var/bar:Z fedora sh
sh-4.3# ls -lZ /var/bar/
ls: cannot open directory '/var/bar/': Permission denied

Comment 5 Daniel Walsh 2017-08-25 11:02:32 UTC
Antonio does docker translate the volume mount of a symbolic link into its realpath before mounting?  If so we should be passing this to SELinux not the symbolic link.  If the kernel is doing this under the covers then we have a different issue.

Comment 6 Daniel Walsh 2017-08-25 11:05:14 UTC
I think kernel is doing this for us.  So we should probably change docker to pass realpath to SELinux for relabeling.

realpath /var/bar
/var/foo

Comment 7 Bradley Childs 2017-09-07 19:50:31 UTC
Moving to RHEL/Docker

Comment 9 Seth Jennings 2017-09-08 18:38:25 UTC
*** Bug 1475137 has been marked as a duplicate of this bug. ***

Comment 10 Antonio Murdaca 2017-09-08 23:34:37 UTC
I've opened an upstream PR: https://github.com/moby/moby/pull/34792

Meanwhile, back ported to 1.12.6 (https://github.com/projectatomic/docker/commit/1a89821641e5527d303ad88ac769316a3ed796e9) and 1.13.1 (https://github.com/projectatomic/docker/commit/05021bc5b87b0a2648f74508ccb8214b763274ed)

Dan, PTAL at the upstream PR.

Moving to POST.

Comment 11 Jan Pazdziora 2017-10-09 14:10:57 UTC
The upstream PR https://github.com/moby/moby/pull/34792 seems to have been merged on 2017-09-27.

Comment 12 Daniel Walsh 2017-10-09 15:05:48 UTC
Looks like we already have back port.

Comment 14 Daniel Walsh 2017-10-10 14:48:15 UTC
Lokesh can we get this into next RHEL Release.