In general, people need an easy way to run kubernetes apps with host mounts. This bug effects Red Hat Atomic 7.1.3 To reproduce, (1) enable selinux and (2) try to run any kubernetes app which mounts a external volume OR Run the kubernetes e2e tests, where we have a pending reproducer for this https://github.com/GoogleCloudPlatform/kubernetes/pull/8254. The output we get from these tests: Expected error: <*errors.errorString | 0xc20825aef0>: { s: "pod pod-b4623a77-fb1a-11e4-8c06-ecf4bbc72674 terminated with failure: &{ExitCode:1 Signal:0 Reason: Message: StartedAt:2015-05-15 11:54:45 -0400 EDT FinishedAt:2015-05-15 11:54:45 -0400 EDT Contain erID:docker://bee5e651dd1ecd1346e53afe8bd293555dd2af55ca174caea6b3e860b4d72df4}", } pod pod-b4623a77-fb1a-11e4-8c06-ecf4bbc72674 terminated with failure: &{ExitCode:1 Signal:0 Reason: Message: StartedAt:2015-05-15 11:54:45 -0400 EDT FinishedAt:2015-05-15 11:54:45 -0400 EDT ContainerID:doc ker://bee5e651dd1ecd1346e53afe8bd293555dd2af55ca174caea6b3e860b4d72df4} not to have occurred
To add to this. The Kubernetes hostPath volume is the method to access local storage from a container on RHEL Atomic. It is really important that we get this working with SELinux enabled. As a simple unit test, I suspect that there is likely an issue with just docker run -v /some/content:/usr/share/nginx/html -d nginx The container should start but you'll get permission denied if you try and curl a webpage from the mounted directory.
docker run -v /some/content:/usr/share/nginx/html:Z -d nginx Would do the right thing and relabel /some/content. (Not merged upstream yet). If k8s supports RO/RW switch we should add Private/Shared switch.
Great unit test idea ! Steve is right... Thats a simple way to test - -bash-4.2# touch /tmp/index.html -bash-4.2# docker run -v /tmp:/usr/share/nginx/html -p 8001:80 -d nginx 418886f1c83877d40cd9cd25b2c01d13ac037d4be4c2e34f4bd57bdaf12b7b8a -bash-4.2# curl localhost:8001/index.html <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.7.12</center> </body> </html> -bash-4.2# curl localhost:8002/index.html curl: (7) Failed connect to localhost:8002; Connection refused -bash-4.2# curl localhost:8001/index.html <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.7.12</center> </body> </html> -bash-4.2# getenforce Enforcing -bash-4.2# setenforce 0 -bash-4.2# curl localhost:8001/index.html
Dan, it's not clear to me what the :Z opt on the bind does.
Also, all, FYI: https://github.com/GoogleCloudPlatform/kubernetes/issues/7925
TL;DR recap of my phone convo w/ Tim SC: There is no treatment for selinux wrt hostDir plugin. Eventually, the security work going on now will ensure that the selinux context is relabeled so that it plays nice with the selinux context a container is running in. For now, I would look at whether the policy changed around /var/lib/kubelet during 7.1.1 -> present.
Z (private) option causes docker daemon do: chcon -R -t svirt_sandbox_file_t -l MCSLEVEL /some/content Basically it labels the content under /some/content with a label that the process labels inside the container can read/write. Lower case z changes the label to something all containers can use. (Shared)
Looking at Atomic Host 7.1.2, docker already understands :z and :Z options in -v argument, so what we really need to do is to pass this option from Kubernetes. Still, how should it work? Random user posts new pod with say "hostDir: path: /etc". Kubernetes+docker should probably not relabel whole /etc on the host (!) to svirt_sandbox_file_t. There must be some policy. For the time being, I would emphasize in our documentation that the system admin is responsible for manually labeling the directories that should be shared with pods.
Right there should be some indication whether the direcory is Shared or Private to the POD. Doesn't K8s handle this someway with R/O versus R/W? THe z/Z patch prevents relabeling of /etc, /usr and /, but that is just a quick hack to stop the easiest stumbled upon problem. I think if the person building the app does not specify type of shareing, then we do nothing. And let SELinux block the access if necessary. There is no real easy way to do this. Either the user or the k8s has someway to know whether the content is private to the pod or not.
BTW this just got merged to docker-1.7 yesterday, So all docker should grow this ability now.
(In reply to Daniel Walsh from comment #11) > Right there should be some indication whether the direcory is Shared or > Private to the POD. Doesn't K8s handle this someway with R/O versus R/W? Yes, there is RO/RW flag, we need to add Shared/Private flag (:z/:Z). > THe z/Z patch prevents relabeling of /etc, /usr and /, but that is just a > quick hack to stop the easiest stumbled upon problem. I think if the person > building the app does not specify type of shareing, then we do nothing. And > let SELinux block the access if necessary. Ok, so all we need to do in Kubernetes is to add :z/:Z to hostDir mounts and Docker will re-label only the safe parts, right? Even when using upstream Docker?
When docker-1.7 ships, which is supposed to happen mid June. Safe Parts? It will relabel the Src directory if told to relabel it, but will fail if you pass in /etc, /usr or /. It would gladly relabel subdirs though. /usr/lib for example.
Fixed in docker-1.7 With :z and :Z support.