Description of problem: if selinux is set to enforcing, kubeadm is unable to init containers. Further dabugging show this is caused by selinux blocking access to keys and certificates to cert_t, as seen as by a quick look on audit.log: type=AVC msg=audit(1518750783.683:5323): avc: denied { open } for pid=27754 comm="kube-controller" path="/etc/kubernetes/pki/sa.key" dev="dm-0" ino=260121 scontext=system_u:system_r:container_t:s0:c803,c962 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1 using setenforce 0 is a solution, but not satisfying. Running the component as spc_t is better, and I wrote a patch for that. Version-Release number of selected component (if applicable): kubernetes-kubeadm-1.9.1-2.fc27.x86_64 How reproducible: each time Steps to Reproduce: 1. install FAH 2. install kubernetes-kubeadm (rpm-ostree install kubernetes-kubeadm && reboot) 3. type "kubeadm init" 4. wait 24h and see that it do not work. Start to despair and learn golang to fix. Actual results: kubeadm init block waiting on apiserver and controller to start. Expected results: the command resturn and things work (tm) Additional info: Until https://github.com/kubernetes/kubernetes/pull/49410 is merged, you also need "--ignore-preflight-errors=SystemVerification". The problem was diagnosed by Jason Brooks, and is caused by kubeadm starting containers as container_t domain by default, who has no access to various file. I created a patch for that, on https://github.com/mscherer/kubernetes/commit/87fe9669bf27759d61c06f09d645d40f240b6294 I tested it on Fedora Atomic Host 27, and at the moment trying to figure the CLA thing and what I am exactly signing, but in the mean time, I think we could carry the patch downstream.
We don't have any selinux policty provided by the Kubernetes itself. Switching to docker. Michael, what is your container runtime environment? Docker or runc?
A better solution would be to label your certs with a label that containers can read like container_share_t. This would run the containers in lock down mode but still allow them to read the cert data. Is this cert data going to be used by other confined processes or just containers?
To answer comment #2, that's likely using docker. I didn't touched to the default and kubeadm seems to talk to the docker socket, the process running is docker-current, etc. For the files, that's /etc/kubernetes/pki/sa.key and /etc/kubernetes/pki/apiserver.crt, and they are already labelled as cert_t. I suspect container_share_t might reduce the security for such files, and maybe break non containerized kubernetes installation. So far, I didn't see who use those files. I guess apiserver.crt is used only by the apiserver. No idea of sa.key, and I am also surprised to not see more AVC in the log, but I also didn't used the cluster yet.
Well the use of them for containerized installs is the error you are showing. container_share_t would allow any container to read them. But would futher confine kubernetes when run in a container.
We still have an issue w/ etcd, which needs to write to /var/lib/etcd. There was a fix in kubeadm that ran etcd as spc_t, but it got pulled out (https://github.com/kubernetes/kubernetes/pull/49328/files) because coreos doesn't have spc_t in its selinux policy, so kubeadm wouldn't run there at all. We could carry a patch like this just for fedora's kubeadm, and that wouldn't impact upstream. Regarding better solutions for this, it's worth pointing out that "oc cluster up" runs the origin master, which includes etcd and the apiserver, as spc_t.
For reference, this issue was discussed in the past it seems (I just rediscovered the proposed path) : https://github.com/kubernetes/kubeadm/issues/279 I am unsure on how the label cert_t is applied on /etc/kubernetes/pki, and trying with semanage fcontext didn't made kubeadm work (cause it still create thing with cert_t label), so I am a bit lost at the moment.
You get spc_t just by disabling SELinux in the container. Can Kubernetes pass in --security-opt label:disable
Seems I missed the last comment. I did implement more verification on the branch: https://github.com/mscherer/kubernetes/tree/fix_selinux_kubeadm (by checking that spc_t is working). I will make one with label:disable as well, as I guess there is less coupling with the selinux policy.
So, i tested my patch, and it work with and without selinux. Now, deploying a container for network fail. For example, if I want to deploy flannel with: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml It fail first with: type=AVC msg=audit(1519433567.749:1290): avc: denied { write } for pid=18598 comm="cp" name="net.d" dev="dm-0" ino=262099 scontext=system_u:system_r:container_t:s0:c535,c546 tcontext=system_u:object_r:container_share_t:s0 tclass=dir permissive=0 that I bypassed with: chcon -t container_t /etc/cni/net.d/ (since container_t can only write there) Now, that's blocked by: type=AVC msg=audit(1519433870.351:1413): avc: denied { mounton } for pid=20221 comm="runc:[2:INIT]" path="/var/lib/docker/overlay2/0356bfb61449296a783dbd381bff84c3e90ba12a3e92278c8bf7f3b73ba36874/merged/etc/cni/net.d" dev="dm-0" ino=262099 scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:container_t:s0 tclass=dir permissive=0 I guess the solution is to run flannel also as spc_t, but that mean we would have to fix every CNI pod to work with selinux, that's not really optimal.
contaner_t is a process type. container_file_t would be the label you want to set. chcon -t container_file_t /etc/cni/net.d/
Indeed, with the right label, it work. So should we get /etc/cni/net.d/ tagged as container_file_t by default ? And what about the patch, since --security-opt label:disable seems to be a docker level options, and I didn't found (yet) how/if this can be expressed on kubernetes level (since the pod are started by dropping file in /etc/kubernetes/manifests/ ) ?
I think on kubernetes you could specify to run as spc_t which would be equivalent. container_file_t allows all confined containers to read/write that content. If you want confined containers to only read it, you could set it to container_share_t.
This message is a reminder that Fedora 27 is nearing its end of life. On 2018-Nov-30 Fedora will stop maintaining and issuing updates for Fedora 27. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '27'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 27 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.