Description of problem: Apologies I don't 100% know where to file this. Note that this is using the upstream kubernetes rpms with our Docker 1.12 packages. Roughly this surfaced when following the kubeadm docs but reproducer steps are spelled out below: https://kubernetes.io/docs/getting-started-guides/kubeadm/ Kubeadm creates an etcd pod in kubernetes with the spc_t container type: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/master/manifests.go#L109 On RHEL 7.3 currently, this etcd pod will fail if you have selinux enforcing: type=AVC msg=audit(1484249804.933:1300): avc: denied { write } for pid=5817 comm="etcd" name="etcd" dev="dm-0" ino=1048971 scontext=system_u:system_r:svirt_lxc_net_t:s0:c247,c296 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir It's running as svirt_lvc_net not spc_t. Switching to permissive so the etcd container will run, and inspecting it, we see: "SecurityOpt": [ "seccomp=unconfined", "label:type:spc_t", "label=user:system_u", "label=role:system_r", "label=type:svirt_lxc_net_t", "label=level:s0:c88,c651" ], Per Dan Walsh, Docker switched from ":" to "=" for this syntax so it appears the security opt may not be correct here. Version-Release number of selected component (if applicable): (root@n3 ~) $ rpm -qa | grep kube kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 kubelet-1.5.1-0.x86_64 kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64 kubectl-1.5.1-0.x86_64 (root@n3 ~) $ rpm -qa | grep docker python-docker-py-1.10.6-1.el7.noarch python-docker-pycreds-1.10.6-1.el7.noarch docker-client-1.12.5-14.el7.x86_64 docker-logrotate-1.12.5-14.el7.x86_64 docker-common-1.12.5-14.el7.x86_64 docker-rhel-push-plugin-1.12.5-14.el7.x86_64 docker-lvm-plugin-1.12.5-14.el7.x86_64 docker-1.12.5-14.el7.x86_64 docker-novolume-plugin-1.12.5-14.el7.x86_64 (root@n3 ~) $ rpm -qa | grep selinux container-selinux-1.12.5-14.el7.x86_64 selinux-policy-targeted-3.13.1-102.el7_3.7.noarch libselinux-2.5-6.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 selinux-policy-3.13.1-102.el7_3.7.noarch libselinux-python-2.5-6.el7.x86_64 How reproducible: 100% Steps to Reproduce: Install RHEL 7.3 with Docker 1.12. cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF yum install -y docker kubelet kubeadm kubectl kubernetes-cni setenforce 0 systemctl enable docker && systemctl start docker systemctl enable kubelet && systemctl start kubelet kubeadm init Actual results: (root@n3 ~) $ docker ps | grep etcd ff275f8f97db gcr.io/google_containers/etcd-amd64@sha256:b7b54201ba7ae22e1b7993d86d90615646a736a23abd8561f6012bb0e3dcc075 "etcd --listen-client" About an hour ago Up About an hour k8s_etcd.a55f99d6_etcd-n3.aos.example.com_kube-system_978d1f0ddcff3f7fb13ef31f88895050_d5c6686e 89ccb5d6c328 gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 "/pause" About an hour ago Up About an hour k8s_POD.9959e668_etcd-n3.aos.example.com_kube-system_978d1f0ddcff3f7fb13ef31f88895050_54c1f777 (root@n3 ~) $ docker inspect ff275f8f97db (root@n3 ~) $ cat /var/log/audit/audit.log | grep denied type=AVC msg=audit(1484320607.358:6936): avc: denied { write } for pid=30786 comm="etcd" name="snap" dev="dm-0" ino=1051502 scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1484320607.358:6936): avc: denied { add_name } for pid=30786 comm="etcd" name="0000000000000003-0000000000002711.snap" scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1484320607.358:6936): avc: denied { create } for pid=30786 comm="etcd" name="0000000000000003-0000000000002711.snap" scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=file type=AVC msg=audit(1484320607.358:6936): avc: denied { write open } for pid=30786 comm="etcd" path="/var/lib/etcd/member/snap/0000000000000003-0000000000002711.snap" dev="dm-0" ino=1051506 scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=file Expected results: No denials, container running as spc_t. Additional info:
https://github.com/eparis/kubernetes/commit/ddd43b1f4345eb7c80a62ed0bef49dda2645cf4f Seems like it would help the : vs = thing, but the real problem is how we got the duplicate. So I'm not going to try to fix this. Just something for pmorie to think/know about.
Issue in upstream: https://github.com/kubernetes/kubernetes/issues/37807
@Eric Paris Could you help look at another selinux issue too? thanks https://github.com/kubernetes/kubernetes/issues/37809
We can't change the separator wholesale without losing compatibility with older dockers. There is already code in kubelet that deals with the separator differences based on the docker version, but it looks like it is applied to the security opt before the SELinux options are set by the security context provider. I'll make a change to apply the separator code later, which I believe should fix it.
Created https://github.com/kubernetes/kubernetes/pull/40179 to fix this
ORIGIN PR: https://github.com/openshift/origin/pull/12831
I'm using docker-1.10.3-45.gite03ddb8.fc23.x86_64, and https://github.com/openshift/origin/pull/12831 means I can't run pods any more: Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Invalid --security-opt: \"label=level:s0:c8,c2\"" Is this intended?
This has been merged into ocp and is in OCP v3.5.0.31 or newer.
Verify on openshift v3.5.0.32-1+4f84c83 [root@ip-172-18-12-128 ~]# docker version Client: Version: 1.12.5 API version: 1.24 Package version: docker-common-1.12.5-14.el7.x86_64 Go version: go1.7.4 Git commit: 047e51b/1.12.5 Built: Wed Jan 11 17:53:20 2017 OS/Arch: linux/amd64 Server: Version: 1.12.5 API version: 1.24 Package version: docker-common-1.12.5-14.el7.x86_64 Go version: go1.7.4 Git commit: 047e51b/1.12.5 Built: Wed Jan 11 17:53:20 2017 OS/Arch: linux/amd64 //Steps 1. Add privileged to user oadm policy add-scc-to-user privileged dma 2. Create a pod with selinux oc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/securityContext/pod-selinux.yaml 3. Check selinux options in container oc get pod hello-pod -o json docker inspect $container-id [root@dhcp-128-7 Desktop]# oc get pod hello-pod -o json ..... spec.containers.securityContext "securityContext": { "privileged": false, "seLinuxOptions": { "level": "s0:c25,c968", "role": "unconfined_r", "user": "unconfined_u" } }, ...... spec.securityContext "securityContext": { "seLinuxOptions": { "level": "s0:c25,c968", "role": "unconfined_r", "user": "unconfined_u" } }, ...... //no node inspect container check detail info docker inspect 388ae9a17c9f "SecurityOpt": [ "seccomp=unconfined", "label=user:unconfined_u", "label=role:unconfined_r", "label=level:s0:c25,c968", "label=user:unconfined_u", "label=role:unconfined_r", "label=type:svirt_lxc_net_t", "label=level:s0:c25,c968" ], //note theres is still small issue When `docker inspect $container-id`, label shows twice,this will be fiexed by https://github.com/docker/docker/pull/30652 info https://github.com/kubernetes/kubeadm/issues/107#issuecomment-278979359 This can't affect the function, I'll make the bug verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884