Bug 1413100

Summary: Issues with Docker, Kubernetes, and SELinux
Product: OpenShift Container Platform Reporter: Devan Goodwin <dgoodwin>
Component: NodeAssignee: Paul Morie <pmorie>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.5.0CC: amurdaca, aos-bugs, bbennett, decarr, eparis, jminter, jokerman, lsm5, mmccomas, tdawson
Target Milestone: ---Keywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 19:09:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Devan Goodwin 2017-01-13 15:48:44 UTC
Description of problem:

Apologies I don't 100% know where to file this. Note that this is using the upstream kubernetes rpms with our Docker 1.12 packages. Roughly this surfaced when following the kubeadm docs but reproducer steps are spelled out below:

https://kubernetes.io/docs/getting-started-guides/kubeadm/

Kubeadm creates an etcd pod in kubernetes with the spc_t container type:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/master/manifests.go#L109

On RHEL 7.3 currently, this etcd pod will fail if you have selinux enforcing:

type=AVC msg=audit(1484249804.933:1300): avc:  denied  { write } for  pid=5817 comm="etcd" name="etcd" dev="dm-0" ino=1048971 scontext=system_u:system_r:svirt_lxc_net_t:s0:c247,c296 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir

It's running as svirt_lvc_net not spc_t.

Switching to permissive so the etcd container will run, and inspecting it, we see:

            "SecurityOpt": [
                "seccomp=unconfined",
                "label:type:spc_t",
                "label=user:system_u",
                "label=role:system_r",
                "label=type:svirt_lxc_net_t",
                "label=level:s0:c88,c651"
            ],

Per Dan Walsh, Docker switched from ":" to "=" for this syntax so it appears the security opt may not be correct here.

Version-Release number of selected component (if applicable):

(root@n3 ~) $ rpm -qa | grep kube
kubernetes-cni-0.3.0.1-0.07a8a2.x86_64
kubelet-1.5.1-0.x86_64
kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64
kubectl-1.5.1-0.x86_64
(root@n3 ~) $ rpm -qa | grep docker
python-docker-py-1.10.6-1.el7.noarch
python-docker-pycreds-1.10.6-1.el7.noarch
docker-client-1.12.5-14.el7.x86_64
docker-logrotate-1.12.5-14.el7.x86_64
docker-common-1.12.5-14.el7.x86_64
docker-rhel-push-plugin-1.12.5-14.el7.x86_64
docker-lvm-plugin-1.12.5-14.el7.x86_64
docker-1.12.5-14.el7.x86_64
docker-novolume-plugin-1.12.5-14.el7.x86_64
(root@n3 ~) $ rpm -qa | grep selinux
container-selinux-1.12.5-14.el7.x86_64
selinux-policy-targeted-3.13.1-102.el7_3.7.noarch
libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-3.13.1-102.el7_3.7.noarch
libselinux-python-2.5-6.el7.x86_64



How reproducible:

100%


Steps to Reproduce:

Install RHEL 7.3 with Docker 1.12.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

yum install -y docker kubelet kubeadm kubectl kubernetes-cni
setenforce 0
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
kubeadm init

Actual results:

(root@n3 ~) $ docker ps | grep etcd
ff275f8f97db        gcr.io/google_containers/etcd-amd64@sha256:b7b54201ba7ae22e1b7993d86d90615646a736a23abd8561f6012bb0e3dcc075                      "etcd --listen-client"   About an hour ago   Up About an hour                        k8s_etcd.a55f99d6_etcd-n3.aos.example.com_kube-system_978d1f0ddcff3f7fb13ef31f88895050_d5c6686e
89ccb5d6c328        gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516                     "/pause"                 About an hour ago   Up About an hour                        k8s_POD.9959e668_etcd-n3.aos.example.com_kube-system_978d1f0ddcff3f7fb13ef31f88895050_54c1f777
(root@n3 ~) $ docker inspect ff275f8f97db
(root@n3 ~) $ cat /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1484320607.358:6936): avc:  denied  { write } for  pid=30786 comm="etcd" name="snap" dev="dm-0" ino=1051502 scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1484320607.358:6936): avc:  denied  { add_name } for  pid=30786 comm="etcd" name="0000000000000003-0000000000002711.snap" scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1484320607.358:6936): avc:  denied  { create } for  pid=30786 comm="etcd" name="0000000000000003-0000000000002711.snap" scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=file
type=AVC msg=audit(1484320607.358:6936): avc:  denied  { write open } for  pid=30786 comm="etcd" path="/var/lib/etcd/member/snap/0000000000000003-0000000000002711.snap" dev="dm-0" ino=1051506 scontext=system_u:system_r:svirt_lxc_net_t:s0:c88,c651 tcontext=system_u:object_r:var_lib_t:s0 tclass=file



Expected results:

No denials, container running as spc_t.


Additional info:

Comment 1 Eric Paris 2017-01-13 17:04:59 UTC
https://github.com/eparis/kubernetes/commit/ddd43b1f4345eb7c80a62ed0bef49dda2645cf4f

Seems like it would help the : vs = thing, but the real problem is how we got the duplicate. So I'm not going to try to fix this. Just something for pmorie to think/know about.

Comment 2 DeShuai Ma 2017-01-16 01:51:59 UTC
Issue in upstream: https://github.com/kubernetes/kubernetes/issues/37807

Comment 3 DeShuai Ma 2017-01-16 08:48:14 UTC
@Eric Paris Could you help look at another selinux issue too? thanks
https://github.com/kubernetes/kubernetes/issues/37809

Comment 4 Paul Morie 2017-01-17 20:35:07 UTC
We can't change the separator wholesale without losing compatibility with older dockers.

There is already code in kubelet that deals with the separator differences based on the docker version, but it looks like it is applied to the security opt before the SELinux options are set by the security context provider.

I'll make a change to apply the separator code later, which I believe should fix it.

Comment 5 Paul Morie 2017-01-19 22:05:22 UTC
Created https://github.com/kubernetes/kubernetes/pull/40179 to fix this

Comment 7 Derek Carr 2017-02-06 19:07:33 UTC
ORIGIN PR:
https://github.com/openshift/origin/pull/12831

Comment 8 Jim Minter 2017-02-16 13:26:58 UTC
I'm using docker-1.10.3-45.gite03ddb8.fc23.x86_64, and https://github.com/openshift/origin/pull/12831 means I can't run pods any more:

Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: Invalid --security-opt: \"label=level:s0:c8,c2\""

Is this intended?

Comment 9 Troy Dawson 2017-02-17 20:37:00 UTC
This has been merged into ocp and is in OCP v3.5.0.31 or newer.

Comment 11 DeShuai Ma 2017-02-21 06:46:19 UTC
Verify on openshift v3.5.0.32-1+4f84c83
[root@ip-172-18-12-128 ~]# docker version
Client:
 Version:         1.12.5
 API version:     1.24
 Package version: docker-common-1.12.5-14.el7.x86_64
 Go version:      go1.7.4
 Git commit:      047e51b/1.12.5
 Built:           Wed Jan 11 17:53:20 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.5
 API version:     1.24
 Package version: docker-common-1.12.5-14.el7.x86_64
 Go version:      go1.7.4
 Git commit:      047e51b/1.12.5
 Built:           Wed Jan 11 17:53:20 2017
 OS/Arch:         linux/amd64

//Steps
1. Add privileged to user
oadm policy add-scc-to-user privileged dma

2. Create a pod with selinux
oc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/securityContext/pod-selinux.yaml 

3. Check selinux options in container
oc get pod hello-pod -o json
docker inspect $container-id

[root@dhcp-128-7 Desktop]# oc get pod hello-pod -o json
.....
spec.containers.securityContext
                "securityContext": {
                    "privileged": false,
                    "seLinuxOptions": {
                        "level": "s0:c25,c968",
                        "role": "unconfined_r",
                        "user": "unconfined_u"
                    }
                },
......
spec.securityContext
        "securityContext": {
            "seLinuxOptions": {
                "level": "s0:c25,c968",
                "role": "unconfined_r",
                "user": "unconfined_u"
            }
        },
......
//no node inspect container check detail info
docker inspect 388ae9a17c9f
            "SecurityOpt": [
                "seccomp=unconfined",
                "label=user:unconfined_u",
                "label=role:unconfined_r",
                "label=level:s0:c25,c968",
                "label=user:unconfined_u",
                "label=role:unconfined_r",
                "label=type:svirt_lxc_net_t",
                "label=level:s0:c25,c968"
            ],

//note theres is still small issue
When `docker inspect $container-id`, label shows twice,this will be fiexed by https://github.com/docker/docker/pull/30652
info https://github.com/kubernetes/kubeadm/issues/107#issuecomment-278979359
This can't affect the function, I'll make the bug verified.

Comment 13 errata-xmlrpc 2017-04-12 19:09:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0884