Bug 1358819 - docker is prevented from running container by selinux
Summary: docker is prevented from running container by selinux
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
Tomas Capek
URL:
Whiteboard:
Keywords: Extras
: 1373430 (view as bug list)
Depends On:
Blocks: 1366991 1375561 1400333
TreeView+ depends on / blocked
 
Reported: 2016-07-21 14:33 UTC by Marek Haicman
Modified: 2019-03-06 02:36 UTC (History)
25 users (show)

(edit)
SELinux prevents Docker from running a container

Due to a missing label for the `/usr/bin/docker-current` binary file, Docker is prevented from running a container by SELinux.
Clone Of:
: 1400333 (view as bug list)
(edit)
Last Closed: 2016-11-04 09:08:56 UTC


Attachments (Terms of Use)
AVCs over whole session [multiple runs] (57.57 KB, text/x-vhdl)
2016-07-21 14:33 UTC, Marek Haicman
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2634 normal SHIPPED_LIVE Moderate: docker security and bug fix update 2016-11-03 20:51:48 UTC

Description Marek Haicman 2016-07-21 14:33:27 UTC
Created attachment 1182540 [details]
AVCs over whole session [multiple runs]

Description of problem:
:: [  BEGIN   ] :: Running 'docker run --name "container_rhel6" -d "rhel6" /bin/sleep 1d'
ea2d6379eddb4a53819f4f9407d060320b0ba9e1cb18296627e18939b93cfc24
docker: Error response from daemon: Cannot start container ea2d6379eddb4a53819f4f9407d060320b0ba9e1cb18296627e18939b93cfc24: [9] System error: exit status 1.
:: [   FAIL   ] :: Command 'docker run --name "container_rhel6" -d "rhel6" /bin/sleep 1d' (Expected 0, got 125)

with AVC raised on the occasion [for the rest, and for the run with setenforce 0, see attachment]:
type=USER_AVC msg=audit(07/21/2016 09:24:18.111:829) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=0)  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(07/21/2016 09:24:18.111:830) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=1)  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(07/21/2016 09:24:18.155:834) : pid=635 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { 0x2 } for msgtype=error error_name=org.freedesktop.DBus.Error.AccessDenied dest=:1.116 spid=23033 tpid=23031 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=(null)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=SYSCALL msg=audit(07/21/2016 09:24:18.154:833) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x7ffc4a41b490 a1=O_RDONLY|O_CLOEXEC a2=0x1b6 a3=0x24 items=0 ppid=1 pid=23033 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-machine exe=/usr/lib/systemd/systemd-machined subj=system_u:system_r:systemd_machined_t:s0 key=(null) 
type=AVC msg=audit(07/21/2016 09:24:18.154:833) : avc:  denied  { search } for  pid=23033 comm=systemd-machine name=23026 dev="proc" ino=76666 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dir 
----
type=USER_AVC msg=audit(07/21/2016 09:24:48.268:838) : pid=635 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { 0x2 } for msgtype=error error_name=org.freedesktop.machine1.NoSuchMachine dest=:1.118 spid=23044 tpid=23042 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=(null)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----


Here is initrc state just a few moments after the command finished:

[0 root@qeos-30 tmp.J8WQxsGFD8]# ps -efZ | grep initrc
system_u:system_r:initrc_t:s0   root     22840     1  0 09:23 ?        00:00:00 /bin/sh -c /usr/bin/docker-current daemon            --authorization-plugin=rhel-push-plugin            --exec-opt native.cgroupdriver=systemd            $OPTIONS            $DOCKER_STORAGE_OPTIONS            $DOCKER_NETWORK_OPTIONS            $ADD_REGISTRY            $BLOCK_REGISTRY            $INSECURE_REGISTRY            2>&1 | /usr/bin/forward-journald -tag docker
system_u:system_r:initrc_t:s0   root     22842 22840  7 09:23 ?        00:00:06 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enabled --log-driver=journald --add-registry registry.access.redhat.com
system_u:system_r:initrc_t:s0   root     22843 22840  0 09:23 ?        00:00:00 /usr/bin/forward-journald -tag docker
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 23077 23060  0 09:25 pts/0 00:00:00 grep --color=auto initrc


Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-89.el7.noarch
selinux-policy-targeted-3.13.1-89.el7.noarch
docker-selinux-1.10.3-44.el7.x86_64
docker-1.10.3-44.el7.x86_64

Comment 3 Lukas Vrabec 2016-07-27 16:12:53 UTC
Looks like missing label for /usr/bin/docker-current binary file.
Moving to docker component.

Comment 4 Daniel Walsh 2016-07-27 16:57:50 UTC
Lokesh do we have the latest policy updates?

Comment 6 Lukas Vrabec 2016-08-18 14:05:55 UTC
Could somebody please re-test it with the latest selinux-policy rpm version from brew?

Comment 7 Daniel Walsh 2016-08-19 12:29:47 UTC
Fixed in latest selinux-policy package.

Comment 8 Matus Marhefka 2016-08-23 10:24:01 UTC
Still occuring for me.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 Beta (Maipo)

# rpm -q docker-selinux
docker-selinux-1.10.3-46.el7.10.x86_64
# rpm -q selinux-policy
selinux-policy-3.13.1-94.el7.noarch
# rpm -q docker
docker-1.10.3-46.el7.10.x86_64


# docker run --rm -it rhel7 bash
docker: Error response from daemon: Cannot start container d53b70c28f342c4d82c5b15ac73bb166ad69f7c9c26fef464b0da35350317b78: [9] System error: exit status 1.

# ausearch -m avc --start recent
----
type=SYSCALL msg=audit(1471947404.352:413): arch=c000003e syscall=2 success=no exit=-13 a0=7ffc29625950 a1=80000 a2=1b6 a3=24 items=0 ppid=1 pid=20072 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-machine" exe="/usr/lib/systemd/systemd-machined" subj=system_u:system_r:systemd_machined_t:s0 key=(null)
type=AVC msg=audit(1471947404.352:413): avc:  denied  { search } for  pid=20072 comm="systemd-machine" name="20064" dev="proc" ino=55611 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dir

# rpm -qf /usr/lib/systemd/systemd-machined
systemd-219-26.el7.x86_64

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

It works though when SELinux is disabled:
# setenforce 0
# docker run --rm -it rhel7 bash
[root@0611ac10610c /]#

Probably an issue in the selinux-policy?

Comment 9 Daniel Walsh 2016-08-23 10:40:49 UTC
ps -eZ | grep unconfined_service_t

See if docker is running with 

ls -lZ /usr/bin/docker

Comment 10 Matus Marhefka 2016-08-23 10:57:11 UTC
# ps -eZ | grep unconfined_service_t
system_u:system_r:unconfined_service_t:s0 1576 ? 00:00:00 rhel-push-plugi
system_u:system_r:unconfined_service_t:s0 19576 ? 00:00:01 docker-current
# ls -lZ /usr/bin/docker
-rwxr-xr-x. root root system_u:object_r:docker_exec_t:s0 /usr/bin/docker

Comment 11 Daniel Walsh 2016-08-23 11:01:33 UTC
ls -lZ /usr/bin/docker-current

Comment 12 Matus Marhefka 2016-08-23 11:03:06 UTC
# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current

Comment 13 Daniel Walsh 2016-08-24 11:59:18 UTC
So that is the problem.

restorecon -v /usr/bin/docker-current
matchpatchcon /usr/bin/docker-current

If this does not change the label to docker_exec_t, then we have a bug.

dnf reinstall docker-selinux

Then check the commands above.  If it is still bin_t, then we have a problem.

Comment 14 Marek Haicman 2016-08-24 12:34:20 UTC
I have just checked it and there is still bin_t


[0 root@qeos-37 atomic-scan]# rpm -qa docker-selinux selinux-policy
docker-selinux-1.10.3-47.el7.x86_64
selinux-policy-3.13.1-95.el7.noarch

[0 root@qeos-37 atomic-scan]# restorecon -v /usr/bin/docker-current
[0 root@qeos-37 atomic-scan]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current system_u:object_r:bin_t:s0

[0 root@qeos-37 atomic-scan]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current

Comment 15 Matus Marhefka 2016-08-24 12:42:14 UTC
I can confirm, /usr/bin/docker-current is still bin_t.

Comment 16 Lukas Vrabec 2016-08-24 13:16:30 UTC
Moving to ASSIGNED. We need to add proper label to /usr/bin/docker-current binary.

Comment 17 Daniel Walsh 2016-08-24 13:18:28 UTC
This was fixed a long time ago in git.

commit 032bcda7b1eb6d9d75d3c0ce64d9d35cdb9c7b85
Author: Dan Walsh <dwalsh@redhat.com>
Date:   Fri Apr 29 08:22:01 2016 -0400

    Fix labeling of docker executables and kubelet data

diff --git a/docker.fc b/docker.fc
index 4a4beb5..2cdbc27 100644
--- a/docker.fc
+++ b/docker.fc
@@ -1,7 +1,7 @@
 /root/\.docker gen_context(system_u:object_r:docker_home_t,s0)
 
-/usr/bin/docker.*                      --      gen_context(system_u:object_r:docker_exec_t,s0)
-/usr/libexec/docker/docker.*           --      gen_context(system_u:object_r:docker_exec_t,s0)
+/usr/libexec/docker/docker.*   --      gen_context(system_u:object_r:docker_exec_t,s0)
+/usr/bin/docker.*              --      gen_context(system_u:object_r:docker_exec_t,s0)
 /usr/bin/docker-latest                 --      gen_context(system_u:object_r:docker_exec_t,s0)

Comment 18 Lukas Vrabec 2016-08-25 11:28:52 UTC
Fix is in docker-selinux repo. 

https://github.com/projectatomic/docker-selinux/commit/59a8d6b93b6b1475b88db69788760bbaed2a0516

Lokesh, could you create new build for docker-selinux?

Comment 19 Eduard Benes 2016-08-25 11:35:36 UTC
(In reply to Lukas Vrabec from comment #18)
> Fix is in docker-selinux repo. 
> 
> https://github.com/projectatomic/docker-selinux/commit/
> 59a8d6b93b6b1475b88db69788760bbaed2a0516
> 
> Lokesh, could you create new build for docker-selinux?

What a speed! Perfect, Thanks!

Matus, Marek, please give it a try ... once it is built?

Comment 21 Daniel Walsh 2016-09-07 12:47:01 UTC
*** Bug 1373430 has been marked as a duplicate of this bug. ***

Comment 22 Marek Haicman 2016-09-14 11:04:23 UTC
It is fixed in latest version.

[0 root@qeos-186 scan-all]# rpm -qa docker-selinux selinux-policy
docker-selinux-1.10.3-53.el7.x86_64
selinux-policy-3.13.1-97.el7.noarch

[0 root@qeos-186 scan-all]# restorecon -v /usr/bin/docker-current
[0 root@qeos-186 scan-all]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current system_u:object_r:docker_exec_t:s0

[0 root@qeos-186 scan-all]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:docker_exec_t:s0 /usr/bin/docker-current

Comment 25 errata-xmlrpc 2016-11-04 09:08:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2634.html

Comment 30 Marko Myllynen 2017-01-12 08:49:08 UTC
I'm seeing OCP 3.3 installation on RHEL 7.3 with packages of 2017-01-12 failing due to this.

For some reason I can't reopen this BZ, this definitely should be reopened.

[root@infra01 ~]# yum reinstall docker-selinux
Loaded plugins: priorities, product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package docker-selinux.x86_64 0:1.10.3-57.el7 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch       Version           Repository                Size
================================================================================
Reinstalling:
 docker-selinux     x86_64     1.10.3-57.el7     rhel-7-extras-rpms        79 k

Transaction Summary
================================================================================
Reinstall  1 Package

Total download size: 79 k
Installed size: 27 k
Is this ok [y/d/N]: y
Downloading packages:
docker-selinux-1.10.3-57.el7.x86_64.rpm                    |  79 kB   00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-selinux-1.10.3-57.el7.x86_64                          1/1 
Re-declaration of type docker_t
Failed to create node
Bad type declaration at /etc/selinux/targeted/tmp/modules/200/docker/cil:1
/usr/sbin/semodule:  Failed!
  Verifying  : docker-selinux-1.10.3-57.el7.x86_64                          1/1 

Installed:
  docker-selinux.x86_64 0:1.10.3-57.el7                                         

Complete!
[root@infra01 ~]# rpm -q docker-selinux selinux-policy
docker-selinux-1.10.3-57.el7.x86_64
selinux-policy-3.13.1-102.el7_3.7.noarch
[root@infra01 ~]# restorecon -v /usr/bin/docker-current
[root@infra01 ~]# matchpathcon /usr/bin/docker-current 
/usr/bin/docker-current	system_u:object_r:bin_t:s0
[root@infra01 ~]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
[root@infra01 ~]# docker run -it test
docker: Error response from daemon: Cannot start container 8f3718c4e282e36ce234749adbea8c2ed2054267a8775f663019cd84cfd0ff68: [9] System error: exit status 1.
[root@infra01 ~]# 

Thanks.

Comment 31 Daniel Walsh 2017-01-12 13:34:37 UTC
Your docker package is out of date for this release I believe.

Comment 32 Marko Myllynen 2017-01-12 13:39:08 UTC
(In reply to Daniel Walsh from comment #31)
> Your docker package is out of date for this release I believe.

docker-1.10.3-59.el7.x86_64 is the latest available on public channels:

https://rhn.redhat.com/errata/RHBA-2016-2859.html

Thanks.

Comment 33 Daniel Walsh 2017-01-12 14:02:11 UTC
Weird, I would figure everyone would be having this issue.

semodule -d docker
yum reinstall docker-selinux

Should fix this problem.

You don't have docker-selinux installed?

Comment 34 Marko Myllynen 2017-01-12 16:22:42 UTC
(In reply to Daniel Walsh from comment #33)
> Weird, I would figure everyone would be having this issue.
> 
> semodule -d docker
> yum reinstall docker-selinux
> 
> Should fix this problem.
> 
> You don't have docker-selinux installed?

It was (see the paste) but you're right in that sense that due to automation docker and docker-selinux were installed at different stages, I think docker was started at some point without docker-selinux being in place, and perhaps that caused local issues, now starting with fresh VMs and making sure docker and docker-selinux always get installed at the same time I don't see the above issue anymore. However, as part of OCP 3.3.1.7 installation I'm now seeing this to block container creation:

type=SYSCALL msg=audit(1484237991.759:3684): arch=c000003e syscall=56 success=yes exit=27447 a0=6c020011 a1=0 a2=0 a3=0 items=0 ppid=1 pid=14380 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="docker-current" exe="/usr/bin/docker-current" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1484236755.707:2173): avc:  denied  { transition } for  pid=17014 comm="exe" path="/usr/bin/pod" dev="dm-4" ino=2104360 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c6 tclass=process

I will try to get more details around this in the coming days and will a new BZ if needed.

Thanks.

Comment 35 Daniel Walsh 2017-01-12 17:23:13 UTC
Sorry that should have said docker-engine-selinux.

This means that docker-current does not have the docker_exec_t label on it, which should have been set in the docker-selinux package.

ls -lZ /usr/bin/docker-current
If this is labeled docker_exec_t, then restart the docker service and docker should run with the correct label.

Comment 36 Milos Malik 2017-01-12 17:26:36 UTC
I thought that docker-selinux RPM was replaced by container-selinux RPM.

Comment 37 Daniel Walsh 2017-01-12 17:34:25 UTC
Yes it is, or will be.  Just covering the transition.

Comment 38 Marko Myllynen 2017-01-12 18:38:05 UTC
(In reply to Daniel Walsh from comment #35)
> Sorry that should have said docker-engine-selinux.

There no such package available for RHEL 7.

> This means that docker-current does not have the docker_exec_t label on it,
> which should have been set in the docker-selinux package.
> 
> ls -lZ /usr/bin/docker-current
> If this is labeled docker_exec_t, then restart the docker service and docker
> should run with the correct label.

[root@infra02 ~]# rpm -q selinux-policy docker-selinux container-selinux
selinux-policy-3.13.1-102.el7_3.7.noarch
docker-selinux-1.10.3-57.el7.x86_64
container-selinux-1.10.3-59.el7.x86_64
[root@infra02 ~]# restorecon -v /usr/bin/docker-current
[root@infra02 ~]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current	system_u:object_r:bin_t:s0
[root@infra02 ~]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
[root@infra02 ~]# 

Thanks.

Comment 39 Daniel Walsh 2017-01-12 19:08:49 UTC
That is the wrong label. Which is why it is running with the wrong context.

docker-selinux and container-selinux are not supposed to be installed at the same time.  cotnainer-selinux should replace docker-selinux.

Remove the docker-selinux package and then reinstall container-selinux.

Comment 40 Marko Myllynen 2017-01-12 20:20:22 UTC
(In reply to Daniel Walsh from comment #39)
> That is the wrong label. Which is why it is running with the wrong context.
> 
> docker-selinux and container-selinux are not supposed to be installed at the
> same time.  cotnainer-selinux should replace docker-selinux.
> 
> Remove the docker-selinux package and then reinstall container-selinux.

Thanks, finally got it - perhaps consider adding Conflicts or Obsoletes/Provides for these packages on RPM level?

I'm still seeing the earlier issue when e.g. docker-registry-N-deploy / router-N-deploy pods are in ContainerCreating state during OCP deployment, I'll file a separate BZ about this.

[root@infra01 ~]# rpm -q selinux-policy docker docker-selinux container-selinux
selinux-policy-3.13.1-102.el7_3.7.noarch
docker-1.10.3-59.el7.x86_64
package docker-selinux is not installed
container-selinux-1.10.3-59.el7.x86_64
[root@infra01 ~]# grep denied /var/log/audit/audit.log 
type=AVC msg=audit(1484251796.227:1917): avc:  denied  { transition } for  pid=15134 comm="exe" path="/usr/bin/pod" dev="dm-4" ino=2104360 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=process

Thanks.

Comment 41 Marko Myllynen 2017-01-12 20:29:42 UTC
For reference, the new BZ I filed is https://bugzilla.redhat.com/show_bug.cgi?id=1412803. Thanks.

Comment 42 Ryan Howe 2017-02-01 22:02:57 UTC
Do we have any progress on changing the line for the rpm scripts for docker-selinux-1.10

Making sure that we run restorecon on /usr/bin/docker*  and not just /usr/bin/docker

I understand that docker 1.12 uses a new package but OpenShift still support 1.10


Note You need to log in before you can comment on or make changes to this bug.