RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1358819 - docker is prevented from running container by selinux
Summary: docker is prevented from running container by selinux
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
Tomas Capek
URL:
Whiteboard:
: 1373430 (view as bug list)
Depends On:
Blocks: 1366991 1375561 1400333
TreeView+ depends on / blocked
 
Reported: 2016-07-21 14:33 UTC by Marek Haicman
Modified: 2020-03-11 15:11 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
SELinux prevents Docker from running a container Due to a missing label for the `/usr/bin/docker-current` binary file, Docker is prevented from running a container by SELinux.
Clone Of:
: 1400333 (view as bug list)
Environment:
Last Closed: 2016-11-04 09:08:56 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
AVCs over whole session [multiple runs] (57.57 KB, text/x-vhdl)
2016-07-21 14:33 UTC, Marek Haicman 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2634 0 normal SHIPPED_LIVE Moderate: docker security and bug fix update 2016-11-03 20:51:48 UTC

Description Marek Haicman 2016-07-21 14:33:27 UTC 
Created attachment 1182540 [details]
AVCs over whole session [multiple runs]

Description of problem:
:: [  BEGIN   ] :: Running 'docker run --name "container_rhel6" -d "rhel6" /bin/sleep 1d'
ea2d6379eddb4a53819f4f9407d060320b0ba9e1cb18296627e18939b93cfc24
docker: Error response from daemon: Cannot start container ea2d6379eddb4a53819f4f9407d060320b0ba9e1cb18296627e18939b93cfc24: [9] System error: exit status 1.
:: [   FAIL   ] :: Command 'docker run --name "container_rhel6" -d "rhel6" /bin/sleep 1d' (Expected 0, got 125)

with AVC raised on the occasion [for the rest, and for the run with setenforce 0, see attachment]:
type=USER_AVC msg=audit(07/21/2016 09:24:18.111:829) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=0)  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(07/21/2016 09:24:18.111:830) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=1)  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' 
----
type=USER_AVC msg=audit(07/21/2016 09:24:18.155:834) : pid=635 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { 0x2 } for msgtype=error error_name=org.freedesktop.DBus.Error.AccessDenied dest=:1.116 spid=23033 tpid=23031 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=(null)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----
type=SYSCALL msg=audit(07/21/2016 09:24:18.154:833) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x7ffc4a41b490 a1=O_RDONLY|O_CLOEXEC a2=0x1b6 a3=0x24 items=0 ppid=1 pid=23033 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-machine exe=/usr/lib/systemd/systemd-machined subj=system_u:system_r:systemd_machined_t:s0 key=(null) 
type=AVC msg=audit(07/21/2016 09:24:18.154:833) : avc:  denied  { search } for  pid=23033 comm=systemd-machine name=23026 dev="proc" ino=76666 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dir 
----
type=USER_AVC msg=audit(07/21/2016 09:24:48.268:838) : pid=635 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { 0x2 } for msgtype=error error_name=org.freedesktop.machine1.NoSuchMachine dest=:1.118 spid=23044 tpid=23042 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=(null)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?' 
----


Here is initrc state just a few moments after the command finished:

[0 root@qeos-30 tmp.J8WQxsGFD8]# ps -efZ | grep initrc
system_u:system_r:initrc_t:s0   root     22840     1  0 09:23 ?        00:00:00 /bin/sh -c /usr/bin/docker-current daemon            --authorization-plugin=rhel-push-plugin            --exec-opt native.cgroupdriver=systemd            $OPTIONS            $DOCKER_STORAGE_OPTIONS            $DOCKER_NETWORK_OPTIONS            $ADD_REGISTRY            $BLOCK_REGISTRY            $INSECURE_REGISTRY            2>&1 | /usr/bin/forward-journald -tag docker
system_u:system_r:initrc_t:s0   root     22842 22840  7 09:23 ?        00:00:06 /usr/bin/docker-current daemon --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --selinux-enabled --log-driver=journald --add-registry registry.access.redhat.com
system_u:system_r:initrc_t:s0   root     22843 22840  0 09:23 ?        00:00:00 /usr/bin/forward-journald -tag docker
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 23077 23060  0 09:25 pts/0 00:00:00 grep --color=auto initrc


Version-Release number of selected component (if applicable):
selinux-policy-3.13.1-89.el7.noarch
selinux-policy-targeted-3.13.1-89.el7.noarch
docker-selinux-1.10.3-44.el7.x86_64
docker-1.10.3-44.el7.x86_64
Comment 3 Lukas Vrabec 2016-07-27 16:12:53 UTC 
Looks like missing label for /usr/bin/docker-current binary file.
Moving to docker component.
Comment 4 Daniel Walsh 2016-07-27 16:57:50 UTC 
Lokesh do we have the latest policy updates?
Comment 6 Lukas Vrabec 2016-08-18 14:05:55 UTC 
Could somebody please re-test it with the latest selinux-policy rpm version from brew?
Comment 7 Daniel Walsh 2016-08-19 12:29:47 UTC 
Fixed in latest selinux-policy package.
Comment 8 Matus Marhefka 2016-08-23 10:24:01 UTC 
Still occuring for me.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 Beta (Maipo)

# rpm -q docker-selinux
docker-selinux-1.10.3-46.el7.10.x86_64
# rpm -q selinux-policy
selinux-policy-3.13.1-94.el7.noarch
# rpm -q docker
docker-1.10.3-46.el7.10.x86_64


# docker run --rm -it rhel7 bash
docker: Error response from daemon: Cannot start container d53b70c28f342c4d82c5b15ac73bb166ad69f7c9c26fef464b0da35350317b78: [9] System error: exit status 1.

# ausearch -m avc --start recent
----
type=SYSCALL msg=audit(1471947404.352:413): arch=c000003e syscall=2 success=no exit=-13 a0=7ffc29625950 a1=80000 a2=1b6 a3=24 items=0 ppid=1 pid=20072 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-machine" exe="/usr/lib/systemd/systemd-machined" subj=system_u:system_r:systemd_machined_t:s0 key=(null)
type=AVC msg=audit(1471947404.352:413): avc:  denied  { search } for  pid=20072 comm="systemd-machine" name="20064" dev="proc" ino=55611 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dir

# rpm -qf /usr/lib/systemd/systemd-machined
systemd-219-26.el7.x86_64

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

It works though when SELinux is disabled:
# setenforce 0
# docker run --rm -it rhel7 bash
[root@0611ac10610c /]#

Probably an issue in the selinux-policy?
Comment 9 Daniel Walsh 2016-08-23 10:40:49 UTC 
ps -eZ | grep unconfined_service_t

See if docker is running with 

ls -lZ /usr/bin/docker
Comment 10 Matus Marhefka 2016-08-23 10:57:11 UTC 
# ps -eZ | grep unconfined_service_t
system_u:system_r:unconfined_service_t:s0 1576 ? 00:00:00 rhel-push-plugi
system_u:system_r:unconfined_service_t:s0 19576 ? 00:00:01 docker-current
# ls -lZ /usr/bin/docker
-rwxr-xr-x. root root system_u:object_r:docker_exec_t:s0 /usr/bin/docker
Comment 11 Daniel Walsh 2016-08-23 11:01:33 UTC 
ls -lZ /usr/bin/docker-current
Comment 12 Matus Marhefka 2016-08-23 11:03:06 UTC 
# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
Comment 13 Daniel Walsh 2016-08-24 11:59:18 UTC 
So that is the problem.

restorecon -v /usr/bin/docker-current
matchpatchcon /usr/bin/docker-current

If this does not change the label to docker_exec_t, then we have a bug.

dnf reinstall docker-selinux

Then check the commands above.  If it is still bin_t, then we have a problem.
Comment 14 Marek Haicman 2016-08-24 12:34:20 UTC 
I have just checked it and there is still bin_t


[0 root@qeos-37 atomic-scan]# rpm -qa docker-selinux selinux-policy
docker-selinux-1.10.3-47.el7.x86_64
selinux-policy-3.13.1-95.el7.noarch

[0 root@qeos-37 atomic-scan]# restorecon -v /usr/bin/docker-current
[0 root@qeos-37 atomic-scan]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current system_u:object_r:bin_t:s0

[0 root@qeos-37 atomic-scan]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
Comment 15 Matus Marhefka 2016-08-24 12:42:14 UTC 
I can confirm, /usr/bin/docker-current is still bin_t.
Comment 16 Lukas Vrabec 2016-08-24 13:16:30 UTC 
Moving to ASSIGNED. We need to add proper label to /usr/bin/docker-current binary.
Comment 17 Daniel Walsh 2016-08-24 13:18:28 UTC 
This was fixed a long time ago in git.

commit 032bcda7b1eb6d9d75d3c0ce64d9d35cdb9c7b85
Author: Dan Walsh <dwalsh>
Date:   Fri Apr 29 08:22:01 2016 -0400

    Fix labeling of docker executables and kubelet data

diff --git a/docker.fc b/docker.fc
index 4a4beb5..2cdbc27 100644
--- a/docker.fc
+++ b/docker.fc
@@ -1,7 +1,7 @@
 /root/\.docker gen_context(system_u:object_r:docker_home_t,s0)
 
-/usr/bin/docker.*                      --      gen_context(system_u:object_r:docker_exec_t,s0)
-/usr/libexec/docker/docker.*           --      gen_context(system_u:object_r:docker_exec_t,s0)
+/usr/libexec/docker/docker.*   --      gen_context(system_u:object_r:docker_exec_t,s0)
+/usr/bin/docker.*              --      gen_context(system_u:object_r:docker_exec_t,s0)
 /usr/bin/docker-latest                 --      gen_context(system_u:object_r:docker_exec_t,s0)
Comment 18 Lukas Vrabec 2016-08-25 11:28:52 UTC 
Fix is in docker-selinux repo. 

https://github.com/projectatomic/docker-selinux/commit/59a8d6b93b6b1475b88db69788760bbaed2a0516

Lokesh, could you create new build for docker-selinux?
Comment 19 Eduard Benes 2016-08-25 11:35:36 UTC 
(In reply to Lukas Vrabec from comment #18)
> Fix is in docker-selinux repo. 
> 
> https://github.com/projectatomic/docker-selinux/commit/
> 59a8d6b93b6b1475b88db69788760bbaed2a0516
> 
> Lokesh, could you create new build for docker-selinux?

What a speed! Perfect, Thanks!

Matus, Marek, please give it a try ... once it is built?
Comment 21 Daniel Walsh 2016-09-07 12:47:01 UTC 
*** Bug 1373430 has been marked as a duplicate of this bug. ***
Comment 22 Marek Haicman 2016-09-14 11:04:23 UTC 
It is fixed in latest version.

[0 root@qeos-186 scan-all]# rpm -qa docker-selinux selinux-policy
docker-selinux-1.10.3-53.el7.x86_64
selinux-policy-3.13.1-97.el7.noarch

[0 root@qeos-186 scan-all]# restorecon -v /usr/bin/docker-current
[0 root@qeos-186 scan-all]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current system_u:object_r:docker_exec_t:s0

[0 root@qeos-186 scan-all]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:docker_exec_t:s0 /usr/bin/docker-current
Comment 25 errata-xmlrpc 2016-11-04 09:08:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2634.html
Comment 30 Marko Myllynen 2017-01-12 08:49:08 UTC 
I'm seeing OCP 3.3 installation on RHEL 7.3 with packages of 2017-01-12 failing due to this.

For some reason I can't reopen this BZ, this definitely should be reopened.

[root@infra01 ~]# yum reinstall docker-selinux
Loaded plugins: priorities, product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package docker-selinux.x86_64 0:1.10.3-57.el7 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch       Version           Repository                Size
================================================================================
Reinstalling:
 docker-selinux     x86_64     1.10.3-57.el7     rhel-7-extras-rpms        79 k

Transaction Summary
================================================================================
Reinstall  1 Package

Total download size: 79 k
Installed size: 27 k
Is this ok [y/d/N]: y
Downloading packages:
docker-selinux-1.10.3-57.el7.x86_64.rpm                    |  79 kB   00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-selinux-1.10.3-57.el7.x86_64                          1/1 
Re-declaration of type docker_t
Failed to create node
Bad type declaration at /etc/selinux/targeted/tmp/modules/200/docker/cil:1
/usr/sbin/semodule:  Failed!
  Verifying  : docker-selinux-1.10.3-57.el7.x86_64                          1/1 

Installed:
  docker-selinux.x86_64 0:1.10.3-57.el7                                         

Complete!
[root@infra01 ~]# rpm -q docker-selinux selinux-policy
docker-selinux-1.10.3-57.el7.x86_64
selinux-policy-3.13.1-102.el7_3.7.noarch
[root@infra01 ~]# restorecon -v /usr/bin/docker-current
[root@infra01 ~]# matchpathcon /usr/bin/docker-current 
/usr/bin/docker-current	system_u:object_r:bin_t:s0
[root@infra01 ~]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
[root@infra01 ~]# docker run -it test
docker: Error response from daemon: Cannot start container 8f3718c4e282e36ce234749adbea8c2ed2054267a8775f663019cd84cfd0ff68: [9] System error: exit status 1.
[root@infra01 ~]# 

Thanks.
Comment 31 Daniel Walsh 2017-01-12 13:34:37 UTC 
Your docker package is out of date for this release I believe.
Comment 32 Marko Myllynen 2017-01-12 13:39:08 UTC 
(In reply to Daniel Walsh from comment #31)
> Your docker package is out of date for this release I believe.

docker-1.10.3-59.el7.x86_64 is the latest available on public channels:

https://rhn.redhat.com/errata/RHBA-2016-2859.html

Thanks.
Comment 33 Daniel Walsh 2017-01-12 14:02:11 UTC 
Weird, I would figure everyone would be having this issue.

semodule -d docker
yum reinstall docker-selinux

Should fix this problem.

You don't have docker-selinux installed?
Comment 34 Marko Myllynen 2017-01-12 16:22:42 UTC 
(In reply to Daniel Walsh from comment #33)
> Weird, I would figure everyone would be having this issue.
> 
> semodule -d docker
> yum reinstall docker-selinux
> 
> Should fix this problem.
> 
> You don't have docker-selinux installed?

It was (see the paste) but you're right in that sense that due to automation docker and docker-selinux were installed at different stages, I think docker was started at some point without docker-selinux being in place, and perhaps that caused local issues, now starting with fresh VMs and making sure docker and docker-selinux always get installed at the same time I don't see the above issue anymore. However, as part of OCP 3.3.1.7 installation I'm now seeing this to block container creation:

type=SYSCALL msg=audit(1484237991.759:3684): arch=c000003e syscall=56 success=yes exit=27447 a0=6c020011 a1=0 a2=0 a3=0 items=0 ppid=1 pid=14380 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="docker-current" exe="/usr/bin/docker-current" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1484236755.707:2173): avc:  denied  { transition } for  pid=17014 comm="exe" path="/usr/bin/pod" dev="dm-4" ino=2104360 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c6 tclass=process

I will try to get more details around this in the coming days and will a new BZ if needed.

Thanks.
Comment 35 Daniel Walsh 2017-01-12 17:23:13 UTC 
Sorry that should have said docker-engine-selinux.

This means that docker-current does not have the docker_exec_t label on it, which should have been set in the docker-selinux package.

ls -lZ /usr/bin/docker-current
If this is labeled docker_exec_t, then restart the docker service and docker should run with the correct label.
Comment 36 Milos Malik 2017-01-12 17:26:36 UTC 
I thought that docker-selinux RPM was replaced by container-selinux RPM.
Comment 37 Daniel Walsh 2017-01-12 17:34:25 UTC 
Yes it is, or will be.  Just covering the transition.
Comment 38 Marko Myllynen 2017-01-12 18:38:05 UTC 
(In reply to Daniel Walsh from comment #35)
> Sorry that should have said docker-engine-selinux.

There no such package available for RHEL 7.

> This means that docker-current does not have the docker_exec_t label on it,
> which should have been set in the docker-selinux package.
> 
> ls -lZ /usr/bin/docker-current
> If this is labeled docker_exec_t, then restart the docker service and docker
> should run with the correct label.

[root@infra02 ~]# rpm -q selinux-policy docker-selinux container-selinux
selinux-policy-3.13.1-102.el7_3.7.noarch
docker-selinux-1.10.3-57.el7.x86_64
container-selinux-1.10.3-59.el7.x86_64
[root@infra02 ~]# restorecon -v /usr/bin/docker-current
[root@infra02 ~]# matchpathcon /usr/bin/docker-current
/usr/bin/docker-current	system_u:object_r:bin_t:s0
[root@infra02 ~]# ls -lZ /usr/bin/docker-current
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/docker-current
[root@infra02 ~]# 

Thanks.
Comment 39 Daniel Walsh 2017-01-12 19:08:49 UTC 
That is the wrong label. Which is why it is running with the wrong context.

docker-selinux and container-selinux are not supposed to be installed at the same time.  cotnainer-selinux should replace docker-selinux.

Remove the docker-selinux package and then reinstall container-selinux.
Comment 40 Marko Myllynen 2017-01-12 20:20:22 UTC 
(In reply to Daniel Walsh from comment #39)
> That is the wrong label. Which is why it is running with the wrong context.
> 
> docker-selinux and container-selinux are not supposed to be installed at the
> same time.  cotnainer-selinux should replace docker-selinux.
> 
> Remove the docker-selinux package and then reinstall container-selinux.

Thanks, finally got it - perhaps consider adding Conflicts or Obsoletes/Provides for these packages on RPM level?

I'm still seeing the earlier issue when e.g. docker-registry-N-deploy / router-N-deploy pods are in ContainerCreating state during OCP deployment, I'll file a separate BZ about this.

[root@infra01 ~]# rpm -q selinux-policy docker docker-selinux container-selinux
selinux-policy-3.13.1-102.el7_3.7.noarch
docker-1.10.3-59.el7.x86_64
package docker-selinux is not installed
container-selinux-1.10.3-59.el7.x86_64
[root@infra01 ~]# grep denied /var/log/audit/audit.log 
type=AVC msg=audit(1484251796.227:1917): avc:  denied  { transition } for  pid=15134 comm="exe" path="/usr/bin/pod" dev="dm-4" ino=2104360 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:svirt_lxc_net_t:s0:c0,c5 tclass=process

Thanks.
Comment 41 Marko Myllynen 2017-01-12 20:29:42 UTC 
For reference, the new BZ I filed is https://bugzilla.redhat.com/show_bug.cgi?id=1412803. Thanks.
Comment 42 Ryan Howe 2017-02-01 22:02:57 UTC 
Do we have any progress on changing the line for the rpm scripts for docker-selinux-1.10

Making sure that we run restorecon on /usr/bin/docker*  and not just /usr/bin/docker

I understand that docker 1.12 uses a new package but OpenShift still support 1.10
Note You need to log in before you can comment on or make changes to this bug.