Bug 1405131 - Docker refuses to start containers (SELINUX).
Summary: Docker refuses to start containers (SELINUX).
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Daniel Walsh
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-15 16:44 UTC by Carles Mateu
Modified: 2020-08-13 08:45 UTC (History)
19 users (show)

Fixed In Version: docker-1.12.4-7.gita7cae3f.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-21 17:13:49 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Carles Mateu 2016-12-15 16:44:01 UTC
Description of problem:

Docker refuses to start containers on an freshly and dnf upgraded F25.

Version-Release number of selected component (if applicable):

docker-1.12.4-2.git1b5971a.fc25.x86_64
docker-common-1.12.4-2.git1b5971a.fc25.x86_64
container-selinux-1.12.4-2.git1b5971a.fc25.x86_64


How reproducible:

Easy to reproduce.



Steps to Reproduce:
1. Install docker: dnf install docker-engine
2. Start it: systemctl start docker
3. Try to run simple container: docker run hello-world


Actual results:

[carlesm@carlesm ~]$ sudo docker run hello-world
container_linux.go:247: starting container process caused  "process_linux.go:334: running prestart hook 1 caused \"error   running hook: exit status 1, stdout: , stderr: \""
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error:  container_linux.go:247: starting container process caused  \"process_linux.go:334: running prestart hook 1 caused  \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".


Expected results:

[carlesm@carlesm ~]$ sudo docker run hello-world
[sudo] contrasenya per a carlesm: 

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/


Additional info:

If selinux is disabled things work flawlessly.

Comment 1 Carles Mateu 2016-12-15 16:48:38 UTC
Adding the following 3 policies to selinux allows things to start correctly (i created them in 3 steps):

docker.te -------------------
module docker 1.0;

require {
	type unconfined_service_t;
	type systemd_machined_t;
	class dir search;
	class file { open read };
}

#============= systemd_machined_t ==============

#!!!! This avc is allowed in the current policy
allow systemd_machined_t unconfined_service_t:dir search;
allow systemd_machined_t unconfined_service_t:file open;

#!!!! This avc is allowed in the current policy
allow systemd_machined_t unconfined_service_t:file read;

docker2.te --------------------

module docker2 1.0;

require {
	type container_t;
	type unconfined_service_t;
	class process transition;
}

#============= unconfined_service_t ==============

#!!!! The file '/hello' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /hello
allow unconfined_service_t container_t:process transition;

docker3.te --------------------

module docker3 1.0;

require {
	type systemd_machined_t;
	type container_t;
	type unconfined_service_t;
	class process sigchld;
	class file getattr;
}

#============= container_t ==============
allow container_t unconfined_service_t:process sigchld;

#============= systemd_machined_t ==============
allow systemd_machined_t unconfined_service_t:file getattr;


-----------------------


Issue seems to be on labels on docker binaries:

[carlesm@carlesm ~]$ ls -Z1 /usr/bin/docker*
system_u:object_r:bin_t:s0 /usr/bin/docker
system_u:object_r:bin_t:s0 /usr/bin/docker-current
system_u:object_r:bin_t:s0 /usr/bin/dockerd-current
system_u:object_r:bin_t:s0 /usr/bin/docker-storage-setup


on (https://bugzilla.redhat.com/show_bug.cgi?id=1358819) it's stated they should be:

system_u:object_r:docker_exec_t:s0

Comment 2 Antonio Murdaca 2016-12-15 16:50:31 UTC
docker-1.12.4-2.git1b5971a.fc25.x86_64 was a bad update wrt container-selinux - I'm updating docker just now.

Comment 3 Fedora Update System 2016-12-15 16:52:51 UTC
docker-1.12.4-7.gita7cae3f.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f1fa11634

Comment 4 Anass Ahmed 2016-12-15 17:05:14 UTC
docker-1.12.4-7.gita7cae3f.fc25 fixed it for me. It works with and without :Z option at the end of the volume.

Comment 5 Daniel Walsh 2016-12-15 18:04:01 UTC
Right.  If you use it without the :Z then the volume could be shared between multiple containers, if you use :Z it will be private to the container.

BTW, I think a second :Z on a volume would remove it from access from the initial container.

Comment 6 Daniel Walsh 2016-12-15 18:09:08 UTC
Temporary fix

chcon -t container_runtime_exec_t /usr/bin/docker*

systemctl restart docker docker-containerd

Should fix the issue until the package gets updated.

Comment 7 Fedora Update System 2016-12-16 05:33:57 UTC
docker-1.12.4-7.gita7cae3f.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f1fa11634

Comment 8 Fedora Update System 2016-12-16 21:01:50 UTC
docker-1.12.4-7.gita7cae3f.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 9 Eric Sauer 2017-01-09 20:45:05 UTC
I am experiencing this issue again with docker 1.12.5;

[root@localhost ~]# rpm -qa | grep docker
docker-common-1.12.5-4.git03508cc.fc25.x86_64
docker-1.12.5-4.git03508cc.fc25.x86_64

[root@localhost ~]# rpm -qa | grep container-selinux
container-selinux-1.12.5-4.git03508cc.fc25.x86_64

[root@localhost ~]# docker run hello-world
container_linux.go:247: starting container process caused "process_linux.go:334: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \""
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".

Comment 10 Daniel Walsh 2017-01-09 22:47:57 UTC
Are you seeing AVCs?

docker-1.12.5-6.git03508cc.fc25 Seems to be in updates-testing.

Comment 11 Eric Sauer 2017-01-11 15:25:47 UTC
(In reply to Daniel Walsh from comment #10)
> Are you seeing AVCs?
> 
> docker-1.12.5-6.git03508cc.fc25 Seems to be in updates-testing.

Yes I'm seeing one AVC denial:

Jan 11 10:20:02 localhost audit: AVC avc:  denied  { search } for  pid=3104 comm="systemd-machine" name="9736" dev="proc" ino=677169 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dir permissive=0
Jan 11 10:20:02 localhost oci-register-machine[9760]: 2017/01/11 10:20:02 Register machine failed: Failed to determine unit of process 9736 : Permission denied


Here's a full system log capture during an attempt to run `docker run hello-world`

http://pastebin.com/JR5Abrup

Comment 12 Daniel Walsh 2017-01-11 15:28:28 UTC
dnf reinstall container-selinux
restorecon -R -v /usr/bin
systemctl restart docker

The problem I see is that your docker is running with the wrong label.

ps -eZ | grep unconfined_service_t

Comment 13 Eric Sauer 2017-01-11 18:22:23 UTC
Looks like I still have the same labels after the restorecon.

[root@localhost ~]# dnf reinstall container-selinux -y
[root@localhost ~]# restorecon -R -v /usr/bin
[root@localhost ~]# systemctl restart docker
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# ps -eZ | grep unconfined_service_t
system_u:system_r:unconfined_service_t:s0 1076 ? 00:00:01 rngd
system_u:system_r:unconfined_service_t:s0 22212 ? 00:00:00 docker-containe
system_u:system_r:unconfined_service_t:s0 26316 ? 00:00:00 dockerd-current


Also, what is docker-containe? that looks like it's supossed to be docker-containerd?

Comment 14 Mike Goodwin 2017-05-21 17:08:18 UTC
This is still broken: 

Unable to find image 'hello-world:latest' locally
Trying to pull repository docker.io/library/hello-world ...
sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7: Pulling from docker.io/library/hello-world
78445dd45222: Pull complete
Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7
Status: Downloaded newer image for docker.io/hello-world:latest
container_linux.go:247: starting container process caused "process_linux.go:334: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: \""
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".

`docker hello-world` after a newly truncated audit.log and set in permissive mode:

type=VIRT_CONTROL msg=audit(1495386442.101:2075): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='vm-pid=? user=xenith auid=1000 exe=? hostname=? reason=api op=create vm=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1495386442.103:2076): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='vm=? vm-pid=? user=xenith auid=1000 exe=? hostname=? reason=api op=create  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1495386443.875:2077): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='user=xenith auid=1000 exe=? hostname=? reason=api op=create vm=? vm-pid=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1495386444.277:2078): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='vm=? vm-pid=? user=xenith auid=1000 exe=? hostname=? reason=api op=attach  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=VIRT_CONTROL msg=audit(1495386444.277:2079): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='exe=? hostname=? reason=api op=start vm=? vm-pid=? user=xenith auid=1000  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=ANOM_PROMISCUOUS msg=audit(1495386444.333:2080): dev=veth001fd62 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
type=AVC msg=audit(1495386444.533:2081): avc:  denied  { search } for  pid=13440 comm="systemd-machine" name="31998" dev="proc" ino=676813 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1495386444.533:2082): avc:  denied  { read } for  pid=13440 comm="systemd-machine" name="cgroup" dev="proc" ino=677779 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=file permissive=1
type=AVC msg=audit(1495386444.533:2083): avc:  denied  { open } for  pid=13440 comm="systemd-machine" path="/proc/31998/cgroup" dev="proc" ino=677779 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=file permissive=1
type=AVC msg=audit(1495386444.534:2084): avc:  denied  { getattr } for  pid=13440 comm="systemd-machine" path="/proc/31998/cgroup" dev="proc" ino=677779 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=file permissive=1
type=AVC msg=audit(1495386444.606:2085): avc:  denied  { transition } for  pid=31998 comm="exe" path="/hello" dev="dm-2" ino=133 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c530,c573 tclass=process permissive=1
type=AVC msg=audit(1495386444.641:2086): avc:  denied  { sigchld } for  pid=31982 comm="docker-containe" scontext=system_u:system_r:container_t:s0:c530,c573 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=1
type=VIRT_CONTROL msg=audit(1495386444.733:2087): pid=13602 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='hostname=? reason=api op=wait vm=? vm-pid=? user=xenith auid=1000 exe=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
type=ANOM_PROMISCUOUS msg=audit(1495386444.814:2088): dev=veth001fd62 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295

Comment 15 Mike Goodwin 2017-05-21 17:13:49 UTC
Please disregard I just saw version 2.14 of container-selinux in updates-testing. This fixes the issue. Thanks dan!


Note You need to log in before you can comment on or make changes to this bug.