Bug 1289851 - Docker.service does not require docker.socket which can lead to Docker crash when docker.sock is host mounted
Summary: Docker.service does not require docker.socket which can lead to Docker crash ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 23
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Antonio Murdaca
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1332613
TreeView+ depends on / blocked
 
Reported: 2015-12-09 07:22 UTC by David Parrish
Modified: 2016-06-10 02:26 UTC (History)
13 users (show)

Fixed In Version: docker-1.10.3-24.gitf476348.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-10 02:26:28 UTC


Attachments (Terms of Use)
Require docker.socket before starting docker service (674 bytes, patch)
2015-12-09 07:43 UTC, David Parrish
no flags Details | Diff
Logs created when reproducing issue (3.68 KB, application/x-gzip)
2015-12-11 20:06 UTC, David Parrish
no flags Details

Description David Parrish 2015-12-09 07:22:52 UTC
Description of problem:

If the /var/run/docker.sock is mounted as is required for some Docker containers, then when the docker service is restarted, /var/run/docker.sock is created as a directory and Docker will fail because it cannot find/create the docker.sock file.

Version-Release number of selected component (if applicable):

docker 1.9.1

How reproducible:

Start a container that needs to see docker.sock such as jwilder/nginx-proxy. Make it restart always. Enable Docker service so it will start when docker is restarted. Restart docker.

Steps to Reproduce:
1. systemctl start docker
2. docker run -d --restart=always -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy
3. systemctl restart docker

Actual results:

/var/run/docker.sock should not be a directory and docker daemon has stopped running.

Expected results:

docker.sock is not created as a directory and docker daemon is running

Additional info:

The fix for me is to add "Requires=docker.socket" as upstream does here: https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service

Comment 1 David Parrish 2015-12-09 07:43:30 UTC
Created attachment 1103806 [details]
Require docker.socket before starting docker service

Comment 2 Daniel Walsh 2015-12-09 14:42:44 UTC
We don't want to use docker.socket since this breaks starting docker containers on docker service start.  If you use docker.socket, your container will not start on reboot, and will only start when someone actually communicates with the docker.socket.

I have no idea why you are seeing docker.sock as a directory?  docker.sock should be a socket and it created when the docker.service starts.  I do question why this container needs access to the docker.socket, Seems like a bad design from a security perspective.

Comment 3 David Parrish 2015-12-11 15:12:30 UTC
Daniel, If you run through the "Steps to Reproduce" you will how docker.sock becomes a directory. Docker creates it because the container has a volume mount to docker.sock. When docker.sock does not exist, Docker goes ahead and creates a directory. The code that does that is deprecated but it is still there.

The nginx-proxy, as well as other container projects, mount docker.sock so they can monitor other containers and change their behavior accoudingly. The only way around this is to have a host process monitor docker.sock and pass in configuration files through a host mount.

I'm confused as to why docker containers will not start on reboot. When I added docker.socket all my containers rebooted fine. They wouldn't reboot if I didn't have docker.socket for the reasons already explained.

Comment 4 Daniel Walsh 2015-12-11 15:19:23 UTC
The docker.socket unit file says wait for someone to connect to the /run/docker.socket before starting docker.service (Docker daemon).  Therefore if I boot my machine with autostart containers, then docker daemon will not start until someone hits the /run/docker.socket.  If you booted an atomic host with no users on it, then noone will hit the docker.socket and docker daemon will not run, autostarted docker containers will not run.

Comment 5 Daniel Walsh 2015-12-11 15:25:17 UTC
docker run -v /var/run/docker.sock:/tmp/docker.sock fedora ls -lZ /tmp/docker.sock
srw-rw----. 1 root 975 system_u:object_r:docker_var_run_t:s0 0 Dec 11 13:38 /tmp/docker.sock


This looks fine.

Are you saying docker is starting containers before it has created /run/docker.sock, this could be a problem.

Comment 6 Daniel Walsh 2015-12-11 15:26:09 UTC
Tony could you check this out.

Comment 7 David Parrish 2015-12-11 15:42:16 UTC
Daniel, Yes, I believe that could be the problem. Try --restart always and reboot.

Comment 8 David Parrish 2015-12-11 15:43:30 UTC
That makes sense about docker.socket, but I cannot reproduce on my test environment. With the patch I included, I do the following:

1. docker run -d --restart always fedora bash -c "while true; do echo sleep now; sleep 2; done"
2. reboot

When I log back in:

3. ps -aux | grep bash

And I can see my process running.

[vagrant@localhost ~]$ ps -aux | grep bash                                                                                                                                                    
root       753  0.0  0.5  11756  2512 ?        Ss   15:32   0:00 bash -c while true; do echo sleep now; sleep 2; done
vagrant    866  0.0  0.8  16280  4344 pts/0    Ss   15:32   0:00 -bash
vagrant    973  0.0  0.4  12716  2228 pts/0    S+   15:34   0:00 grep --color=auto bash
[vagrant@localhost ~]$ pgrep -aux docker
pgrep: invalid user name: x
[vagrant@localhost ~]$ ps -aux | grep docker
root       474  0.2  7.4 441424 37368 ?        Ssl  15:32   0:00 /usr/bin/docker daemon --log-driver=journald
vagrant    989  0.0  0.4  12716  2296 pts/0    S+   15:34   0:00 grep --color=auto docker


What am I missing?

Comment 9 Antonio Murdaca 2015-12-11 17:47:22 UTC
This is probably a race and I remember this was hit and fixed upstream once (but I don't really remember if it was fixed in 1.9.x or it will in 1.10.x and I'm not sure it's the same issue)

David, could you please provide full docker daemon logs so I can better inspect what's happening? I'll try to reproduce asap though.

Comment 10 Antonio Murdaca 2015-12-11 17:48:38 UTC
This is the issue I'm talking about https://github.com/docker/docker/issues/15912
which was fixed in https://github.com/docker/docker/issues/15912

Still have to reproduce though

Comment 11 David Parrish 2015-12-11 20:06:01 UTC
Created attachment 1104825 [details]
Logs created when reproducing issue

Comment 12 David Parrish 2015-12-11 20:07:51 UTC
I hope the logs help. I cleared out my docker environment, reverted my systemd service changes and reproduced the issue I'm seeing.

Comment 13 Antonio Murdaca 2015-12-12 09:39:15 UTC
I reproduced on a fedora rawhide with docker 1.9.1 built from projectatomic/docker#fedora-1.9

Luckily, the latest version of docker (which is on projectatomic/docker#fedora-1.10) doesn't suffer this issue. It's definitively fixed in docker-1.10.

I might be able to backport the fix to 1.9.x on our branch, though much code changed between 1.9.x and 1.10.x.

Dan, should I try to backport the fix or should I mark this as fixed in docker-1.10?

Comment 14 Daniel Walsh 2015-12-14 17:18:06 UTC
David Parrish How critical is this. You have a work around until docker-1.10 is shipped?

Comment 15 David Parrish 2015-12-14 17:34:25 UTC
I have a work around but I would still consider the severity medium until docker 1.10 is released.

Comment 16 Jan Provaznik 2016-02-03 08:46:00 UTC
I can confirm this issue. I hit it when deploying openshift using openshift-ansible in containers. Using workaround "Requires=docker.socket" fixes the issue during the openshift deployment but because of comment 2 it seems it's not sufficient for real/production deployment. If it's not planned to backport this what is time estimation of shipping docker 1.10?

versions:
centos-atomic host build 2015-Nov-18 which uses docker-1.8.2-7.el7.centos.x86_64

Comment 17 Antonio Murdaca 2016-02-03 10:05:00 UTC
(In reply to Jan Provaznik from comment #16)
> I can confirm this issue. I hit it when deploying openshift using
> openshift-ansible in containers. Using workaround "Requires=docker.socket"
> fixes the issue during the openshift deployment but because of comment 2 it
> seems it's not sufficient for real/production deployment. If it's not
> planned to backport this what is time estimation of shipping docker 1.10?

2 weeks from now hopefully

> 
> versions:
> centos-atomic host build 2015-Nov-18 which uses
> docker-1.8.2-7.el7.centos.x86_64

Comment 18 Antonio Murdaca 2016-02-03 10:05:22 UTC
(In reply to Antonio Murdaca from comment #17)
> (In reply to Jan Provaznik from comment #16)
> > I can confirm this issue. I hit it when deploying openshift using
> > openshift-ansible in containers. Using workaround "Requires=docker.socket"
> > fixes the issue during the openshift deployment but because of comment 2 it
> > seems it's not sufficient for real/production deployment. If it's not
> > planned to backport this what is time estimation of shipping docker 1.10?
> 
> 2 weeks from now hopefully
> 

probably less

> > 
> > versions:
> > centos-atomic host build 2015-Nov-18 which uses
> > docker-1.8.2-7.el7.centos.x86_64

Comment 19 Lokesh Mandvekar 2016-02-16 17:44:26 UTC
docker 1.10.1 is now available in updates-testing.

Comment 20 Fedora Update System 2016-02-16 17:46:59 UTC
docker-1.10.1-5.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8

Comment 21 Fedora Update System 2016-02-17 06:26:26 UTC
docker-1.10.1-5.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8

Comment 22 Fedora Update System 2016-02-18 16:30:22 UTC
docker-1.10.1-6.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798

Comment 23 Fedora Update System 2016-02-21 17:59:40 UTC
docker-1.10.1-6.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798

Comment 24 Fedora Update System 2016-02-22 12:07:39 UTC
docker-1.10.2-1.git86e59a5.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c

Comment 25 Fedora Update System 2016-02-23 15:20:28 UTC
docker-1.10.2-1.git86e59a5.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c

Comment 26 Fedora Update System 2016-02-24 10:32:52 UTC
docker-1.10.2-4.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b

Comment 27 Fedora Update System 2016-02-26 02:25:20 UTC
docker-1.10.2-4.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b

Comment 28 Fedora Update System 2016-02-26 16:49:10 UTC
docker-1.10.2-5.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106

Comment 29 Fedora Update System 2016-02-28 13:55:37 UTC
docker-1.10.2-5.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106

Comment 30 Fedora Update System 2016-03-02 17:44:36 UTC
docker-1.10.2-6.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5

Comment 31 Fedora Update System 2016-03-03 21:58:23 UTC
docker-1.10.2-6.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5

Comment 32 Fedora Update System 2016-03-17 15:03:06 UTC
docker-1.10.2-8.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c

Comment 33 Fedora Update System 2016-03-19 01:25:30 UTC
docker-1.10.2-8.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c

Comment 34 Fedora Update System 2016-04-23 15:22:51 UTC
docker-1.10.3-14.gitef2fa35.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385

Comment 35 Fedora Update System 2016-04-24 06:22:41 UTC
docker-1.10.3-14.gitef2fa35.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385

Comment 36 Fedora Update System 2016-04-27 08:35:38 UTC
docker-1.10.3-15.git964eda6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19

Comment 37 Fedora Update System 2016-04-28 02:31:38 UTC
docker-1.10.3-15.git964eda6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19

Comment 38 Fedora Update System 2016-05-02 09:22:01 UTC
docker-1.10.3-16.gita41254f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 39 Fedora Update System 2016-05-03 09:27:03 UTC
docker-1.10.3-16.gita41254f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5

Comment 40 Fedora Update System 2016-05-05 09:40:48 UTC
docker-1.10.3-17.gitbba2d6d.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c7e793ee33

Comment 41 Fedora Update System 2016-05-05 14:04:42 UTC
docker-1.10.3-18.git667d6d1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 42 Fedora Update System 2016-05-06 20:56:11 UTC
docker-1.10.3-18.git667d6d1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39

Comment 43 Fedora Update System 2016-05-20 10:34:48 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 44 Fedora Update System 2016-05-21 02:25:43 UTC
docker-1.10.3-19.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9

Comment 45 Fedora Update System 2016-05-24 16:44:35 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 46 Fedora Update System 2016-05-26 05:01:41 UTC
docker-1.10.3-20.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308

Comment 47 Fedora Update System 2016-05-26 18:27:18 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 48 Fedora Update System 2016-05-28 03:25:16 UTC
docker-1.10.3-21.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f

Comment 49 Fedora Update System 2016-05-30 17:35:28 UTC
docker-1.10.3-22.git4158ccc.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 50 Fedora Update System 2016-05-31 09:52:12 UTC
docker-1.10.3-22.git4158ccc.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7

Comment 51 Fedora Update System 2016-06-01 09:00:59 UTC
docker-1.10.3-23.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d73f74a557

Comment 52 Fedora Update System 2016-06-01 14:52:28 UTC
docker-1.10.3-24.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 53 Fedora Update System 2016-06-02 16:22:36 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088

Comment 54 Fedora Update System 2016-06-10 02:25:35 UTC
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.