Bug 1289851 - Docker.service does not require docker.socket which can lead to Docker crash when docker.sock is host mounted
Docker.service does not require docker.socket which can lead to Docker crash ...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: docker (Show other bugs)
23
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Antonio Murdaca
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks: 1332613
  Show dependency treegraph
 
Reported: 2015-12-09 02:22 EST by David Parrish
Modified: 2016-06-09 22:26 EDT (History)
13 users (show)

See Also:
Fixed In Version: docker-1.10.3-24.gitf476348.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-09 22:26:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Require docker.socket before starting docker service (674 bytes, patch)
2015-12-09 02:43 EST, David Parrish
no flags Details | Diff
Logs created when reproducing issue (3.68 KB, application/x-gzip)
2015-12-11 15:06 EST, David Parrish
no flags Details

  None (edit)
Description David Parrish 2015-12-09 02:22:52 EST
Description of problem:

If the /var/run/docker.sock is mounted as is required for some Docker containers, then when the docker service is restarted, /var/run/docker.sock is created as a directory and Docker will fail because it cannot find/create the docker.sock file.

Version-Release number of selected component (if applicable):

docker 1.9.1

How reproducible:

Start a container that needs to see docker.sock such as jwilder/nginx-proxy. Make it restart always. Enable Docker service so it will start when docker is restarted. Restart docker.

Steps to Reproduce:
1. systemctl start docker
2. docker run -d --restart=always -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy
3. systemctl restart docker

Actual results:

/var/run/docker.sock should not be a directory and docker daemon has stopped running.

Expected results:

docker.sock is not created as a directory and docker daemon is running

Additional info:

The fix for me is to add "Requires=docker.socket" as upstream does here: https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service
Comment 1 David Parrish 2015-12-09 02:43 EST
Created attachment 1103806 [details]
Require docker.socket before starting docker service
Comment 2 Daniel Walsh 2015-12-09 09:42:44 EST
We don't want to use docker.socket since this breaks starting docker containers on docker service start.  If you use docker.socket, your container will not start on reboot, and will only start when someone actually communicates with the docker.socket.

I have no idea why you are seeing docker.sock as a directory?  docker.sock should be a socket and it created when the docker.service starts.  I do question why this container needs access to the docker.socket, Seems like a bad design from a security perspective.
Comment 3 David Parrish 2015-12-11 10:12:30 EST
Daniel, If you run through the "Steps to Reproduce" you will how docker.sock becomes a directory. Docker creates it because the container has a volume mount to docker.sock. When docker.sock does not exist, Docker goes ahead and creates a directory. The code that does that is deprecated but it is still there.

The nginx-proxy, as well as other container projects, mount docker.sock so they can monitor other containers and change their behavior accoudingly. The only way around this is to have a host process monitor docker.sock and pass in configuration files through a host mount.

I'm confused as to why docker containers will not start on reboot. When I added docker.socket all my containers rebooted fine. They wouldn't reboot if I didn't have docker.socket for the reasons already explained.
Comment 4 Daniel Walsh 2015-12-11 10:19:23 EST
The docker.socket unit file says wait for someone to connect to the /run/docker.socket before starting docker.service (Docker daemon).  Therefore if I boot my machine with autostart containers, then docker daemon will not start until someone hits the /run/docker.socket.  If you booted an atomic host with no users on it, then noone will hit the docker.socket and docker daemon will not run, autostarted docker containers will not run.
Comment 5 Daniel Walsh 2015-12-11 10:25:17 EST
docker run -v /var/run/docker.sock:/tmp/docker.sock fedora ls -lZ /tmp/docker.sock
srw-rw----. 1 root 975 system_u:object_r:docker_var_run_t:s0 0 Dec 11 13:38 /tmp/docker.sock


This looks fine.

Are you saying docker is starting containers before it has created /run/docker.sock, this could be a problem.
Comment 6 Daniel Walsh 2015-12-11 10:26:09 EST
Tony could you check this out.
Comment 7 David Parrish 2015-12-11 10:42:16 EST
Daniel, Yes, I believe that could be the problem. Try --restart always and reboot.
Comment 8 David Parrish 2015-12-11 10:43:30 EST
That makes sense about docker.socket, but I cannot reproduce on my test environment. With the patch I included, I do the following:

1. docker run -d --restart always fedora bash -c "while true; do echo sleep now; sleep 2; done"
2. reboot

When I log back in:

3. ps -aux | grep bash

And I can see my process running.

[vagrant@localhost ~]$ ps -aux | grep bash                                                                                                                                                    
root       753  0.0  0.5  11756  2512 ?        Ss   15:32   0:00 bash -c while true; do echo sleep now; sleep 2; done
vagrant    866  0.0  0.8  16280  4344 pts/0    Ss   15:32   0:00 -bash
vagrant    973  0.0  0.4  12716  2228 pts/0    S+   15:34   0:00 grep --color=auto bash
[vagrant@localhost ~]$ pgrep -aux docker
pgrep: invalid user name: x
[vagrant@localhost ~]$ ps -aux | grep docker
root       474  0.2  7.4 441424 37368 ?        Ssl  15:32   0:00 /usr/bin/docker daemon --log-driver=journald
vagrant    989  0.0  0.4  12716  2296 pts/0    S+   15:34   0:00 grep --color=auto docker


What am I missing?
Comment 9 Antonio Murdaca 2015-12-11 12:47:22 EST
This is probably a race and I remember this was hit and fixed upstream once (but I don't really remember if it was fixed in 1.9.x or it will in 1.10.x and I'm not sure it's the same issue)

David, could you please provide full docker daemon logs so I can better inspect what's happening? I'll try to reproduce asap though.
Comment 10 Antonio Murdaca 2015-12-11 12:48:38 EST
This is the issue I'm talking about https://github.com/docker/docker/issues/15912
which was fixed in https://github.com/docker/docker/issues/15912

Still have to reproduce though
Comment 11 David Parrish 2015-12-11 15:06 EST
Created attachment 1104825 [details]
Logs created when reproducing issue
Comment 12 David Parrish 2015-12-11 15:07:51 EST
I hope the logs help. I cleared out my docker environment, reverted my systemd service changes and reproduced the issue I'm seeing.
Comment 13 Antonio Murdaca 2015-12-12 04:39:15 EST
I reproduced on a fedora rawhide with docker 1.9.1 built from projectatomic/docker#fedora-1.9

Luckily, the latest version of docker (which is on projectatomic/docker#fedora-1.10) doesn't suffer this issue. It's definitively fixed in docker-1.10.

I might be able to backport the fix to 1.9.x on our branch, though much code changed between 1.9.x and 1.10.x.

Dan, should I try to backport the fix or should I mark this as fixed in docker-1.10?
Comment 14 Daniel Walsh 2015-12-14 12:18:06 EST
David Parrish How critical is this. You have a work around until docker-1.10 is shipped?
Comment 15 David Parrish 2015-12-14 12:34:25 EST
I have a work around but I would still consider the severity medium until docker 1.10 is released.
Comment 16 Jan Provaznik 2016-02-03 03:46:00 EST
I can confirm this issue. I hit it when deploying openshift using openshift-ansible in containers. Using workaround "Requires=docker.socket" fixes the issue during the openshift deployment but because of comment 2 it seems it's not sufficient for real/production deployment. If it's not planned to backport this what is time estimation of shipping docker 1.10?

versions:
centos-atomic host build 2015-Nov-18 which uses docker-1.8.2-7.el7.centos.x86_64
Comment 17 Antonio Murdaca 2016-02-03 05:05:00 EST
(In reply to Jan Provaznik from comment #16)
> I can confirm this issue. I hit it when deploying openshift using
> openshift-ansible in containers. Using workaround "Requires=docker.socket"
> fixes the issue during the openshift deployment but because of comment 2 it
> seems it's not sufficient for real/production deployment. If it's not
> planned to backport this what is time estimation of shipping docker 1.10?

2 weeks from now hopefully

> 
> versions:
> centos-atomic host build 2015-Nov-18 which uses
> docker-1.8.2-7.el7.centos.x86_64
Comment 18 Antonio Murdaca 2016-02-03 05:05:22 EST
(In reply to Antonio Murdaca from comment #17)
> (In reply to Jan Provaznik from comment #16)
> > I can confirm this issue. I hit it when deploying openshift using
> > openshift-ansible in containers. Using workaround "Requires=docker.socket"
> > fixes the issue during the openshift deployment but because of comment 2 it
> > seems it's not sufficient for real/production deployment. If it's not
> > planned to backport this what is time estimation of shipping docker 1.10?
> 
> 2 weeks from now hopefully
> 

probably less

> > 
> > versions:
> > centos-atomic host build 2015-Nov-18 which uses
> > docker-1.8.2-7.el7.centos.x86_64
Comment 19 Lokesh Mandvekar 2016-02-16 12:44:26 EST
docker 1.10.1 is now available in updates-testing.
Comment 20 Fedora Update System 2016-02-16 12:46:59 EST
docker-1.10.1-5.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8
Comment 21 Fedora Update System 2016-02-17 01:26:26 EST
docker-1.10.1-5.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8
Comment 22 Fedora Update System 2016-02-18 11:30:22 EST
docker-1.10.1-6.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798
Comment 23 Fedora Update System 2016-02-21 12:59:40 EST
docker-1.10.1-6.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798
Comment 24 Fedora Update System 2016-02-22 07:07:39 EST
docker-1.10.2-1.git86e59a5.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c
Comment 25 Fedora Update System 2016-02-23 10:20:28 EST
docker-1.10.2-1.git86e59a5.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c
Comment 26 Fedora Update System 2016-02-24 05:32:52 EST
docker-1.10.2-4.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b
Comment 27 Fedora Update System 2016-02-25 21:25:20 EST
docker-1.10.2-4.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b
Comment 28 Fedora Update System 2016-02-26 11:49:10 EST
docker-1.10.2-5.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106
Comment 29 Fedora Update System 2016-02-28 08:55:37 EST
docker-1.10.2-5.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106
Comment 30 Fedora Update System 2016-03-02 12:44:36 EST
docker-1.10.2-6.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5
Comment 31 Fedora Update System 2016-03-03 16:58:23 EST
docker-1.10.2-6.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5
Comment 32 Fedora Update System 2016-03-17 11:03:06 EDT
docker-1.10.2-8.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c
Comment 33 Fedora Update System 2016-03-18 21:25:30 EDT
docker-1.10.2-8.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c
Comment 34 Fedora Update System 2016-04-23 11:22:51 EDT
docker-1.10.3-14.gitef2fa35.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385
Comment 35 Fedora Update System 2016-04-24 02:22:41 EDT
docker-1.10.3-14.gitef2fa35.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385
Comment 36 Fedora Update System 2016-04-27 04:35:38 EDT
docker-1.10.3-15.git964eda6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19
Comment 37 Fedora Update System 2016-04-27 22:31:38 EDT
docker-1.10.3-15.git964eda6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19
Comment 38 Fedora Update System 2016-05-02 05:22:01 EDT
docker-1.10.3-16.gita41254f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5
Comment 39 Fedora Update System 2016-05-03 05:27:03 EDT
docker-1.10.3-16.gita41254f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5
Comment 40 Fedora Update System 2016-05-05 05:40:48 EDT
docker-1.10.3-17.gitbba2d6d.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c7e793ee33
Comment 41 Fedora Update System 2016-05-05 10:04:42 EDT
docker-1.10.3-18.git667d6d1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39
Comment 42 Fedora Update System 2016-05-06 16:56:11 EDT
docker-1.10.3-18.git667d6d1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39
Comment 43 Fedora Update System 2016-05-20 06:34:48 EDT
docker-1.10.3-19.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9
Comment 44 Fedora Update System 2016-05-20 22:25:43 EDT
docker-1.10.3-19.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9
Comment 45 Fedora Update System 2016-05-24 12:44:35 EDT
docker-1.10.3-20.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308
Comment 46 Fedora Update System 2016-05-26 01:01:41 EDT
docker-1.10.3-20.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308
Comment 47 Fedora Update System 2016-05-26 14:27:18 EDT
docker-1.10.3-21.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f
Comment 48 Fedora Update System 2016-05-27 23:25:16 EDT
docker-1.10.3-21.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f
Comment 49 Fedora Update System 2016-05-30 13:35:28 EDT
docker-1.10.3-22.git4158ccc.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7
Comment 50 Fedora Update System 2016-05-31 05:52:12 EDT
docker-1.10.3-22.git4158ccc.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7
Comment 51 Fedora Update System 2016-06-01 05:00:59 EDT
docker-1.10.3-23.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d73f74a557
Comment 52 Fedora Update System 2016-06-01 10:52:28 EDT
docker-1.10.3-24.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088
Comment 53 Fedora Update System 2016-06-02 12:22:36 EDT
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088
Comment 54 Fedora Update System 2016-06-09 22:25:35 EDT
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.