Description of problem: If the /var/run/docker.sock is mounted as is required for some Docker containers, then when the docker service is restarted, /var/run/docker.sock is created as a directory and Docker will fail because it cannot find/create the docker.sock file. Version-Release number of selected component (if applicable): docker 1.9.1 How reproducible: Start a container that needs to see docker.sock such as jwilder/nginx-proxy. Make it restart always. Enable Docker service so it will start when docker is restarted. Restart docker. Steps to Reproduce: 1. systemctl start docker 2. docker run -d --restart=always -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy 3. systemctl restart docker Actual results: /var/run/docker.sock should not be a directory and docker daemon has stopped running. Expected results: docker.sock is not created as a directory and docker daemon is running Additional info: The fix for me is to add "Requires=docker.socket" as upstream does here: https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service
Created attachment 1103806 [details] Require docker.socket before starting docker service
We don't want to use docker.socket since this breaks starting docker containers on docker service start. If you use docker.socket, your container will not start on reboot, and will only start when someone actually communicates with the docker.socket. I have no idea why you are seeing docker.sock as a directory? docker.sock should be a socket and it created when the docker.service starts. I do question why this container needs access to the docker.socket, Seems like a bad design from a security perspective.
Daniel, If you run through the "Steps to Reproduce" you will how docker.sock becomes a directory. Docker creates it because the container has a volume mount to docker.sock. When docker.sock does not exist, Docker goes ahead and creates a directory. The code that does that is deprecated but it is still there. The nginx-proxy, as well as other container projects, mount docker.sock so they can monitor other containers and change their behavior accoudingly. The only way around this is to have a host process monitor docker.sock and pass in configuration files through a host mount. I'm confused as to why docker containers will not start on reboot. When I added docker.socket all my containers rebooted fine. They wouldn't reboot if I didn't have docker.socket for the reasons already explained.
The docker.socket unit file says wait for someone to connect to the /run/docker.socket before starting docker.service (Docker daemon). Therefore if I boot my machine with autostart containers, then docker daemon will not start until someone hits the /run/docker.socket. If you booted an atomic host with no users on it, then noone will hit the docker.socket and docker daemon will not run, autostarted docker containers will not run.
docker run -v /var/run/docker.sock:/tmp/docker.sock fedora ls -lZ /tmp/docker.sock srw-rw----. 1 root 975 system_u:object_r:docker_var_run_t:s0 0 Dec 11 13:38 /tmp/docker.sock This looks fine. Are you saying docker is starting containers before it has created /run/docker.sock, this could be a problem.
Tony could you check this out.
Daniel, Yes, I believe that could be the problem. Try --restart always and reboot.
That makes sense about docker.socket, but I cannot reproduce on my test environment. With the patch I included, I do the following: 1. docker run -d --restart always fedora bash -c "while true; do echo sleep now; sleep 2; done" 2. reboot When I log back in: 3. ps -aux | grep bash And I can see my process running. [vagrant@localhost ~]$ ps -aux | grep bash root 753 0.0 0.5 11756 2512 ? Ss 15:32 0:00 bash -c while true; do echo sleep now; sleep 2; done vagrant 866 0.0 0.8 16280 4344 pts/0 Ss 15:32 0:00 -bash vagrant 973 0.0 0.4 12716 2228 pts/0 S+ 15:34 0:00 grep --color=auto bash [vagrant@localhost ~]$ pgrep -aux docker pgrep: invalid user name: x [vagrant@localhost ~]$ ps -aux | grep docker root 474 0.2 7.4 441424 37368 ? Ssl 15:32 0:00 /usr/bin/docker daemon --log-driver=journald vagrant 989 0.0 0.4 12716 2296 pts/0 S+ 15:34 0:00 grep --color=auto docker What am I missing?
This is probably a race and I remember this was hit and fixed upstream once (but I don't really remember if it was fixed in 1.9.x or it will in 1.10.x and I'm not sure it's the same issue) David, could you please provide full docker daemon logs so I can better inspect what's happening? I'll try to reproduce asap though.
This is the issue I'm talking about https://github.com/docker/docker/issues/15912 which was fixed in https://github.com/docker/docker/issues/15912 Still have to reproduce though
Created attachment 1104825 [details] Logs created when reproducing issue
I hope the logs help. I cleared out my docker environment, reverted my systemd service changes and reproduced the issue I'm seeing.
I reproduced on a fedora rawhide with docker 1.9.1 built from projectatomic/docker#fedora-1.9 Luckily, the latest version of docker (which is on projectatomic/docker#fedora-1.10) doesn't suffer this issue. It's definitively fixed in docker-1.10. I might be able to backport the fix to 1.9.x on our branch, though much code changed between 1.9.x and 1.10.x. Dan, should I try to backport the fix or should I mark this as fixed in docker-1.10?
David Parrish How critical is this. You have a work around until docker-1.10 is shipped?
I have a work around but I would still consider the severity medium until docker 1.10 is released.
I can confirm this issue. I hit it when deploying openshift using openshift-ansible in containers. Using workaround "Requires=docker.socket" fixes the issue during the openshift deployment but because of comment 2 it seems it's not sufficient for real/production deployment. If it's not planned to backport this what is time estimation of shipping docker 1.10? versions: centos-atomic host build 2015-Nov-18 which uses docker-1.8.2-7.el7.centos.x86_64
(In reply to Jan Provaznik from comment #16) > I can confirm this issue. I hit it when deploying openshift using > openshift-ansible in containers. Using workaround "Requires=docker.socket" > fixes the issue during the openshift deployment but because of comment 2 it > seems it's not sufficient for real/production deployment. If it's not > planned to backport this what is time estimation of shipping docker 1.10? 2 weeks from now hopefully > > versions: > centos-atomic host build 2015-Nov-18 which uses > docker-1.8.2-7.el7.centos.x86_64
(In reply to Antonio Murdaca from comment #17) > (In reply to Jan Provaznik from comment #16) > > I can confirm this issue. I hit it when deploying openshift using > > openshift-ansible in containers. Using workaround "Requires=docker.socket" > > fixes the issue during the openshift deployment but because of comment 2 it > > seems it's not sufficient for real/production deployment. If it's not > > planned to backport this what is time estimation of shipping docker 1.10? > > 2 weeks from now hopefully > probably less > > > > versions: > > centos-atomic host build 2015-Nov-18 which uses > > docker-1.8.2-7.el7.centos.x86_64
docker 1.10.1 is now available in updates-testing.
docker-1.10.1-5.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8
docker-1.10.1-5.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-766e3821e8
docker-1.10.1-6.git6c71d8f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798
docker-1.10.1-6.git6c71d8f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f06313798
docker-1.10.2-1.git86e59a5.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c
docker-1.10.2-1.git86e59a5.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f9d972be2c
docker-1.10.2-4.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b
docker-1.10.2-4.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b9dc51a02b
docker-1.10.2-5.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106
docker-1.10.2-5.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e63d91c106
docker-1.10.2-6.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5
docker-1.10.2-6.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-863b6725b5
docker-1.10.2-8.git0f5ac89.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c
docker-1.10.2-8.git0f5ac89.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-41a553b86c
docker-1.10.3-14.gitef2fa35.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385
docker-1.10.3-14.gitef2fa35.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0bfa795385
docker-1.10.3-15.git964eda6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19
docker-1.10.3-15.git964eda6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b59d274e19
docker-1.10.3-16.gita41254f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5
docker-1.10.3-16.gita41254f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-87f810b0f5
docker-1.10.3-17.gitbba2d6d.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c7e793ee33
docker-1.10.3-18.git667d6d1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39
docker-1.10.3-18.git667d6d1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-7a1fb10a39
docker-1.10.3-19.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9
docker-1.10.3-19.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dd133dc2e9
docker-1.10.3-20.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308
docker-1.10.3-20.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-373d4f6308
docker-1.10.3-21.git8ecd47f.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f
docker-1.10.3-21.git8ecd47f.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-f5cdae8c6f
docker-1.10.3-22.git4158ccc.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7
docker-1.10.3-22.git4158ccc.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6472a8cdc7
docker-1.10.3-23.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d73f74a557
docker-1.10.3-24.gitf476348.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-6a0d540088
docker-1.10.3-24.gitf476348.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.