Bug 1403264

Summary: systemctl start docker for docker-1.12.3-10.el7.x86_64 fails to start
Product: Red Hat Enterprise Linux 7 Reporter: Jan Pazdziora <jpazdziora>
Component: dockerAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: amurdaca, dwalsh, ghuang, jpazdziora, lsm5, lsu, santiago, weshi
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: docker-1.12.5-9.el7_3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 20:44:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Pazdziora 2016-12-09 14:35:53 UTC
Description of problem:

With docker-1.12.3-10.el7.x86_64, the docker.service fails to even start.

Version-Release number of selected component (if applicable):

docker-1.12.3-10.el7.x86_64

How reproducible:

Seen once, assume deterministic.

Steps to Reproduce:
1. Install docker-1.12.3-10.el7.x86_64 from extras.
2. Run systemctl start docker

Actual results:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

Expected results:

No error, daemon running.

Additional info:

# systemctl status docker.service -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2016-12-09 09:18:52 EST; 11min ago
     Docs: http://docs.docker.com
  Process: 12603 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE)
 Main PID: 12603 (code=exited, status=1/FAILURE)

Dec 09 09:18:52 machine.example.com systemd[1]: Starting Docker Application Container Engine...
Dec 09 09:18:52 machine.example.com dockerd-current[12603]: time="2016-12-09T09:18:52.598697535-05:00" level=fatal msg="Failed to connect to containerd. Please make sure containerd is installed in your PATH or you have specificed the correct address. Got error: exec: \"docker-containerd\": executable file not found in $PATH"
Dec 09 09:18:52 machine.example.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 09 09:18:52 machine.example.com systemd[1]: Failed to start Docker Application Container Engine.
Dec 09 09:18:52 machine.example.com systemd[1]: Unit docker.service entered failed state.
Dec 09 09:18:52 machine.example.com systemd[1]: docker.service failed.

Comment 1 Antonio Murdaca 2016-12-09 14:39:34 UTC
This seems a bug with the docker.service unit file; the unit is missing:

          --containerd /run/containerd.sock \

Lokesh, can you check this one?

Comment 10 Daniel Walsh 2016-12-09 15:26:10 UTC
We could also ship /usr/libexec/docker/docker-containerd and have it figure out which docker-container-* to link with.

Comment 11 Antonio Murdaca 2016-12-09 15:30:07 UTC
(In reply to Daniel Walsh from comment #10)
> We could also ship /usr/libexec/docker/docker-containerd and have it figure
> out which docker-container-* to link with.

that's fine - we still need a patch in docker to specify the containerd path (or add /usr/libexec/docker in $PATH which I wouldn't...)

Comment 12 Daniel Walsh 2016-12-09 15:34:27 UTC
I thought docker execs /usr/libexec/docker/docker-containerd if containerd is not running?  Or does it look for /usr/bin/docker-containerd?

Comment 13 Antonio Murdaca 2016-12-09 15:35:35 UTC
(In reply to Daniel Walsh from comment #12)
> I thought docker execs /usr/libexec/docker/docker-containerd if containerd
> is not running?  Or does it look for /usr/bin/docker-containerd?

it looks for $PATH/docker-containerd

Comment 14 Daniel Walsh 2016-12-09 15:36:59 UTC
Ok we ship /usr/bin/docker-containerd with the same logic that /usr/bin/docker has to find /usr/libexec/docker/docker-containerd-*

Same for runc.

Comment 15 Antonio Murdaca 2016-12-09 15:48:59 UTC
(In reply to Daniel Walsh from comment #14)
> Ok we ship /usr/bin/docker-containerd with the same logic that
> /usr/bin/docker has to find /usr/libexec/docker/docker-containerd-*
> 
> Same for runc.

Dan, something like this would work according to my testing on F25:

$ cat docker-containerd-common.sh                     
#!/bin/sh
. /etc/sysconfig/docker
[ -e "${DOCKER_CONTAINERD_BINARY}" ] || DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-current
if [ ! -f /usr/bin/docker-containerd-current ]; then
    DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
fi
if [[ ${DOCKER_CONTAINERD_BINARY} != "/usr/bin/docker-containerd-current" && ${DOCKER_CONTAINERD_BINARY} != /usr/bin/docker-containerd-latest ]]; then
    echo "DOCKER_CONTAINERD_BINARY has been set to an invalid value:" $DOCKER_CONTAINERD_BINARY
    echo ""
    echo "Please set DOCKERBINARY to /usr/bin/docker-containerd-current or /usr/bin/docker-containerd-latest
by editing /etc/sysconfig/docker"
else
    exec ${DOCKER_CONTAINERD_BINARY} "$@"
fi

Comment 16 Antonio Murdaca 2016-12-09 15:55:30 UTC
Dan, does this fix (no containerd own service unit) need to be done in Fedora as well? or is this just for RHEL?

Comment 17 Daniel Walsh 2016-12-09 15:59:31 UTC
No just RHEL.

Comment 20 Ed Santiago 2016-12-12 21:24:56 UTC
Also, if you get dockerd running (via symlinking docker-containerd -> docker-containerd-current), running docker yields:

    /usr/bin/docker: line 13: /usr/bin/docker-latest: No such file or directory

root cause: /usr/bin/docker first looks for docker-current, which doesn't exist.

Have you run rpmdiff against this? I suspect rpmdiff might find more problems.

Comment 22 Jan Pazdziora 2016-12-19 13:08:04 UTC
FAILS_QA with docker-common-1.12.5-3.el7.x86_64: bug 1405989.

Comment 23 Lokesh Mandvekar 2017-01-05 19:03:06 UTC
(In reply to Ed Santiago from comment #20)
> Also, if you get dockerd running (via symlinking docker-containerd ->
> docker-containerd-current), running docker yields:
> 
>     /usr/bin/docker: line 13: /usr/bin/docker-latest: No such file or
> directory
> 
> root cause: /usr/bin/docker first looks for docker-current, which doesn't
> exist.
> 
> Have you run rpmdiff against this? I suspect rpmdiff might find more
> problems.

I have fixed this in docker-common.sh with a friendlier error message. I'll see what rpmdiff says.

Comment 25 Luwen Su 2017-01-10 02:45:04 UTC
Works well in docker-1.12.5-9.el7.x86_64 , move to verified

Comment 27 errata-xmlrpc 2017-01-17 20:44:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0116.html