Bug 1474724

Summary: docker daemon cannot start: exec: "dockerd": executable file not found in $PATH
Product: [Fedora] Fedora Reporter: Paolo Antinori <pantinor>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 26CC: adimania, admiller, amurdaca, dwalsh, fkluknav, ichavero, jcajka, lsm5, marianne, nalin, riek, vbatts
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-26 15:55:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paolo Antinori 2017-07-25 09:13:48 UTC
Description of problem:

Cannot start docker dameon either manually or with systemd scripts


Version-Release number of selected component (if applicable):

docker-1.13.1-21.git27e468e.fc26.x86_64 that comes from updates-testing repo, but it was failing even we the official version 



How reproducible:
Always


Steps to Reproduce:
1.$ sudo docker -D daemon
2.
3.

Actual results:
Command "daemon" is deprecated, and will be removed in Docker 1.16. Please run `dockerd` directly.
exec: "dockerd": executable file not found in $PATH


Expected results:
A docker daemon started


Additional info:

I get the same error if I just try to start systemd docker.service unit:

sudo systemctl start docker


If I look for what package provides dockerd binary I cannot find any:

```
$ sudo dnf provides "dockerd"
Last metadata expiration check: 0:50:54 ago on Tue 25 Jul 2017 10:18:32 CEST.
Error: No Matches found
```

I get the same error even with selinux off.


sudo rpm -qa | ag docker
docker-rhel-push-plugin-1.13.1-21.git27e468e.fc26.x86_64
docker-1.13.1-21.git27e468e.fc26.x86_64
docker-common-1.13.1-21.git27e468e.fc26.x86_64

Comment 1 Antonio Murdaca 2017-07-25 09:17:55 UTC
It's docker-current indeed, we may provide a link i guess.

Comment 2 Antonio Murdaca 2017-07-25 09:18:37 UTC
dockerd-current (typo)

Comment 3 Paolo Antinori 2017-07-25 09:27:34 UTC
Hi Antonio, sorry but I need a little more clarification.

```
11:25:00 (/home/pantinor)$ sudo dockerd-current -D
INFO[0000] SUSE:secrets :: enabled                      
DEBU[0000] docker group found. gid: 981                 
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock) 
Failed to connect to containerd. Please make sure containerd is installed in your PATH or you have specified the correct address. Got error: exec: "docker-containerd": executable file not found in $PATH
```

```
11:24:21 (/home/pantinor)$ sudo systemctl daemon-reload 
11:24:59 (/home/pantinor)$ sudo systemctl list-units | ag docker
  sys-devices-virtual-net-docker0.device                                                                                                                        loaded    active plugged   /sys/devices/virtual/net/docker0                                             
  sys-subsystem-net-devices-docker0.device                                                                                                                      loaded    active plugged   /sys/subsystem/net/devices/docker0                                           
● docker-latest-containerd.service                                                                                                                              not-found active running   docker-latest-containerd.service                                             
● docker-latest-storage-setup.service                                                                                                                           not-found failed failed    docker-latest-storage-setup.service                                          
● docker.service 
```


What's the correct way to start docker daemon?

Comment 4 Paolo Antinori 2017-07-26 14:14:15 UTC
Trying to investigate further. The error message is definitely misleading now that containerd binary has changed name, but a part of that I see something odd here:

```
$ updatedb
$ locate containerd
/usr/lib/systemd/system/docker-containerd.service
/usr/libexec/docker/docker-containerd-current
/usr/libexec/docker/docker-containerd-shim-current


$ sudo  strace dockerd-current -D
[sudo] password for pantinor: 
.
.
.
stat("/var/run/docker/libcontainerd", {st_mode=S_IFDIR|0700, st_size=60, ...}) = 0
openat(AT_FDCWD, "/var/run/docker/libcontainerd/docker-containerd.pid", O_RDWR|O_CREAT|O_CLOEXEC, 0600) = 6
read(6, "", 8)                          = 0
lseek(6, 0, SEEK_SET)                   = 0
ftruncate(6, 0)                         = 0
stat("/sbin/docker-containerd", 0xc42031b2e8) = -1 ENOENT (No such file or directory)
stat("/bin/docker-containerd", 0xc42031b3b8) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/docker-containerd", 0xc42031b488) = -1 ENOENT (No such file or directory)
stat("/usr/bin/docker-containerd", 0xc42031b558) = -1 ENOENT (No such file or directory)
close(6)                                = 0
unlinkat(AT_FDCWD, "/var/run/docker.pid", 0) = 0
write(2, "Failed to connect to containerd."..., 203Failed to connect to containerd. Please make sure containerd is installed in your PATH or you have specified the correct address. Got error: exec: "docker-containerd": executable file not found in $PATH
) = 203
exit_group(1)                           = ?
+++ exited with 1 +++


$ sudo dnf --enablerepo updates-testing  provides **docker-containerd
Last metadata expiration check: 2:01:58 ago on Wed 26 Jul 2017 14:11:12 CEST.
Error: No Matches found


```

is this just a naming issue after some refactoring?

Comment 5 Paolo Antinori 2017-07-26 14:25:47 UTC
And I'm even more confused now:

```
$ sudo docker -D daemon
[sudo] password for pantinor: 
Command "daemon" is deprecated, and will be removed in Docker 1.16. Please run `dockerd` directly.
exec: "dockerd": executable file not found in $PATH



sudo dnf --enablerepo updates-testing  provides **dockerd
Last metadata expiration check: 2:12:25 ago on Wed 26 Jul 2017 14:11:12 CEST.
docker-devel-2:1.13.1-21.git27e468e.fc26.noarch : A golang registry for global request variables (source libraries)
Repo        : updates-testing

docker-latest-devel-2:1.13-32.git27e468e.fc26.noarch : A golang registry for global request variables (source libraries)
Repo        : updates-testing

docker-latest-unit-test-2:1.13-32.git27e468e.fc26.x86_64 : A golang registry for global request variables (source libraries) - for running unit tests
Repo        : updates-testing

docker-unit-test-2:1.13.1-21.git27e468e.fc26.x86_64 : A golang registry for global request variables (source libraries) - for running unit tests
Repo        : updates-testing

ocid-devel-0.2-0.git7d7570e.fc26.noarch : OCI-based implementation of Kubernetes Container Runtime Interface
Repo        : updates-testing

docker-latest-devel-2:1.13-31.git27e468e.fc26.noarch : A golang registry for global request variables (source libraries)
Repo        : updates

docker-latest-unit-test-2:1.13-31.git27e468e.fc26.x86_64 : A golang registry for global request variables (source libraries) - for running unit tests
Repo        : updates

docker-devel-2:1.13.1-19.git27e468e.fc26.noarch : A golang registry for global request variables (source libraries)
Repo        : fedora

docker-latest-devel-2:1.13-30.git51eb16e.fc26.noarch : A golang registry for global request variables (source libraries)
Repo        : fedora

docker-latest-unit-test-2:1.13-30.git51eb16e.fc26.x86_64 : A golang registry for global request variables (source libraries) - for running unit tests
Repo        : fedora

docker-unit-test-2:1.13.1-19.git27e468e.fc26.x86_64 : A golang registry for global request variables (source libraries) - for running unit tests
Repo        : fedora

ocid-devel-0.2-0.git7d7570e.fc26.noarch : OCI-based implementation of Kubernetes Container Runtime Interface
Repo        : fedora


```

Comment 6 Paolo Antinori 2017-07-26 14:32:37 UTC
and if I try to install the only package that is not a `docker-latest` or a docker devel` from fedora (not updates-testing) repo this is what I get:

```
$ sudo dnf install ocid-devel
[sudo] password for pantinor: 
Last metadata expiration check: 2:18:48 ago on Wed 26 Jul 2017 14:11:12 CEST.
Error: 
 Problem: conflicting requests
  - nothing provides golang(github.com/Microsoft/go-winio) needed by ocid-devel-0.2-0.git7d7570e.fc26.noarch

```

Comment 7 Lokesh Mandvekar 2017-07-26 15:40:15 UTC
'systemctl start docker' works just fine for me, could you reinstall docker and retry this?

Also, could you please post your /etc/sysconfig/docker file contents?

Comment 8 Lokesh Mandvekar 2017-07-26 15:41:18 UTC
(In reply to Paolo Antinori from comment #6)
> and if I try to install the only package that is not a `docker-latest` or a
> docker devel` from fedora (not updates-testing) repo this is what I get:
> 
> ```
> $ sudo dnf install ocid-devel
> [sudo] password for pantinor: 
> Last metadata expiration check: 2:18:48 ago on Wed 26 Jul 2017 14:11:12 CEST.
> Error: 
>  Problem: conflicting requests
>   - nothing provides golang(github.com/Microsoft/go-winio) needed by
> ocid-devel-0.2-0.git7d7570e.fc26.noarch
> 
> ```


The golang -devel packages are an abomination right now, I'm hoping they just _go_ away.

Comment 9 Paolo Antinori 2017-07-26 15:54:51 UTC
Hi Lokesh, thanks for looking into this.

I did that a couple of times already without luck.

This last time, after erasing and before installing again, I have performed this set of systemd manual house cleaning that seemed to fix my problem upon reinstallation:

```
rm /etc/systemd/system/docker
systemctl daemon-reload
systemctl reset-failed
```

Thanks again