Bug 1387831

Debug log from docker-containerd
sh# /usr/libexec/docker/docker-containerd-current --listen unix:///run/containerd.sock --shim /usr/libexec/docker/docker-containerd-shim-current --debug
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=4096
DEBU[0000] containerd: read past events                  count=12
DEBU[0000] containerd: supervisor running                cpus=8 memory=15676 runtime=runc runtimeArgs=[] stateDir=/run/containerd
DEBU[0000] containerd: grpc api on /run/containerd.sock



ERRO[0021] containerd: start container                   error=shim error: docker-runc not installed on system id=5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40

Debug log from docker.service:
sh# /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald -s btrfs --debug  DEBU[0000] Warning: could not change group /var/run/docker.sock to docker: Group docker not found
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock)
DEBU[0000] libcontainerd: containerd connection state change: CONNECTING
DEBU[0000] libcontainerd: containerd connection state change: READY
DEBU[0000] Using default logging driver journald
DEBU[0000] Golang's threads limit set to 112680
DEBU[0000] [graphdriver] trying provided driver "btrfs"
DEBU[0000] Using graph driver btrfs
DEBU[0000] Max Concurrent Downloads: 3
DEBU[0000] Max Concurrent Uploads: 5
INFO[0000] Graph migration to content-addressability took 0.00 seconds
DEBU[0000] Loaded container 0449d7df907def40a9e1c563579220132f223acd453e3f0c36afcbee4df4d765

//snip

DEBU[0000] Registering GET, /networks/{id:.*}
DEBU[0000] Registering POST, /networks/create
DEBU[0000] Registering POST, /networks/{id:.*}/connect
DEBU[0000] Registering POST, /networks/{id:.*}/disconnect
DEBU[0000] Registering DELETE, /networks/{id:.*}
INFO[0000] API listen on /var/run/docker.sock





DEBU[0016] Calling POST /v1.24/containers/5279c4f1a2ea/start
INFO[0016] {Action=start, Username=alcik, LoginUID=1000, PID=4446}
DEBU[0016] container mounted via layerStore: /var/lib/docker/btrfs/subvolumes/6467d03ccee5866f531d59276ad439a5417d66339165048fc1ec9dc7276a1c8a
DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge
DEBU[0016] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])
DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge
DEBU[0016] Programming external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9)
DEBU[0016] createSpec: cgroupsPath: system.slice:docker:5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40
ERRO[0016] Create container failed with error: shim error: docker-runc not installed on system
DEBU[0016] Revoking external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9)
DEBU[0016] Releasing addresses for endpoint broken_container's interface on network bridge
DEBU[0016] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)
ERRO[0016] Handler for POST /v1.24/containers/5279c4f1a2ea/start returned error: shim error: docker-runc not installed on system

sh# docker inspect 5279c4f1a2ea
[
    {
        "Id": "5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40",
        "Created": "2016-02-26T15:45:24.955957382Z",
        "Path": "bash",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "shim error: docker-runc not installed on system",
            "StartedAt": "2016-10-22T09:59:26.447631798Z",
            "FinishedAt": "2016-10-22T09:59:28.491022428Z"
        },
        "Image": "sha256:cbebc878a8f322e110e6b923e0e9752866b805f6d09643d6d0fc0b56d875b813",
        "ResolvConfPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hostname",
        "HostsPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hosts",
        "LogPath": "",
        "Name": "/broken_container",
        "RestartCount": 0,
        "Driver": "btrfs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/dev/shm:/sssd_workdir",
                "/home/user:/home/user"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label:disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 2147483648,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "0-4",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 4294967296,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "btrfs",
            "Data": null
        },
        "Mounts": [
            {
                "Source": "/home/user",
                "Destination": "/home/user",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/dev/shm",
                "Destination": "/sssd_workdir",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "5279c4f1a2ea",
            "Domainname": "",
            "User": "user",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "USER=user",
                "KRB5CCNAME=KEYRING:persistent:1000:1000"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "lslebodn/beaker",
            "Volumes": null,
            "WorkingDir": "/home/user",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a5b75a2313a674c45b16641c236fe78a372d9bf8927ff5df3c1ff04e9ff704a7",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "/var/run/docker/netns/a5b75a2313a6",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "7a159955a7e83f6e27cc4c5d456da296bc21d7a79c2491888d6886403c2a57ee",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

Antonio could this be a path issue?

So this is likely a container created with a released version of Docker which had a bug I later fixed (it was a bug in how the Docker service was configured).

Unfortunately I'm not sure we can do anything about this exept telling people to re-create the container (which will then use the correct oci runtime path).

Again, this is an issue with a specific Docker version released some time ago, this version set a wrong runtime path.

Sorry didn't mean to close it tbh

But yeah, what I described before is probably what's happening. Your container is from February 2016 and it has been likely created with a bugged Docker.

I checked my dnf history and I docker was upgraded to 1:1.10.2-1.git86e59a5.fc23.x86_64 on Wed Feb 24 21:50:56 2016.

And the container was created few days later
"Created": "2016-02-26T15:45:24.955957382Z"

Can you confirm that I used buggy version of docker?

I'v got after update:

$ docker start gitlab-runner
Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory
Error: failed to start containers: gitlab-runner

Container re-creation does not help.

$ sudo dnf history info 1071
…
Transaction ID : 1071
Begin time     : Thu Oct 27 11:45:07 2016
Begin rpmdb    : 4652:421d8fe32fec89cd51f8e8d5ba603e43ccceb52a
End time       :            11:45:48 2016 (41 seconds)
End rpmdb      : 4653:4fb3939aedd9f8df602295b1f5737ee0e5582f20
User           :  <pasha>
Return-Code    : Success
Command Line   : upgrade --refresh
Transaction performed with:
…
    Obsoleting container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64      @fedora
    Install    container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64      @fedora
    Upgraded   docker-2:1.12.1-13.git9a3752d.fc25.x86_64                @@commandline
    Upgrade           2:1.12.2-3.git15c82b8.fc25.x86_64                 @fedora
    Install    docker-common-2:1.12.2-3.git15c82b8.fc25.x86_64          @fedora
    Obsoleted  docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64        @@commandline
    Upgraded   docker-v1.10-migrator-2:1.12.1-13.git9a3752d.fc25.x86_64 @@commandline
    Upgrade                          2:1.12.2-3.git15c82b8.fc25.x86_64  @fedora

I'm experiencing same problems as Pavel described in comment 9, except I'm trying to create and run a container, not running an existing one.

$ docker run -ti fedora:24 bash
/usr/bin/docker-current: Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory.

$ rpm -q docker
docker-1.12.3-5.git9a594b9.fc25.x86_64

$ sudo dnf history info
[sudo] password for bkabrda: 
Transaction ID : 608
Begin time     : Wed Nov  9 14:41:05 2016
Begin rpmdb    : 2880:38d14aec1b5b24dbb13521e66060c665bae7ffee
End time       :            14:41:52 2016 (47 seconds)
End rpmdb      : 2882:e86f1267f8a154faf4b3067720c74c11c90bbece
User           : Slavek Kabrda <bkabrda>
Return-Code    : Success
Command Line   : update --enablerepo=updates-testing
Transaction performed with:
    Installed     dnf-1.1.10-3.fc25.noarch @updates-testing
    Installed     rpm-4.13.0-1.fc25.x86_64 @updates-testing
Packages Altered:
    Install    skopeo-containers-0.1.14-5.git550a480.fc25.x86_64   @updates-testing
    Obsoleting container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing
    Install    container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing
    Upgraded   docker-2:1.12.1-13.git9a3752d.fc25.x86_64           @fedora
    Upgrade           2:1.12.3-5.git9a594b9.fc25.x86_64            @updates-testing
    Install    docker-common-2:1.12.3-5.git9a594b9.fc25.x86_64     @updates-testing
    Obsoleted  docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64   @fedora

Ok, I just found out that I had to restart both docker and docker-containerd and everything started working again.

(In reply to Bohuslav "Slavek" Kabrda from comment #10)
> I'm experiencing same problems as Pavel described in comment 9, except I'm
> trying to create and run a container, not running an existing one.
> 
> $ docker run -ti fedora:24 bash
> /usr/bin/docker-current: Error response from daemon: fork/exec
> /usr/libexec/docker/docker-containerd-shim: no such file or directory.
> 
> $ rpm -q docker
> docker-1.12.3-5.git9a594b9.fc25.x86_64
> 
This BZ is about starting already existing contianer.

You want to start new container.

IIRC you hit other bug with upgrading docker. Following command should help :-)
systemctl restart docker.service docker-containerd.service

I believe I ran into this issue when I upgraded my RHEL Atomic Host from 7.3.1 to 7.3.2.

On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers.


I had a private registry created on my system like so:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

After I upgraded to RHELAH 7.3.2 (docker 1.12.5-8), my registry container did not restart as expected:

# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS               NAMES
776b6478ced3        registry:2          "/entrypoint.sh /etc/"   3 minutes ago       Exited (128) 24 seconds ago                       registry


I had to delete the existing container and then run the 'docker run' command again to start a fresh container.


# docker rm registry
registry
# docker run -d -p 5000:5000 --restart=always --name registry registry:2
40cc1deb93cd8447b7b3dcc1b31e9a51aba00cf15e5e5d4711525154355e3922
# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
40cc1deb93cd        registry:2          "/entrypoint.sh /etc/"   4 seconds ago       Up 2 seconds        0.0.0.0:5000->5000/tcp   registry


In my experience, it seems like this only affects containers that have exposed ports.  

For example, when I use 'atomic run docker.io/cockpit/ws' using docker 1.10 backend, then try to 'docker start' the container with docker 1.12, I encounter no problems.

(In reply to Micah Abbott from comment #13)
> I believe I ran into this issue when I upgraded my RHEL Atomic Host from
> 7.3.1 to 7.3.2.
> 
> On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers.

Oops, I see this was originally opened against Fedora/docker but it looks like the same problem exists on RHEL.  I'll have to clone this bug for RHEL.

A very basic reproducer for this one just for the record:

- with docker-1.10.x: docker run -d -p 5000:5000 --restart=always --name registry registry:2
- upgrade to docker-1.12.6
- docker ps should show the "registry" container previously started

(without the fix, the last point will fail to show that container as running)

docker-1.12.6-4.gitf499e8b.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4909cf95eb

docker-1.12.6-4.gitf499e8b.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4909cf95eb

docker-1.12.6-4.gitf499e8b.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.