Description of problem: Cannot start existing containers Version-Release number of selected component (if applicable): sh$ rpm -qa "docker*" docker-1.12.2-3.git15c82b8.fc25.x86_64 docker-common-1.12.2-3.git15c82b8.fc25.x86_64 How reproducible: Deterministic Steps to Reproduce: They are not ideal because I cannot reproduce with newly created container 1. sh# docker start 5279c4f1a2ea Error response from daemon: shim error: docker-runc not installed on system Error: failed to start containers: 5279c4f1a2ea Actual results: Error response from daemon: shim error: docker-runc not installed on system Error: failed to start containers: 5279c4f1a2ea Expected results: no error reported Additional info: sh# rpm -ql docker | grep runc /usr/libexec/docker/docker-runc-current sh# rpm -ql docker-common | grep runc and my workaround is: sh# cd /usr/local/sbin sh# ln -s /usr/libexec/docker/docker-runc-current docker-runc
Debug log from docker-containerd sh# /usr/libexec/docker/docker-containerd-current --listen unix:///run/containerd.sock --shim /usr/libexec/docker/docker-containerd-shim-current --debug WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096 DEBU[0000] containerd: read past events count=12 DEBU[0000] containerd: supervisor running cpus=8 memory=15676 runtime=runc runtimeArgs=[] stateDir=/run/containerd DEBU[0000] containerd: grpc api on /run/containerd.sock ERRO[0021] containerd: start container error=shim error: docker-runc not installed on system id=5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40
Debug log from docker.service: sh# /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald -s btrfs --debug DEBU[0000] Warning: could not change group /var/run/docker.sock to docker: Group docker not found DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock) DEBU[0000] libcontainerd: containerd connection state change: CONNECTING DEBU[0000] libcontainerd: containerd connection state change: READY DEBU[0000] Using default logging driver journald DEBU[0000] Golang's threads limit set to 112680 DEBU[0000] [graphdriver] trying provided driver "btrfs" DEBU[0000] Using graph driver btrfs DEBU[0000] Max Concurrent Downloads: 3 DEBU[0000] Max Concurrent Uploads: 5 INFO[0000] Graph migration to content-addressability took 0.00 seconds DEBU[0000] Loaded container 0449d7df907def40a9e1c563579220132f223acd453e3f0c36afcbee4df4d765 //snip DEBU[0000] Registering GET, /networks/{id:.*} DEBU[0000] Registering POST, /networks/create DEBU[0000] Registering POST, /networks/{id:.*}/connect DEBU[0000] Registering POST, /networks/{id:.*}/disconnect DEBU[0000] Registering DELETE, /networks/{id:.*} INFO[0000] API listen on /var/run/docker.sock DEBU[0016] Calling POST /v1.24/containers/5279c4f1a2ea/start INFO[0016] {Action=start, Username=alcik, LoginUID=1000, PID=4446} DEBU[0016] container mounted via layerStore: /var/lib/docker/btrfs/subvolumes/6467d03ccee5866f531d59276ad439a5417d66339165048fc1ec9dc7276a1c8a DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge DEBU[0016] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[]) DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge DEBU[0016] Programming external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9) DEBU[0016] createSpec: cgroupsPath: system.slice:docker:5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40 ERRO[0016] Create container failed with error: shim error: docker-runc not installed on system DEBU[0016] Revoking external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9) DEBU[0016] Releasing addresses for endpoint broken_container's interface on network bridge DEBU[0016] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2) ERRO[0016] Handler for POST /v1.24/containers/5279c4f1a2ea/start returned error: shim error: docker-runc not installed on system
sh# docker inspect 5279c4f1a2ea [ { "Id": "5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40", "Created": "2016-02-26T15:45:24.955957382Z", "Path": "bash", "Args": [], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 128, "Error": "shim error: docker-runc not installed on system", "StartedAt": "2016-10-22T09:59:26.447631798Z", "FinishedAt": "2016-10-22T09:59:28.491022428Z" }, "Image": "sha256:cbebc878a8f322e110e6b923e0e9752866b805f6d09643d6d0fc0b56d875b813", "ResolvConfPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/resolv.conf", "HostnamePath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hostname", "HostsPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hosts", "LogPath": "", "Name": "/broken_container", "RestartCount": 0, "Driver": "btrfs", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/dev/shm:/sssd_workdir", "/home/user:/home/user" ], "ContainerIDFile": "", "LogConfig": { "Type": "journald", "Config": {} }, "NetworkMode": "default", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": true, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "label:disable" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 2147483648, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpusetCpus": "0-4", "CpusetMems": "", "Devices": [], "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 4294967296, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Name": "btrfs", "Data": null }, "Mounts": [ { "Source": "/home/user", "Destination": "/home/user", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Source": "/dev/shm", "Destination": "/sssd_workdir", "Mode": "", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "5279c4f1a2ea", "Domainname": "", "User": "user", "AttachStdin": true, "AttachStdout": true, "AttachStderr": true, "Tty": true, "OpenStdin": true, "StdinOnce": true, "Env": [ "PATH=/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "USER=user", "KRB5CCNAME=KEYRING:persistent:1000:1000" ], "Cmd": [ "bash" ], "Image": "lslebodn/beaker", "Volumes": null, "WorkingDir": "/home/user", "Entrypoint": null, "OnBuild": null, "Labels": {}, "StopSignal": "SIGTERM" }, "NetworkSettings": { "Bridge": "", "SandboxID": "a5b75a2313a674c45b16641c236fe78a372d9bf8927ff5df3c1ff04e9ff704a7", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": null, "SandboxKey": "/var/run/docker/netns/a5b75a2313a6", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "7a159955a7e83f6e27cc4c5d456da296bc21d7a79c2491888d6886403c2a57ee", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "" } } } } ]
Antonio could this be a path issue?
So this is likely a container created with a released version of Docker which had a bug I later fixed (it was a bug in how the Docker service was configured). Unfortunately I'm not sure we can do anything about this exept telling people to re-create the container (which will then use the correct oci runtime path). Again, this is an issue with a specific Docker version released some time ago, this version set a wrong runtime path.
Sorry didn't mean to close it tbh
But yeah, what I described before is probably what's happening. Your container is from February 2016 and it has been likely created with a bugged Docker.
I checked my dnf history and I docker was upgraded to 1:1.10.2-1.git86e59a5.fc23.x86_64 on Wed Feb 24 21:50:56 2016. And the container was created few days later "Created": "2016-02-26T15:45:24.955957382Z" Can you confirm that I used buggy version of docker?
I'v got after update: $ docker start gitlab-runner Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory Error: failed to start containers: gitlab-runner Container re-creation does not help. $ sudo dnf history info 1071 … Transaction ID : 1071 Begin time : Thu Oct 27 11:45:07 2016 Begin rpmdb : 4652:421d8fe32fec89cd51f8e8d5ba603e43ccceb52a End time : 11:45:48 2016 (41 seconds) End rpmdb : 4653:4fb3939aedd9f8df602295b1f5737ee0e5582f20 User : <pasha> Return-Code : Success Command Line : upgrade --refresh Transaction performed with: … Obsoleting container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64 @fedora Install container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64 @fedora Upgraded docker-2:1.12.1-13.git9a3752d.fc25.x86_64 @@commandline Upgrade 2:1.12.2-3.git15c82b8.fc25.x86_64 @fedora Install docker-common-2:1.12.2-3.git15c82b8.fc25.x86_64 @fedora Obsoleted docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64 @@commandline Upgraded docker-v1.10-migrator-2:1.12.1-13.git9a3752d.fc25.x86_64 @@commandline Upgrade 2:1.12.2-3.git15c82b8.fc25.x86_64 @fedora
I'm experiencing same problems as Pavel described in comment 9, except I'm trying to create and run a container, not running an existing one. $ docker run -ti fedora:24 bash /usr/bin/docker-current: Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory. $ rpm -q docker docker-1.12.3-5.git9a594b9.fc25.x86_64 $ sudo dnf history info [sudo] password for bkabrda: Transaction ID : 608 Begin time : Wed Nov 9 14:41:05 2016 Begin rpmdb : 2880:38d14aec1b5b24dbb13521e66060c665bae7ffee End time : 14:41:52 2016 (47 seconds) End rpmdb : 2882:e86f1267f8a154faf4b3067720c74c11c90bbece User : Slavek Kabrda <bkabrda> Return-Code : Success Command Line : update --enablerepo=updates-testing Transaction performed with: Installed dnf-1.1.10-3.fc25.noarch @updates-testing Installed rpm-4.13.0-1.fc25.x86_64 @updates-testing Packages Altered: Install skopeo-containers-0.1.14-5.git550a480.fc25.x86_64 @updates-testing Obsoleting container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing Install container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing Upgraded docker-2:1.12.1-13.git9a3752d.fc25.x86_64 @fedora Upgrade 2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing Install docker-common-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing Obsoleted docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64 @fedora
Ok, I just found out that I had to restart both docker and docker-containerd and everything started working again.
(In reply to Bohuslav "Slavek" Kabrda from comment #10) > I'm experiencing same problems as Pavel described in comment 9, except I'm > trying to create and run a container, not running an existing one. > > $ docker run -ti fedora:24 bash > /usr/bin/docker-current: Error response from daemon: fork/exec > /usr/libexec/docker/docker-containerd-shim: no such file or directory. > > $ rpm -q docker > docker-1.12.3-5.git9a594b9.fc25.x86_64 > This BZ is about starting already existing contianer. You want to start new container. IIRC you hit other bug with upgrading docker. Following command should help :-) systemctl restart docker.service docker-containerd.service
I believe I ran into this issue when I upgraded my RHEL Atomic Host from 7.3.1 to 7.3.2. On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers. I had a private registry created on my system like so: docker run -d -p 5000:5000 --restart=always --name registry registry:2 After I upgraded to RHELAH 7.3.2 (docker 1.12.5-8), my registry container did not restart as expected: # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 776b6478ced3 registry:2 "/entrypoint.sh /etc/" 3 minutes ago Exited (128) 24 seconds ago registry I had to delete the existing container and then run the 'docker run' command again to start a fresh container. # docker rm registry registry # docker run -d -p 5000:5000 --restart=always --name registry registry:2 40cc1deb93cd8447b7b3dcc1b31e9a51aba00cf15e5e5d4711525154355e3922 # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 40cc1deb93cd registry:2 "/entrypoint.sh /etc/" 4 seconds ago Up 2 seconds 0.0.0.0:5000->5000/tcp registry In my experience, it seems like this only affects containers that have exposed ports. For example, when I use 'atomic run docker.io/cockpit/ws' using docker 1.10 backend, then try to 'docker start' the container with docker 1.12, I encounter no problems.
(In reply to Micah Abbott from comment #13) > I believe I ran into this issue when I upgraded my RHEL Atomic Host from > 7.3.1 to 7.3.2. > > On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers. Oops, I see this was originally opened against Fedora/docker but it looks like the same problem exists on RHEL. I'll have to clone this bug for RHEL.
A very basic reproducer for this one just for the record: - with docker-1.10.x: docker run -d -p 5000:5000 --restart=always --name registry registry:2 - upgrade to docker-1.12.6 - docker ps should show the "registry" container previously started (without the fix, the last point will fail to show that container as running)
docker-1.12.6-4.gitf499e8b.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4909cf95eb
docker-1.12.6-4.gitf499e8b.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4909cf95eb
docker-1.12.6-4.gitf499e8b.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.