Bug 1411980 - Cannot start existing containers
Summary: Cannot start existing containers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Antonio Murdaca
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On: 1387831
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-10 22:15 UTC by Micah Abbott
Modified: 2020-04-15 15:04 UTC (History)
25 users (show)

Fixed In Version: docker-1.12.6-1.el7_3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1387831
Environment:
Last Closed: 2017-03-02 19:08:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2876431 0 None None None 2017-01-18 10:05:27 UTC
Red Hat Product Errata RHBA-2017:0406 0 normal SHIPPED_LIVE docker bug fix and enhancement update 2017-03-03 00:06:50 UTC

Description Micah Abbott 2017-01-10 22:15:24 UTC
+++ This bug was initially created as a clone of Bug #1387831 +++

Description of problem:
Cannot start existing containers


Version-Release number of selected component (if applicable):
sh$ rpm -qa "docker*"
docker-1.12.2-3.git15c82b8.fc25.x86_64
docker-common-1.12.2-3.git15c82b8.fc25.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
They are not ideal because I cannot reproduce with newly created container
1. sh# docker start 5279c4f1a2ea
Error response from daemon: shim error: docker-runc not installed on system
Error: failed to start containers: 5279c4f1a2ea

Actual results:
Error response from daemon: shim error: docker-runc not installed on system
Error: failed to start containers: 5279c4f1a2ea

Expected results:
no error reported

Additional info:
sh# rpm -ql docker | grep runc
/usr/libexec/docker/docker-runc-current
sh# rpm -ql docker-common | grep runc

and my workaround is:
sh# cd /usr/local/sbin
sh# ln -s /usr/libexec/docker/docker-runc-current docker-runc

--- Additional comment from Lukas Slebodnik on 2016-10-22 06:15:32 EDT ---

Debug log from docker-containerd
sh# /usr/libexec/docker/docker-containerd-current --listen unix:///run/containerd.sock --shim /usr/libexec/docker/docker-containerd-shim-current --debug
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=4096
DEBU[0000] containerd: read past events                  count=12
DEBU[0000] containerd: supervisor running                cpus=8 memory=15676 runtime=runc runtimeArgs=[] stateDir=/run/containerd
DEBU[0000] containerd: grpc api on /run/containerd.sock



ERRO[0021] containerd: start container                   error=shim error: docker-runc not installed on system id=5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40

--- Additional comment from Lukas Slebodnik on 2016-10-22 06:19:22 EDT ---

Debug log from docker.service:
sh# /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald -s btrfs --debug  DEBU[0000] Warning: could not change group /var/run/docker.sock to docker: Group docker not found
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock)
DEBU[0000] libcontainerd: containerd connection state change: CONNECTING
DEBU[0000] libcontainerd: containerd connection state change: READY
DEBU[0000] Using default logging driver journald
DEBU[0000] Golang's threads limit set to 112680
DEBU[0000] [graphdriver] trying provided driver "btrfs"
DEBU[0000] Using graph driver btrfs
DEBU[0000] Max Concurrent Downloads: 3
DEBU[0000] Max Concurrent Uploads: 5
INFO[0000] Graph migration to content-addressability took 0.00 seconds
DEBU[0000] Loaded container 0449d7df907def40a9e1c563579220132f223acd453e3f0c36afcbee4df4d765

//snip

DEBU[0000] Registering GET, /networks/{id:.*}
DEBU[0000] Registering POST, /networks/create
DEBU[0000] Registering POST, /networks/{id:.*}/connect
DEBU[0000] Registering POST, /networks/{id:.*}/disconnect
DEBU[0000] Registering DELETE, /networks/{id:.*}
INFO[0000] API listen on /var/run/docker.sock





DEBU[0016] Calling POST /v1.24/containers/5279c4f1a2ea/start
INFO[0016] {Action=start, Username=alcik, LoginUID=1000, PID=4446}
DEBU[0016] container mounted via layerStore: /var/lib/docker/btrfs/subvolumes/6467d03ccee5866f531d59276ad439a5417d66339165048fc1ec9dc7276a1c8a
DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge
DEBU[0016] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])
DEBU[0016] Assigning addresses for endpoint broken_container's interface on network bridge
DEBU[0016] Programming external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9)
DEBU[0016] createSpec: cgroupsPath: system.slice:docker:5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40
ERRO[0016] Create container failed with error: shim error: docker-runc not installed on system
DEBU[0016] Revoking external connectivity on endpoint broken_container (96122d5993ad744e6c84622a108e8920844c4766f7c10ef7e60c603941ddcaa9)
DEBU[0016] Releasing addresses for endpoint broken_container's interface on network bridge
DEBU[0016] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)
ERRO[0016] Handler for POST /v1.24/containers/5279c4f1a2ea/start returned error: shim error: docker-runc not installed on system

--- Additional comment from Lukas Slebodnik on 2016-10-22 06:23:39 EDT ---

sh# docker inspect 5279c4f1a2ea
[
    {
        "Id": "5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40",
        "Created": "2016-02-26T15:45:24.955957382Z",
        "Path": "bash",
        "Args": [],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "shim error: docker-runc not installed on system",
            "StartedAt": "2016-10-22T09:59:26.447631798Z",
            "FinishedAt": "2016-10-22T09:59:28.491022428Z"
        },
        "Image": "sha256:cbebc878a8f322e110e6b923e0e9752866b805f6d09643d6d0fc0b56d875b813",
        "ResolvConfPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hostname",
        "HostsPath": "/var/lib/docker/containers/5279c4f1a2eaf153cedaf4f037cc230fd363a9222bded17ecff8e8438a492f40/hosts",
        "LogPath": "",
        "Name": "/broken_container",
        "RestartCount": 0,
        "Driver": "btrfs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/dev/shm:/sssd_workdir",
                "/home/user:/home/user"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label:disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 2147483648,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "0-4",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 4294967296,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "btrfs",
            "Data": null
        },
        "Mounts": [
            {
                "Source": "/home/user",
                "Destination": "/home/user",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/dev/shm",
                "Destination": "/sssd_workdir",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "5279c4f1a2ea",
            "Domainname": "",
            "User": "user",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "USER=user",
                "KRB5CCNAME=KEYRING:persistent:1000:1000"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "lslebodn/beaker",
            "Volumes": null,
            "WorkingDir": "/home/user",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a5b75a2313a674c45b16641c236fe78a372d9bf8927ff5df3c1ff04e9ff704a7",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "/var/run/docker/netns/a5b75a2313a6",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "7a159955a7e83f6e27cc4c5d456da296bc21d7a79c2491888d6886403c2a57ee",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

--- Additional comment from Daniel Walsh on 2016-10-22 06:27:09 EDT ---

Antonio could this be a path issue?

--- Additional comment from Antonio Murdaca on 2016-10-22 10:40:28 EDT ---

So this is likely a container created with a released version of Docker which had a bug I later fixed (it was a bug in how the Docker service was configured).

Unfortunately I'm not sure we can do anything about this exept telling people to re-create the container (which will then use the correct oci runtime path).

Again, this is an issue with a specific Docker version released some time ago, this version set a wrong runtime path.

--- Additional comment from Antonio Murdaca on 2016-10-22 10:42:02 EDT ---

Sorry didn't mean to close it tbh

--- Additional comment from Antonio Murdaca on 2016-10-22 10:45:32 EDT ---

But yeah, what I described before is probably what's happening. Your container is from February 2016 and it has been likely created with a bugged Docker.

--- Additional comment from Lukas Slebodnik on 2016-10-22 13:27:04 EDT ---

I checked my dnf history and I docker was upgraded to 1:1.10.2-1.git86e59a5.fc23.x86_64 on Wed Feb 24 21:50:56 2016.

And the container was created few days later
"Created": "2016-02-26T15:45:24.955957382Z"

Can you confirm that I used buggy version of docker?

--- Additional comment from Pavel Alexeev on 2016-10-27 08:41:27 EDT ---

I'v got after update:

$ docker start gitlab-runner
Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory
Error: failed to start containers: gitlab-runner

Container re-creation does not help.

$ sudo dnf history info 1071
…
Transaction ID : 1071
Begin time     : Thu Oct 27 11:45:07 2016
Begin rpmdb    : 4652:421d8fe32fec89cd51f8e8d5ba603e43ccceb52a
End time       :            11:45:48 2016 (41 seconds)
End rpmdb      : 4653:4fb3939aedd9f8df602295b1f5737ee0e5582f20
User           :  <pasha>
Return-Code    : Success
Command Line   : upgrade --refresh
Transaction performed with:
…
    Obsoleting container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64      @fedora
    Install    container-selinux-2:1.12.2-3.git15c82b8.fc25.x86_64      @fedora
    Upgraded   docker-2:1.12.1-13.git9a3752d.fc25.x86_64                @@commandline
    Upgrade           2:1.12.2-3.git15c82b8.fc25.x86_64                 @fedora
    Install    docker-common-2:1.12.2-3.git15c82b8.fc25.x86_64          @fedora
    Obsoleted  docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64        @@commandline
    Upgraded   docker-v1.10-migrator-2:1.12.1-13.git9a3752d.fc25.x86_64 @@commandline
    Upgrade                          2:1.12.2-3.git15c82b8.fc25.x86_64  @fedora

--- Additional comment from Bohuslav "Slavek" Kabrda on 2016-11-09 09:01:05 EST ---

I'm experiencing same problems as Pavel described in comment 9, except I'm trying to create and run a container, not running an existing one.

$ docker run -ti fedora:24 bash
/usr/bin/docker-current: Error response from daemon: fork/exec /usr/libexec/docker/docker-containerd-shim: no such file or directory.

$ rpm -q docker
docker-1.12.3-5.git9a594b9.fc25.x86_64

$ sudo dnf history info
[sudo] password for bkabrda: 
Transaction ID : 608
Begin time     : Wed Nov  9 14:41:05 2016
Begin rpmdb    : 2880:38d14aec1b5b24dbb13521e66060c665bae7ffee
End time       :            14:41:52 2016 (47 seconds)
End rpmdb      : 2882:e86f1267f8a154faf4b3067720c74c11c90bbece
User           : Slavek Kabrda <bkabrda>
Return-Code    : Success
Command Line   : update --enablerepo=updates-testing
Transaction performed with:
    Installed     dnf-1.1.10-3.fc25.noarch @updates-testing
    Installed     rpm-4.13.0-1.fc25.x86_64 @updates-testing
Packages Altered:
    Install    skopeo-containers-0.1.14-5.git550a480.fc25.x86_64   @updates-testing
    Obsoleting container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing
    Install    container-selinux-2:1.12.3-5.git9a594b9.fc25.x86_64 @updates-testing
    Upgraded   docker-2:1.12.1-13.git9a3752d.fc25.x86_64           @fedora
    Upgrade           2:1.12.3-5.git9a594b9.fc25.x86_64            @updates-testing
    Install    docker-common-2:1.12.3-5.git9a594b9.fc25.x86_64     @updates-testing
    Obsoleted  docker-selinux-2:1.12.1-13.git9a3752d.fc25.x86_64   @fedora

--- Additional comment from Bohuslav "Slavek" Kabrda on 2016-11-09 09:09:15 EST ---

Ok, I just found out that I had to restart both docker and docker-containerd and everything started working again.

--- Additional comment from Lukas Slebodnik on 2016-11-09 09:10:50 EST ---

(In reply to Bohuslav "Slavek" Kabrda from comment #10)
> I'm experiencing same problems as Pavel described in comment 9, except I'm
> trying to create and run a container, not running an existing one.
> 
> $ docker run -ti fedora:24 bash
> /usr/bin/docker-current: Error response from daemon: fork/exec
> /usr/libexec/docker/docker-containerd-shim: no such file or directory.
> 
> $ rpm -q docker
> docker-1.12.3-5.git9a594b9.fc25.x86_64
> 
This BZ is about starting already existing contianer.

You want to start new container.

IIRC you hit other bug with upgrading docker. Following command should help :-)
systemctl restart docker.service docker-containerd.service

--- Additional comment from Micah Abbott on 2017-01-10 15:57:38 EST ---

I believe I ran into this issue when I upgraded my RHEL Atomic Host from 7.3.1 to 7.3.2.

On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers.


I had a private registry created on my system like so:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

After I upgraded to RHELAH 7.3.2 (docker 1.12.5-8), my registry container did not restart as expected:

# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS               NAMES
776b6478ced3        registry:2          "/entrypoint.sh /etc/"   3 minutes ago       Exited (128) 24 seconds ago                       registry


I had to delete the existing container and then run the 'docker run' command again to start a fresh container.


# docker rm registry
registry
# docker run -d -p 5000:5000 --restart=always --name registry registry:2
40cc1deb93cd8447b7b3dcc1b31e9a51aba00cf15e5e5d4711525154355e3922
# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
40cc1deb93cd        registry:2          "/entrypoint.sh /etc/"   4 seconds ago       Up 2 seconds        0.0.0.0:5000->5000/tcp   registry


In my experience, it seems like this only affects containers that have exposed ports.  

For example, when I use 'atomic run docker.io/cockpit/ws' using docker 1.10 backend, then try to 'docker start' the container with docker 1.12, I encounter no problems.

--- Additional comment from Micah Abbott on 2017-01-10 15:59:12 EST ---

(In reply to Micah Abbott from comment #13)
> I believe I ran into this issue when I upgraded my RHEL Atomic Host from
> 7.3.1 to 7.3.2.
> 
> On RHELAH 7.3.1, docker 1.10.3-59 was used to create the containers.

Oops, I see this was originally opened against Fedora/docker but it looks like the same problem exists on RHEL.  I'll have to clone this bug for RHEL.

Comment 1 Antonio Murdaca 2017-01-10 22:29:18 UTC
Micah, thanks for reopening this. The Fedora one got stuck because we didn't have any deterministic way to reproduce. Could you highlight in detail how could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned vm), then? Thanks a lot in advance for the help.

Comment 2 Lukas Slebodnik 2017-01-11 09:17:01 UTC
(In reply to Antonio Murdaca from comment #1)
> Micah, thanks for reopening this. The Fedora one got stuck because we didn't
> have any deterministic way to reproduce. Could you highlight in detail how
> could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned
> vm), then? Thanks a lot in advance for the help.

That is not a true.
@see
https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c5
You wrote there is nothing to do except re-create a container

Unfortunately, there wansn't any reply from docker guys since
https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c8

Comment 3 Antonio Murdaca 2017-01-11 10:05:49 UTC
(In reply to Lukas Slebodnik from comment #2)
> (In reply to Antonio Murdaca from comment #1)
> > Micah, thanks for reopening this. The Fedora one got stuck because we didn't
> > have any deterministic way to reproduce. Could you highlight in detail how
> > could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned
> > vm), then? Thanks a lot in advance for the help.
> 
> That is not a true.
> @see
> https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c5
> You wrote there is nothing to do except re-create a container

Right, it's worth another investigation though if Micah could provide a stable reproducer. 

> 
> Unfortunately, there wansn't any reply from docker guys since
> https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c8

Comment 6 Antonio Murdaca 2017-01-12 19:39:08 UTC
Finally figure this out with https://github.com/docker/docker/pull/30106 (https://bugzilla.redhat.com/show_bug.cgi?id=1387831 is fixed as well with that patch). I'll wait for the upstream review before backporting that to our projectatomic/docker and rebuild RHEL/Fedora. Micah, Lukas thanks for the help (and Lukas above all for the patience!)

Comment 7 Antonio Murdaca 2017-01-12 22:38:09 UTC
Micah, I've pushed the fix to https://github.com/projectatomic/docker/commits/docker-1.12.6. Do you have any way to test it out by rebuilding docker in the atomic host (or pulling a custom docker binary in the compose)? Maybe ask Lokesh to scratch build docker just for testing this out. Let me know if you need any help anyway.

Comment 9 Micah Abbott 2017-01-18 15:36:09 UTC
(In reply to Antonio Murdaca from comment #7)
> Micah, I've pushed the fix to
> https://github.com/projectatomic/docker/commits/docker-1.12.6. Do you have
> any way to test it out by rebuilding docker in the atomic host (or pulling a
> custom docker binary in the compose)? Maybe ask Lokesh to scratch build
> docker just for testing this out. Let me know if you need any help anyway.

The easiest way for me to test this on Atomic Host would be via an RPM.  I *should* be able to just 'ostree admin unlock' and install the newer version of docker.

If that doesn't work, I could still use the RPM to make a custom compose, although that would be a little more work.

Comment 11 Micah Abbott 2017-01-18 16:53:18 UTC
Lokesh marked this as fixed in that 1.12.6 build, but I am unable to verify it on Atomic Host.

I had a 'registry' container created in 1.10 (RHELAH 7.3.1) then upgraded to RHELAH 7.3.2.  I used 'ostree admin unlock --hotfix' to create a bootable deployment and upgraded the 'docker' packages to 1.12.6.

After rebooting into the hotfixed deployment, I still observed the same error about the missing 'docker-runc'.



# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.2 (2017-01-13 22:00:41)
        Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
        OSName: rhel-atomic-host

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.1 (2016-11-30 02:14:24)
        Commit: 42cfe1ca3305defb16dfd59cd0be5c539f19ea720dba861ed11e13941423ae86
        OSName: rhel-atomic-host

# ostree admin unlock --hotfix
Copying /etc changes: 40 modified, 4 removed, 87 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 765.9 MB
Hotfix mode enabled.  A writable overlayfs is now mounted on /usr
for this booted deployment.  A non-hotfixed clone has been created
as the non-default rollback target.

# ls
anaconda-ks.cfg                            docker-1.12.6-1.el7.x86_64.rpm         docker-common-1.12.6-1.el7.x86_64.rpm      docker-novolume-plugin-1.12.6-1.el7.x86_64.rpm   original-ks.cfg
container-selinux-1.12.6-1.el7.x86_64.rpm  docker-client-1.12.6-1.el7.x86_64.rpm  docker-lvm-plugin-1.12.6-1.el7.x86_64.rpm  docker-rhel-push-plugin-1.12.6-1.el7.x86_64.rpm

# rpm -Uhv *rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-common-2:1.12.6-1.el7     ################################# [  7%]
   2:docker-client-2:1.12.6-1.el7     ################################# [ 14%]
   3:docker-rhel-push-plugin-2:1.12.6-################################# [ 21%]
   4:container-selinux-2:1.12.6-1.el7 ################################# [ 29%]
   5:docker-2:1.12.6-1.el7            ################################# [ 36%]
   6:docker-lvm-plugin-2:1.12.6-1.el7 ################################# [ 43%]
   7:docker-novolume-plugin-2:1.12.6-1################################# [ 50%]
Cleaning up / removing...
   8:docker-novolume-plugin-2:1.12.5-1################################# [ 57%]
   9:docker-lvm-plugin-2:1.12.5-14.el7################################# [ 64%]
  10:docker-2:1.12.5-14.el7           ################################# [ 71%]
  11:docker-client-2:1.12.5-14.el7    ################################# [ 79%]
  12:docker-common-2:1.12.5-14.el7    ################################# [ 86%]
  13:container-selinux-2:1.12.5-14.el7################################# [ 93%]
  14:docker-rhel-push-plugin-2:1.12.5-################################# [100%]

# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.2 (2017-01-13 22:00:41)
        Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
        OSName: rhel-atomic-host
      Unlocked: hotfix

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.2 (2017-01-13 22:00:41)
        Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
        OSName: rhel-atomic-host

# systemctl reboot

# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.2 (2017-01-13 22:00:41)
        Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
        OSName: rhel-atomic-host
      Unlocked: hotfix

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3.2 (2017-01-13 22:00:41)
        Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
        OSName: rhel-atomic-host
# rpm -q docker
docker-1.12.6-1.el7.x86_64

# docker info | grep Version
Server Version: 1.12.6
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Kernel Version: 3.10.0-514.6.1.el7.x86_64

# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                       PORTS               NAMES
fee30ab9bac6        registry:2          "/entrypoint.sh /etc/"   10 minutes ago      Exited (128) 9 minutes ago                       registry

# journalctl -b -u docker --no-pager
-- Logs begin at Thu 2017-01-12 14:54:30 UTC, end at Wed 2017-01-18 16:49:07 UTC. --
Jan 18 16:46:46 rhel-atomic-7.2-test systemd[1]: Starting Docker Application Container Engine...
Jan 18 16:46:46 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:46.291807962Z" level=info msg="libcontainerd: new containerd process, pid: 2087"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.433733255Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.435096467Z" level=info msg="Loading containers: start."
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: .time="2017-01-18T16:46:47.527329063Z" level=info msg="Firewalld running: false"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.966568604Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.450963968Z" level=error msg="containerd: start container" error="shim error: docker-runc not installed on system" id=fee30ab9bac6b22dba759cbdeb49e81c51b7f4dad62f9c6d4aa77746221abb4d
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.451985773Z" level=error msg="containerd: deleting container" error="exec: \"docker-runc\": executable file not found in $PATH: \"\""
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.453366216Z" level=error msg="Create container failed with error: shim error: docker-runc not installed on system"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571109277Z" level=error msg="Failed to start container fee30ab9bac6b22dba759cbdeb49e81c51b7f4dad62f9c6d4aa77746221abb4d: shim error: docker-runc not installed on system"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571193023Z" level=info msg="Loading containers: done."
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571449561Z" level=info msg="Daemon has completed initialization"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571482217Z" level=info msg="Docker daemon" commit="037a2f5/1.12.6" graphdriver=devicemapper version=1.12.6
Jan 18 16:46:48 rhel-atomic-7.2-test systemd[1]: Started Docker Application Container Engine.
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.581341119Z" level=info msg="API listen on /var/run/docker.sock"

Comment 12 Antonio Murdaca 2017-01-18 16:58:38 UTC
Micah, Lokesh did marked this ON_QA, not VERIFIED, moving it back I guess.

Also, you did already boot in 7.3.2 and the container configuration was never updated. You need to boot into 7.3.2 with _already_ the patched docker in order to verify this. The new docker won't just fix this if you previously used the bugged docker (unfortunately it was not shipped I guess).

Comment 13 Micah Abbott 2017-01-18 17:02:27 UTC
(In reply to Antonio Murdaca from comment #12)
> Micah, Lokesh did marked this ON_QA, not VERIFIED, moving it back I guess.

Sorry, I saw the 'Fixed In Version: docker-1.12.6-1.el7_3' and over-reacted.

> Also, you did already boot in 7.3.2 and the container configuration was
> never updated. You need to boot into 7.3.2 with _already_ the patched docker
> in order to verify this. The new docker won't just fix this if you
> previously used the bugged docker (unfortunately it was not shipped I guess).

OK, I'll need to create a custom compose or maybe use the internal 'autobrew' stream.

Comment 14 Lokesh Mandvekar 2017-01-18 17:10:45 UTC
(In reply to Micah Abbott from comment #13)
> Sorry, I saw the 'Fixed In Version: docker-1.12.6-1.el7_3' and over-reacted.

That's how I've been notifying what build the fix was included in, once available in brew :)

Let me know if there's a canonical way to do this.

Comment 15 Micah Abbott 2017-01-18 17:33:33 UTC
I took a pointer from @runcom and just used a RHEL Server system to test the fix.

After upgrading to docker 1.12.6, the registry container that was created using docker 1.10 successfully started upon the start of the docker service.

I guess I can mark this as VERIFIED now.


# rpm -q docker
docker-1.10.3-59.el7.x86_64

# docker run -d -p 5000:5000 --restart=always --name registry registry:2
Unable to find image 'registry:2' locally
Trying to pull repository registry.access.redhat.com/registry ... 
unknown: Not Found
Trying to pull repository docker.io/library/registry ... 
2: Pulling from docker.io/library/registry
b7f33cc0b48e: Pull complete 
46730e1e05c9: Pull complete 
458210699647: Pull complete 
0cf045fea0fd: Pull complete 
b78a03aa98b7: Pull complete 
Digest: sha256:0e40793ad06ac099ba63b5a8fae7a83288e64b50fe2eafa2b59741de85fd3b97
Status: Downloaded newer image for docker.io/registry:2
69a939a55a69a4662fe673f14a03ea5fa221724b0d84c92ef84f8abf7292de0f

# docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
docker.io/registry   2                   d1e32b95d8e8        12 hours ago        33.17 MB

# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
69a939a55a69        registry:2          "/entrypoint.sh /etc/"   17 seconds ago      Up 15 seconds       0.0.0.0:5000->5000/tcp   registry

# systemctl stop docker

# rpm -Uhv *rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:docker-common-2:1.12.6-1.el7     ################################# [ 11%]
   2:docker-client-2:1.12.6-1.el7     ################################# [ 22%]
   3:docker-rhel-push-plugin-2:1.12.6-################################# [ 33%]
   4:container-selinux-2:1.12.6-1.el7 ################################# [ 44%]
   5:docker-2:1.12.6-1.el7            ################################# [ 56%]
Cleaning up / removing...
   6:docker-2:1.10.3-59.el7           ################################# [ 67%]
   7:container-selinux-2:1.12.5-14.el7################################# [ 78%]
   8:docker-common-2:1.10.3-59.el7    ################################# [ 89%]
   9:docker-rhel-push-plugin-2:1.10.3-################################# [100%]

# systemctl start docker

# docker info | grep Version
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Server Version: 1.12.6
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Kernel Version: 3.10.0-514.6.1.el7.x86_64
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

# docker images -a
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
docker.io/registry   2                   d1e32b95d8e8        12 hours ago        33.17 MB

# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
69a939a55a69        registry:2          "/entrypoint.sh /etc/"   4 minutes ago       Up 13 seconds       0.0.0.0:5000->5000/tcp   registry

Comment 21 errata-xmlrpc 2017-03-02 19:08:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0406.html


Note You need to log in before you can comment on or make changes to this bug.