Bug 1411980
| Summary: | Cannot start existing containers | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Micah Abbott <miabbott> |
| Component: | docker | Assignee: | Antonio Murdaca <amurdaca> |
| Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | adimania, admiller, amurdaca, bkabrda, dwalsh, eminguez, extras-qa, ichavero, jcajka, jchaloup, lslebodn, lsm5, marianne, miabbott, miminar, mpatel, mwysocki, myllynen, nalin, obockows, pahan, riek, rvdwees, twaugh, vbatts |
| Target Milestone: | rc | Keywords: | Extras, Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | docker-1.12.6-1.el7_3 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1387831 | Environment: | |
| Last Closed: | 2017-03-02 19:08:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1387831 | ||
| Bug Blocks: | |||
|
Description
Micah Abbott
2017-01-10 22:15:24 UTC
Micah, thanks for reopening this. The Fedora one got stuck because we didn't have any deterministic way to reproduce. Could you highlight in detail how could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned vm), then? Thanks a lot in advance for the help. (In reply to Antonio Murdaca from comment #1) > Micah, thanks for reopening this. The Fedora one got stuck because we didn't > have any deterministic way to reproduce. Could you highlight in detail how > could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned > vm), then? Thanks a lot in advance for the help. That is not a true. @see https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c5 You wrote there is nothing to do except re-create a container Unfortunately, there wansn't any reply from docker guys since https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c8 (In reply to Lukas Slebodnik from comment #2) > (In reply to Antonio Murdaca from comment #1) > > Micah, thanks for reopening this. The Fedora one got stuck because we didn't > > have any deterministic way to reproduce. Could you highlight in detail how > > could I reproduce this? I'll start with an rhelah 7.1 (brand new provisioned > > vm), then? Thanks a lot in advance for the help. > > That is not a true. > @see > https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c5 > You wrote there is nothing to do except re-create a container Right, it's worth another investigation though if Micah could provide a stable reproducer. > > Unfortunately, there wansn't any reply from docker guys since > https://bugzilla.redhat.com/show_bug.cgi?id=1387831#c8 Finally figure this out with https://github.com/docker/docker/pull/30106 (https://bugzilla.redhat.com/show_bug.cgi?id=1387831 is fixed as well with that patch). I'll wait for the upstream review before backporting that to our projectatomic/docker and rebuild RHEL/Fedora. Micah, Lukas thanks for the help (and Lukas above all for the patience!) Micah, I've pushed the fix to https://github.com/projectatomic/docker/commits/docker-1.12.6. Do you have any way to test it out by rebuilding docker in the atomic host (or pulling a custom docker binary in the compose)? Maybe ask Lokesh to scratch build docker just for testing this out. Let me know if you need any help anyway. (In reply to Antonio Murdaca from comment #7) > Micah, I've pushed the fix to > https://github.com/projectatomic/docker/commits/docker-1.12.6. Do you have > any way to test it out by rebuilding docker in the atomic host (or pulling a > custom docker binary in the compose)? Maybe ask Lokesh to scratch build > docker just for testing this out. Let me know if you need any help anyway. The easiest way for me to test this on Atomic Host would be via an RPM. I *should* be able to just 'ostree admin unlock' and install the newer version of docker. If that doesn't work, I could still use the RPM to make a custom compose, although that would be a little more work. Lokesh marked this as fixed in that 1.12.6 build, but I am unable to verify it on Atomic Host.
I had a 'registry' container created in 1.10 (RHELAH 7.3.1) then upgraded to RHELAH 7.3.2. I used 'ostree admin unlock --hotfix' to create a bootable deployment and upgraded the 'docker' packages to 1.12.6.
After rebooting into the hotfixed deployment, I still observed the same error about the missing 'docker-runc'.
# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.2 (2017-01-13 22:00:41)
Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
OSName: rhel-atomic-host
rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.1 (2016-11-30 02:14:24)
Commit: 42cfe1ca3305defb16dfd59cd0be5c539f19ea720dba861ed11e13941423ae86
OSName: rhel-atomic-host
# ostree admin unlock --hotfix
Copying /etc changes: 40 modified, 4 removed, 87 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 765.9 MB
Hotfix mode enabled. A writable overlayfs is now mounted on /usr
for this booted deployment. A non-hotfixed clone has been created
as the non-default rollback target.
# ls
anaconda-ks.cfg docker-1.12.6-1.el7.x86_64.rpm docker-common-1.12.6-1.el7.x86_64.rpm docker-novolume-plugin-1.12.6-1.el7.x86_64.rpm original-ks.cfg
container-selinux-1.12.6-1.el7.x86_64.rpm docker-client-1.12.6-1.el7.x86_64.rpm docker-lvm-plugin-1.12.6-1.el7.x86_64.rpm docker-rhel-push-plugin-1.12.6-1.el7.x86_64.rpm
# rpm -Uhv *rpm
Preparing... ################################# [100%]
Updating / installing...
1:docker-common-2:1.12.6-1.el7 ################################# [ 7%]
2:docker-client-2:1.12.6-1.el7 ################################# [ 14%]
3:docker-rhel-push-plugin-2:1.12.6-################################# [ 21%]
4:container-selinux-2:1.12.6-1.el7 ################################# [ 29%]
5:docker-2:1.12.6-1.el7 ################################# [ 36%]
6:docker-lvm-plugin-2:1.12.6-1.el7 ################################# [ 43%]
7:docker-novolume-plugin-2:1.12.6-1################################# [ 50%]
Cleaning up / removing...
8:docker-novolume-plugin-2:1.12.5-1################################# [ 57%]
9:docker-lvm-plugin-2:1.12.5-14.el7################################# [ 64%]
10:docker-2:1.12.5-14.el7 ################################# [ 71%]
11:docker-client-2:1.12.5-14.el7 ################################# [ 79%]
12:docker-common-2:1.12.5-14.el7 ################################# [ 86%]
13:container-selinux-2:1.12.5-14.el7################################# [ 93%]
14:docker-rhel-push-plugin-2:1.12.5-################################# [100%]
# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.2 (2017-01-13 22:00:41)
Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
OSName: rhel-atomic-host
Unlocked: hotfix
rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.2 (2017-01-13 22:00:41)
Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
OSName: rhel-atomic-host
# systemctl reboot
# atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.2 (2017-01-13 22:00:41)
Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
OSName: rhel-atomic-host
Unlocked: hotfix
rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Version: 7.3.2 (2017-01-13 22:00:41)
Commit: 96826a0d917d7ff10f9fd0289581649f2ffbddd76f3b80efd3d95cc11915cacb
OSName: rhel-atomic-host
# rpm -q docker
docker-1.12.6-1.el7.x86_64
# docker info | grep Version
Server Version: 1.12.6
Library Version: 1.02.135-RHEL7 (2016-11-16)
Kernel Version: 3.10.0-514.6.1.el7.x86_64
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fee30ab9bac6 registry:2 "/entrypoint.sh /etc/" 10 minutes ago Exited (128) 9 minutes ago registry
# journalctl -b -u docker --no-pager
-- Logs begin at Thu 2017-01-12 14:54:30 UTC, end at Wed 2017-01-18 16:49:07 UTC. --
Jan 18 16:46:46 rhel-atomic-7.2-test systemd[1]: Starting Docker Application Container Engine...
Jan 18 16:46:46 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:46.291807962Z" level=info msg="libcontainerd: new containerd process, pid: 2087"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.433733255Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.435096467Z" level=info msg="Loading containers: start."
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: .time="2017-01-18T16:46:47.527329063Z" level=info msg="Firewalld running: false"
Jan 18 16:46:47 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:47.966568604Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.450963968Z" level=error msg="containerd: start container" error="shim error: docker-runc not installed on system" id=fee30ab9bac6b22dba759cbdeb49e81c51b7f4dad62f9c6d4aa77746221abb4d
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.451985773Z" level=error msg="containerd: deleting container" error="exec: \"docker-runc\": executable file not found in $PATH: \"\""
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.453366216Z" level=error msg="Create container failed with error: shim error: docker-runc not installed on system"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571109277Z" level=error msg="Failed to start container fee30ab9bac6b22dba759cbdeb49e81c51b7f4dad62f9c6d4aa77746221abb4d: shim error: docker-runc not installed on system"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571193023Z" level=info msg="Loading containers: done."
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571449561Z" level=info msg="Daemon has completed initialization"
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.571482217Z" level=info msg="Docker daemon" commit="037a2f5/1.12.6" graphdriver=devicemapper version=1.12.6
Jan 18 16:46:48 rhel-atomic-7.2-test systemd[1]: Started Docker Application Container Engine.
Jan 18 16:46:48 rhel-atomic-7.2-test dockerd-current[2078]: time="2017-01-18T16:46:48.581341119Z" level=info msg="API listen on /var/run/docker.sock"
Micah, Lokesh did marked this ON_QA, not VERIFIED, moving it back I guess. Also, you did already boot in 7.3.2 and the container configuration was never updated. You need to boot into 7.3.2 with _already_ the patched docker in order to verify this. The new docker won't just fix this if you previously used the bugged docker (unfortunately it was not shipped I guess). (In reply to Antonio Murdaca from comment #12) > Micah, Lokesh did marked this ON_QA, not VERIFIED, moving it back I guess. Sorry, I saw the 'Fixed In Version: docker-1.12.6-1.el7_3' and over-reacted. > Also, you did already boot in 7.3.2 and the container configuration was > never updated. You need to boot into 7.3.2 with _already_ the patched docker > in order to verify this. The new docker won't just fix this if you > previously used the bugged docker (unfortunately it was not shipped I guess). OK, I'll need to create a custom compose or maybe use the internal 'autobrew' stream. (In reply to Micah Abbott from comment #13) > Sorry, I saw the 'Fixed In Version: docker-1.12.6-1.el7_3' and over-reacted. That's how I've been notifying what build the fix was included in, once available in brew :) Let me know if there's a canonical way to do this. I took a pointer from @runcom and just used a RHEL Server system to test the fix. After upgrading to docker 1.12.6, the registry container that was created using docker 1.10 successfully started upon the start of the docker service. I guess I can mark this as VERIFIED now. # rpm -q docker docker-1.10.3-59.el7.x86_64 # docker run -d -p 5000:5000 --restart=always --name registry registry:2 Unable to find image 'registry:2' locally Trying to pull repository registry.access.redhat.com/registry ... unknown: Not Found Trying to pull repository docker.io/library/registry ... 2: Pulling from docker.io/library/registry b7f33cc0b48e: Pull complete 46730e1e05c9: Pull complete 458210699647: Pull complete 0cf045fea0fd: Pull complete b78a03aa98b7: Pull complete Digest: sha256:0e40793ad06ac099ba63b5a8fae7a83288e64b50fe2eafa2b59741de85fd3b97 Status: Downloaded newer image for docker.io/registry:2 69a939a55a69a4662fe673f14a03ea5fa221724b0d84c92ef84f8abf7292de0f # docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/registry 2 d1e32b95d8e8 12 hours ago 33.17 MB # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 69a939a55a69 registry:2 "/entrypoint.sh /etc/" 17 seconds ago Up 15 seconds 0.0.0.0:5000->5000/tcp registry # systemctl stop docker # rpm -Uhv *rpm Preparing... ################################# [100%] Updating / installing... 1:docker-common-2:1.12.6-1.el7 ################################# [ 11%] 2:docker-client-2:1.12.6-1.el7 ################################# [ 22%] 3:docker-rhel-push-plugin-2:1.12.6-################################# [ 33%] 4:container-selinux-2:1.12.6-1.el7 ################################# [ 44%] 5:docker-2:1.12.6-1.el7 ################################# [ 56%] Cleaning up / removing... 6:docker-2:1.10.3-59.el7 ################################# [ 67%] 7:container-selinux-2:1.12.5-14.el7################################# [ 78%] 8:docker-common-2:1.10.3-59.el7 ################################# [ 89%] 9:docker-rhel-push-plugin-2:1.10.3-################################# [100%] # systemctl start docker # docker info | grep Version WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. Server Version: 1.12.6 Library Version: 1.02.135-RHEL7 (2016-11-16) Kernel Version: 3.10.0-514.6.1.el7.x86_64 WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled # docker images -a REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/registry 2 d1e32b95d8e8 12 hours ago 33.17 MB # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 69a939a55a69 registry:2 "/entrypoint.sh /etc/" 4 minutes ago Up 13 seconds 0.0.0.0:5000->5000/tcp registry Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0406.html |