Created attachment 1292753 [details] The upgrade logs Description of problem: When docker is upgraded during OCP upgrade, The Pre-pull image always fails for docker daemon was stopped. Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine... Version-Release number of selected component (if applicable): openshift-ansible-3.6.126-1 How reproducible: Always Steps to Reproduce: 1. install Containerized OCP v3.5 with docker-1.12.5 2. Enable docker-1.12.6 repos and upgrade to OCP 3.6 anible-playbook usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.yml 3. check the docker status Actual results: 2. Pre-pull node failed TASK [openshift_node_upgrade : Pre-pull node image] **************************** fatal: [openshift-225.lab.eng.nay.redhat.com]: FAILED! => { "changed": false, "cmd": [ "docker", "pull", "openshift3/node:v3.6.116" --why version is 3.6.116? ], "delta": "0:00:00.018752", "end": "2017-06-28 21:31:22.048972", "failed": true, "invocation": { "module_args": { "_raw_params": "docker pull openshift3/node:v3.6.116", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true }, "module_name": "command" }, "rc": 1, "start": "2017-06-28 21:31:22.030220", "warnings": [] } STDERR: Cannot connect to the Docker daemon. Is the docker daemon running on this host? NO MORE HOSTS LEFT ************************************************************* NO MORE HOSTS LEFT ************************************************************* to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry 3. docker was stopped root@container--1 ~]# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─custom.conf Active: inactive (dead) since Wed 2017-06-28 21:31:20 EDT; 16min ago Docs: http://docs.docker.com Main PID: 28627 (code=exited, status=0/SUCCESS) Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.493258259-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}" Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498565945-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d... Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498623050-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d... Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500179635-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}" Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500974372-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d... Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.501013708-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d... Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.424894041-04:00" level=info msg="Processing signal 'terminated'" Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine... Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.431450551-04:00" level=info msg="stopping containerd after receiving terminated" Jun 28 21:31:20 container--1.novalocal systemd[1]: Stopped Docker Application Container Engine. Expected results: Additional info:
Anping, is the deployment container based or AH based?
Jan, It is on RHEL7. By the way, on atomic hosts, the docker could not be upgraded by playbook. To upgrade docker, we must upgrade the entire OS using command 'atomic hosts upgrade' manually.
Upstream PR: https://github.com/openshift/openshift-ansible/pull/4666
docker was updated without error when use openshift-ansible-3.6.135
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716