Bug 1466095

Summary: Pre-pull node images fail if docker is upgraded during OCP upgrade
Product: OpenShift Container Platform Reporter: Anping Li <anli>
Component: Cluster Version OperatorAssignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: anli, aos-bugs, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:28:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The upgrade logs none

Description Anping Li 2017-06-29 02:31:28 UTC 
Created attachment 1292753 [details]
The upgrade logs

Description of problem:
When docker is upgraded during OCP upgrade, The Pre-pull image always fails for docker daemon was stopped.

Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine...


Version-Release number of selected component (if applicable):
openshift-ansible-3.6.126-1

How reproducible:
Always

Steps to Reproduce:
1. install Containerized OCP v3.5 with docker-1.12.5
2. Enable docker-1.12.6 repos and upgrade to OCP 3.6
   anible-playbook usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.yml
3. check the docker status

Actual results:
2. Pre-pull node failed
TASK [openshift_node_upgrade : Pre-pull node image] ****************************
fatal: [openshift-225.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "docker", 
        "pull", 
        "openshift3/node:v3.6.116" --why version is 3.6.116?
    ], 
    "delta": "0:00:00.018752", 
    "end": "2017-06-28 21:31:22.048972", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "docker pull openshift3/node:v3.6.116", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-06-28 21:31:22.030220", 
    "warnings": []
}

STDERR:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry


3. docker was stopped 
root@container--1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: inactive (dead) since Wed 2017-06-28 21:31:20 EDT; 16min ago
     Docs: http://docs.docker.com
 Main PID: 28627 (code=exited, status=0/SUCCESS)

Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.493258259-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}"
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498565945-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498623050-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500179635-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}"
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500974372-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.501013708-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.424894041-04:00" level=info msg="Processing signal 'terminated'"
Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine...
Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.431450551-04:00" level=info msg="stopping containerd after receiving terminated"
Jun 28 21:31:20 container--1.novalocal systemd[1]: Stopped Docker Application Container Engine.
Expected results:


Additional info:
Comment 1 Jan Chaloupka 2017-06-30 15:22:16 UTC 
Anping, is the deployment container based or AH based?
Comment 2 Anping Li 2017-07-03 01:50:03 UTC 
Jan, It is on RHEL7.  
By the way, on atomic hosts, the docker could not be upgraded by playbook.  To upgrade docker, we must upgrade the entire OS using command 'atomic hosts upgrade' manually.
Comment 3 Jan Chaloupka 2017-07-03 13:01:45 UTC 
Upstream PR: https://github.com/openshift/openshift-ansible/pull/4666
Comment 5 Anping Li 2017-07-06 07:57:44 UTC 
docker was updated without error when use openshift-ansible-3.6.135
Comment 7 errata-xmlrpc 2017-08-10 05:28:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716