Bug 1466095 - Pre-pull node images fail if docker is upgraded during OCP upgrade
Pre-pull node images fail if docker is upgraded during OCP upgrade
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Jan Chaloupka
Anping Li
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-28 22:31 EDT by Anping Li
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:28:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
The upgrade logs (186.39 KB, application/x-gzip)
2017-06-28 22:31 EDT, Anping Li
no flags Details

  None (edit)
Description Anping Li 2017-06-28 22:31:28 EDT
Created attachment 1292753 [details]
The upgrade logs

Description of problem:
When docker is upgraded during OCP upgrade, The Pre-pull image always fails for docker daemon was stopped.

Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine...


Version-Release number of selected component (if applicable):
openshift-ansible-3.6.126-1

How reproducible:
Always

Steps to Reproduce:
1. install Containerized OCP v3.5 with docker-1.12.5
2. Enable docker-1.12.6 repos and upgrade to OCP 3.6
   anible-playbook usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.yml
3. check the docker status

Actual results:
2. Pre-pull node failed
TASK [openshift_node_upgrade : Pre-pull node image] ****************************
fatal: [openshift-225.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "docker", 
        "pull", 
        "openshift3/node:v3.6.116" --why version is 3.6.116?
    ], 
    "delta": "0:00:00.018752", 
    "end": "2017-06-28 21:31:22.048972", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "docker pull openshift3/node:v3.6.116", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-06-28 21:31:22.030220", 
    "warnings": []
}

STDERR:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry


3. docker was stopped 
root@container--1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─custom.conf
   Active: inactive (dead) since Wed 2017-06-28 21:31:20 EDT; 16min ago
     Docs: http://docs.docker.com
 Main PID: 28627 (code=exited, status=0/SUCCESS)

Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.493258259-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}"
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498565945-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.498623050-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500179635-04:00" level=info msg="{Action=kill, LoginUID=4294967295, PID=28815}"
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.500974372-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:18 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:18.501013708-04:00" level=error msg="Handler for POST /v1.24/containers/f66d58f58b02df2534b908ae434205293963f24f98d5fc428d...
Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.424894041-04:00" level=info msg="Processing signal 'terminated'"
Jun 28 21:31:19 container--1.novalocal systemd[1]: Stopping Docker Application Container Engine...
Jun 28 21:31:19 container--1.novalocal dockerd-current[28627]: time="2017-06-28T21:31:19.431450551-04:00" level=info msg="stopping containerd after receiving terminated"
Jun 28 21:31:20 container--1.novalocal systemd[1]: Stopped Docker Application Container Engine.
Expected results:


Additional info:
Comment 1 Jan Chaloupka 2017-06-30 11:22:16 EDT
Anping, is the deployment container based or AH based?
Comment 2 Anping Li 2017-07-02 21:50:03 EDT
Jan, It is on RHEL7.  
By the way, on atomic hosts, the docker could not be upgraded by playbook.  To upgrade docker, we must upgrade the entire OS using command 'atomic hosts upgrade' manually.
Comment 3 Jan Chaloupka 2017-07-03 09:01:45 EDT
Upstream PR: https://github.com/openshift/openshift-ansible/pull/4666
Comment 5 Anping Li 2017-07-06 03:57:44 EDT
docker was updated without error when use openshift-ansible-3.6.135
Comment 7 errata-xmlrpc 2017-08-10 01:28:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.