Bug 1503995 - Fail to pull ose image during upgrade due to docker auth token was not updated even if oreg_auth_credentials_replace=true is set
Summary: Fail to pull ose image during upgrade due to docker auth token was not update...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.7.0
Assignee: Michael Gugino
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-19 09:18 UTC by liujia
Modified: 2017-11-28 22:18 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-28 22:18:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description liujia 2017-10-19 09:18:18 UTC
Description of problem:
Run upgrade from 3.6 to 3.7 against containerzied ha cluster, when docker token is experired, set oreg_auth_credentials_replace=true,upgrade failed at task [Verify containers are available for upgrade] when authenticated registry updated its key.

fatal: []: FAILED! => {"changed": false, "cmd": ["docker", "pull", "openshift3/ose:v3.7.0-0.158.0"], "delta": "0:00:02.563704", "end": "2017-10-19 04:00:47.702619", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-10-19 04:00:45.138915", "stderr": "unauthorized: authentication required", "stderr_lines": ["unauthorized: authentication required"], "stdout": "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository docker.io/openshift3/ose ... ", "stdout_lines": ["Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository docker.io/openshift3/ose ... "]}

Failure summary:
  1. Hosts:    x.x.x.x
     Play:     Verify upgrade targets
     Task:     Verify containers are available for upgrade
     Message:  non-zero return code

Task [Create credentials for docker cli registry auth] was skipped from upgrade log.

Docker login with new password to update .docker/config.json on all hosts can workaround.

Version-Release number of the following components:
openshift-ansible-3.7.0-0.158.0.git.0.8941dd5.el7.noarch
openshift-ansible-roles-3.7.0-0.158.0.git.0.8941dd5.el7.noarch

How reproducible:
always

Steps to Reproduce:
1.Containerized install 3.6 ha cluster
2.Edit hosts file to update registry password when authenticated registry updated its key(A->B).
<--snip-->
oreg_auth_credentials_replace=true
oreg_auth_password=B
<--snip-->
3.Upgrade ocp v3.6 to v3.7.
#ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_7/upgrade.yml


Actual results:
Upgrade failed due to can not pull images.

Expected results:
Upgrade succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 4 Michael Gugino 2017-10-19 15:32:48 UTC
PR Created: https://github.com/openshift/openshift-ansible/pull/5814

Comment 6 Anping Li 2017-11-05 03:06:03 UTC
Verified on openshift-ansible-3.7.0-0.189.0. once set oreg_auth_credentials_replace=true, the upgrade finished.

Comment 9 errata-xmlrpc 2017-11-28 22:18:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.