Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1503995 - Fail to pull ose image during upgrade due to docker auth token was not updated even if oreg_auth_credentials_replace=true is set
Fail to pull ose image during upgrade due to docker auth token was not update...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade (Show other bugs)
3.7.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.7.0
Assigned To: Michael Gugino
liujia
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-10-19 05:18 EDT by liujia
Modified: 2017-11-28 17:18 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-28 17:18:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-28 21:34:54 EST

  None (edit)
Description liujia 2017-10-19 05:18:18 EDT
Description of problem:
Run upgrade from 3.6 to 3.7 against containerzied ha cluster, when docker token is experired, set oreg_auth_credentials_replace=true,upgrade failed at task [Verify containers are available for upgrade] when authenticated registry updated its key.

fatal: []: FAILED! => {"changed": false, "cmd": ["docker", "pull", "openshift3/ose:v3.7.0-0.158.0"], "delta": "0:00:02.563704", "end": "2017-10-19 04:00:47.702619", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-10-19 04:00:45.138915", "stderr": "unauthorized: authentication required", "stderr_lines": ["unauthorized: authentication required"], "stdout": "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository docker.io/openshift3/ose ... ", "stdout_lines": ["Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository docker.io/openshift3/ose ... "]}

Failure summary:
  1. Hosts:    x.x.x.x
     Play:     Verify upgrade targets
     Task:     Verify containers are available for upgrade
     Message:  non-zero return code

Task [Create credentials for docker cli registry auth] was skipped from upgrade log.

Docker login with new password to update .docker/config.json on all hosts can workaround.

Version-Release number of the following components:
openshift-ansible-3.7.0-0.158.0.git.0.8941dd5.el7.noarch
openshift-ansible-roles-3.7.0-0.158.0.git.0.8941dd5.el7.noarch

How reproducible:
always

Steps to Reproduce:
1.Containerized install 3.6 ha cluster
2.Edit hosts file to update registry password when authenticated registry updated its key(A->B).
<--snip-->
oreg_auth_credentials_replace=true
oreg_auth_password=B
<--snip-->
3.Upgrade ocp v3.6 to v3.7.
#ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_7/upgrade.yml


Actual results:
Upgrade failed due to can not pull images.

Expected results:
Upgrade succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 4 Michael Gugino 2017-10-19 11:32:48 EDT
PR Created: https://github.com/openshift/openshift-ansible/pull/5814
Comment 6 Anping Li 2017-11-04 23:06:03 EDT
Verified on openshift-ansible-3.7.0-0.189.0. once set oreg_auth_credentials_replace=true, the upgrade finished.
Comment 9 errata-xmlrpc 2017-11-28 17:18:08 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.