Description of problem: Run upgrade from 3.6 to 3.7 against containerzied ha cluster, when docker token is experired, set oreg_auth_credentials_replace=true,upgrade failed at task [Verify containers are available for upgrade] when authenticated registry updated its key. fatal: []: FAILED! => {"changed": false, "cmd": ["docker", "pull", "openshift3/ose:v3.7.0-0.158.0"], "delta": "0:00:02.563704", "end": "2017-10-19 04:00:47.702619", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2017-10-19 04:00:45.138915", "stderr": "unauthorized: authentication required", "stderr_lines": ["unauthorized: authentication required"], "stdout": "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... \nTrying to pull repository registry.access.redhat.com/openshift3/ose ... \nTrying to pull repository docker.io/openshift3/ose ... ", "stdout_lines": ["Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose ... ", "Trying to pull repository registry.access.redhat.com/openshift3/ose ... ", "Trying to pull repository docker.io/openshift3/ose ... "]} Failure summary: 1. Hosts: x.x.x.x Play: Verify upgrade targets Task: Verify containers are available for upgrade Message: non-zero return code Task [Create credentials for docker cli registry auth] was skipped from upgrade log. Docker login with new password to update .docker/config.json on all hosts can workaround. Version-Release number of the following components: openshift-ansible-3.7.0-0.158.0.git.0.8941dd5.el7.noarch openshift-ansible-roles-3.7.0-0.158.0.git.0.8941dd5.el7.noarch How reproducible: always Steps to Reproduce: 1.Containerized install 3.6 ha cluster 2.Edit hosts file to update registry password when authenticated registry updated its key(A->B). <--snip--> oreg_auth_credentials_replace=true oreg_auth_password=B <--snip--> 3.Upgrade ocp v3.6 to v3.7. #ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_7/upgrade.yml Actual results: Upgrade failed due to can not pull images. Expected results: Upgrade succeed. Additional info: Please attach logs from ansible-playbook with the -vvv flag
PR Created: https://github.com/openshift/openshift-ansible/pull/5814
Verified on openshift-ansible-3.7.0-0.189.0. once set oreg_auth_credentials_replace=true, the upgrade finished.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188