Bug 1602120 - [Term based registry] Installer fails to authenticate against prod secure registry
Summary: [Term based registry] Installer fails to authenticate against prod secure reg...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.11.0
Assignee: Michael Gugino
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1612104
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-17 20:28 UTC by Vikas Laad
Modified: 2018-11-16 06:06 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-11 07:21:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
installation log with inventory file embedded (107.39 KB, text/plain)
2018-07-20 07:48 UTC, Johnny Liu
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3682911 0 None None None 2018-11-16 06:06:49 UTC
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:22:03 UTC

Description Vikas Laad 2018-07-17 20:28:00 UTC
Description of problem: Installer fails when trying to use Authenticated registry with following error

<ec2-52-33-135-189.us-west-2.compute.amazonaws.com> (0, '\r\n{"exception": "  File \\"/tmp/ansible_PzdEUT/ansible_module_docker_creds.py\\", line 141, in validate_registry_login\\n    urllib.request.urlopen(req)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 154, in urlopen\\n    return opener.open(url, data, timeout)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 437, in open\\n    response = meth(req, response)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 550, in http_response\\n    \'http\', request, response, code, msg, hdrs)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 475, in error\\n    return self._call_chain(*args)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 409, in _call_chain\\n    result = func(*args)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 558, in http_error_default\\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\\n", "changed": false, "failed": true, "state": "unknown", "invocation": {"module_args": {"username": "prod-user-name", "test_login": "True", "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "registry": "registry.redhat.io", "path": "/root/.docker"}}, "msg": "HTTP Error 401: Unauthorized"}\r\n', 'Shared connection to ec2-52-33-135-189.us-west-2.compute.amazonaws.com closed.\r\n')
fatal: [ec2-52-33-135-189.us-west-2.compute.amazonaws.com]: FAILED! => {
    "failed": true, 
    "msg": "The conditional check 'crt_oreg_auth_credentials_create.rc == 0' failed. The error was: error while evaluating conditional (crt_oreg_auth_credentials_create.rc == 0): 'dict object' has no attribute 'rc'"
}


Version-Release number of the following components:
openshift-ansible 0e5dd09aa1d4bc9f938d8ef30a60b42c13358fbb
rpm -q ansible 
ansible-2.4.5.0-1.el7ae.noarch
ansible --version
ansible 2.4.5.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jun 12 2018, 10:42:23) [GCC 4.8.5 20150623 (Red Hat 4.8.5-34)]

How reproducible: Only with prod authenticated registry

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

<ec2-52-33-135-189.us-west-2.compute.amazonaws.com> (0, '\r\n{"exception": "  File \\"/tmp/ansible_PzdEUT/ansible_module_docker_creds.py\\", line 141, in validate_registry_login\\n    urllib.request.urlopen(req)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 154, in urlopen\\n    return opener.open(url, data, timeout)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 437, in open\\n    response = meth(req, response)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 550, in http_response\\n    \'http\', request, response, code, msg, hdrs)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 475, in error\\n    return self._call_chain(*args)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 409, in _call_chain\\n    result = func(*args)\\n  File \\"/usr/lib64/python2.7/urllib2.py\\", line 558, in http_error_default\\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\\n", "changed": false, "failed": true, "state": "unknown", "invocation": {"module_args": {"username": "prod-user-name", "test_login": "True", "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "registry": "registry.redhat.io", "path": "/root/.docker"}}, "msg": "HTTP Error 401: Unauthorized"}\r\n', 'Shared connection to ec2-52-33-135-189.us-west-2.compute.amazonaws.com closed.\r\n')
fatal: [ec2-52-33-135-189.us-west-2.compute.amazonaws.com]: FAILED! => {
    "failed": true, 
    "msg": "The conditional check 'crt_oreg_auth_credentials_create.rc == 0' failed. The error was: error while evaluating conditional (crt_oreg_auth_credentials_create.rc == 0): 'dict object' has no attribute 'rc'"
}


Expected results: Install should succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 4 Scott Dodson 2018-07-17 20:35:40 UTC
Aside from I think we're validating incorrect variables.

https://github.com/openshift/openshift-ansible/pull/9237

Comment 5 Michael Gugino 2018-07-18 13:19:49 UTC
Will also need this patch: https://github.com/openshift/openshift-ansible/pull/9244

Currently, oreg_host is equal to '' if not oreg_url is not explicitly set in inventory; this is the value used for which registry to create credentials to.

Patch updates to account for new enterprise reg.

Comment 9 Johnny Liu 2018-07-20 07:46:43 UTC
I also hit the same issue, openshift-ansible-3.11.0-0.7.0.git.0.6e3e78eNone.noarch + ansible-2.6.

Inventory file and installation log will be attached soon.

This is blocking QE's testing.

Comment 10 Johnny Liu 2018-07-20 07:48:37 UTC
Created attachment 1464871 [details]
installation log with inventory file embedded

Comment 11 Johnny Liu 2018-07-20 08:17:05 UTC
I also tested it with the latest openshift-ansible master branch (the last commit id: ec178734d6e555a87f66e2cc9ced23d854b5c9ba), also reproduce.

Comment 12 Michael Gugino 2018-07-20 14:01:25 UTC
PR Created: https://github.com/openshift/openshift-ansible/pull/9286

Comment 13 Michael Gugino 2018-07-20 20:27:08 UTC
PR merged in master.

Comment 14 Xiaoli Tian 2018-07-25 02:37:37 UTC
The PR is available since openshift-ansible-3.11.0-0.8.0

Comment 18 Michael Gugino 2018-07-25 18:42:52 UTC
I need to get some patches out to resolve this.

For now, workaround, set following inv var:

oreg_test_login: False

Comment 19 Michael Gugino 2018-07-27 01:01:30 UTC
PR Created: https://github.com/openshift/openshift-ansible/pull/9349

Comment 20 Johnny Liu 2018-08-01 03:12:47 UTC
Retest with openshift-ansible-3.11.0-0.10.0.git.0.91bb588None.noarch + "registry.dev.redhat.io" registry + "oreg_test_login=false" setting in inventory file, still reproduced.

Comment 21 liujia 2018-08-01 10:25:56 UTC
Also hit the issue on openshift-ansible-3.11.0-0.10.0.git.0.91bb588None.noarch when do upgrade test with "registry.redhat.io" registry and correct oreg_auth_user+oreg_auth_password.

Comment 23 Michael Gugino 2018-08-06 17:00:14 UTC
PR created: https://github.com/openshift/openshift-ansible/pull/9443

Comment 24 Johnny Liu 2018-08-07 06:45:04 UTC
Based on comment 22, found workaround, removing testblocker keyword.

Comment 26 Scott Dodson 2018-08-09 12:04:49 UTC
https://github.com/openshift/openshift-ansible/pull/9490 follow up fixes

Comment 27 liujia 2018-08-10 02:58:19 UTC
This issue should be fixed on openshift-ansible-3.11.0-0.13.0.git.0.16dc599None.noarch. But upgrade still failed for another blocker bug 1612144.

Comment 28 Scott Dodson 2018-08-14 21:24:58 UTC
Should be in openshift-ansible-3.11.0-0.15.0

Comment 29 Johnny Liu 2018-08-15 08:58:51 UTC
Verified this bug with openshift-ansible-3.11.0-0.15.0.git.0.842d3d1None + "5318290|aosqeaosqe" as oreg_auth_user, and PASS.

TASK [container_runtime : Create credentials for docker cli registry auth (alternative)] ***
Wednesday 15 August 2018  16:06:51 +0800 (0:00:02.721)       0:01:56.806 ****** 
FAILED - RETRYING: Create credentials for docker cli registry auth (alternative) (3 retries left).
FAILED - RETRYING: Create credentials for docker cli registry auth (alternative) (2 retries left).
changed: [host-8-252-102.host.centralci.eng.rdu2.redhat.com] => {"attempts": 3, "changed": true, "rc": 0}

Comment 31 errata-xmlrpc 2018-10-11 07:21:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.