Bug 1769222

Summary: Fail to intall due to a wrong test image in TASK [container_runtime : Create for any additional registries]
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: liujia <jiajliu>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jialiu
Version: 3.11.0Keywords: Regression
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-18 14:52:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liujia 2019-11-06 08:10:00 UTC
Description of problem:
TASK [container_runtime : Create for any additional registries] ****************
Wednesday 06 November 2019  10:13:24 +0800 (0:00:00.131)       0:02:41.522 **** 
FAILED - RETRYING: Create for any additional registries (3 retries left).
...
FAILED - RETRYING: Create for any additional registries (1 retries left).
failed: [ci-vm-10-0-149-232.hosted.upshift.rdu2.redhat.com] (item={'host': u'registry.redhat.io', 'password': u'****', 'user': u'****'}) => {"attempts": 3, "changed": false, "item": {"host": "registry.redhat.io", "password": "****", "user": "****"}, "msg": "time=\"2019-11-05T21:13:45-05:00\" level=fatal msg=\"Error reading manifest v3.11 in registry.redhat.io/rh-osbs/openshift3-ose-pod: unknown: Not Found\" \n", "state": "unknown"}
failed: [ci-vm-10-0-150-104.hosted.upshift.rdu2.redhat.com] (item={'host': u'registry.redhat.io', 'password': u'****', 'user': u'****'}) => {"attempts": 3, "changed": false, "item": {"host": "registry.redhat.io", "password": "****", "user": "****"}, "msg": "time=\"2019-11-05T21:13:44-05:00\" level=fatal msg=\"Error reading manifest v3.11 in registry.redhat.io/rh-osbs/openshift3-ose-pod: unknown: Not Found\" \n", "state": "unknown"}
failed: [ci-vm-10-0-151-95.hosted.upshift.rdu2.redhat.com] (item={'host': u'registry.redhat.io', 'password': u'****', 'user': u'****'}) => {"attempts": 3, "changed": false, "item": {"host": "registry.redhat.io", "password": "****", "user": "****"}, "msg": "time=\"2019-11-05T21:13:44-05:00\" level=fatal msg=\"Error reading manifest v3.11 in registry.redhat.io/rh-osbs/openshift3-ose-pod: unknown: Not Found\" \n", "state": "unknown"}

===========================debug info======================================
Hosts file:
...
oreg_url=registry-proxy.engineering.redhat.com/rh-osbs/openshift3-ose-${component}:${version}
openshift_additional_registry_credentials=[{ 'host': 'registry.redhat.io', 'user': "{{ lookup('env','REG_AUTH_USER2') }}", 'password': "{{ lookup('env','REG_AUTH_PASSWORD2') }}" }]
...

The test_image is not correct.
 {
    "attempts": 3, 
    "changed": false, 
    "invocation": {
        "module_args": {
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "path": "/root/.docker", 
            "proxy_vars": "  ", 
            "registry": "registry.redhat.io", 
            "test_image": "rh-osbs/openshift3-ose-pod:v3.11", 
            "test_login": true, 
            "test_timeout": 60, 
            "tls_verify": true, 
            "username": "11941554|aosqeaosqe"
        }
    }, 
    "item": {
        "host": "registry.redhat.io", 
        "password": "...", 
        "user": "..."
    }, 
    "msg": "time=\"2019-11-06T02:31:18-05:00\" level=fatal msg=\"Error reading manifest v3.11 in registry.redhat.io/rh-osbs/openshift3-ose-pod: unknown: Not Found\" \n", 
    "state": "unknown"
}

Above test_image came from a wrong l_docker_creds_test_image in roles/openshift_facts/defaults/main.yml.

Version-Release number of the following components:
openshift-ansible-3.11.154-1.git.0.7a11cbe.el7.noarch.rpm

How reproducible:
always

Steps to Reproduce:
1. Trigger installation with openshift_additional_registry_credentials set in hosts file and the image schema in oreg_url is different from openshift_additional_registry_credentials.
2.
3.

Actual results:
install fail

Expected results:
install succeed

Additional info:
openshift-ansible-3.11.153-1.git.0.71f7a7c.el7.noarch.rpm works well.
https://github.com/openshift/openshift-ansible/commit/41a566d736e94fc057d873bbcd6143a5a95025fb#diff-23ff377caf1c3db9840ce07eede4eda0
https://github.com/openshift/openshift-ansible/commit/fb484d363605a3f8899992f6d5ec3c998e73451d#diff-23ff377caf1c3db9840ce07eede4eda0

Comment 1 Johnny Liu 2019-11-06 08:36:49 UTC
This bug is introduced when fixing https://bugzilla.redhat.com/show_bug.cgi?id=1729994. 


So it is a regression bug.

Comment 5 errata-xmlrpc 2019-11-18 14:52:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3817