Bug 1884088 - [3.11] - imageconfig.yml fails with error "Create credentials for registry auth (3 retries left).Result was"
Summary: [3.11] - imageconfig.yml fails with error "Create credentials for registry au...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
medium
urgent
Target Milestone: ---
: 3.11.z
Assignee: aos-install
QA Contact: Gaoyun Pei
URL:
Whiteboard:
: 1884089 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-30 23:46 UTC by Vladislav Walek
Modified: 2020-10-10 01:35 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-09 23:28:46 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Vladislav Walek 2020-09-30 23:46:44 UTC
The playbook fails on task "Create credentials for registry auth (3 retries left).Result was".

However, it shows under username, specific value which is not used within the inventory file.

From the definition of the task, I see that the task should be executed only when oreg_auth_username is defined.

https://github.com/openshift/openshift-ansible/blob/ee699b5/roles/openshift_node/tasks/registry_auth.yml#L16-L31

- name: Create credentials for registry auth
  docker_creds:
    path: "{{ oreg_auth_credentials_path }}"
    registry: "{{ oreg_host }}"
    username: "{{ oreg_auth_user }}"
    password: "{{ oreg_auth_password }}"
    # Test that we can actually connect with provided info
    test_login: "{{ oreg_test_login | default(True) }}"
    proxy_vars: "{{ l_docker_creds_proxy_vars }}"
    test_image: "{{ l_docker_creds_test_image }}"
  when:
    - oreg_auth_user is defined


However, even when the parameter is not defined or even defined with different value, the task fails on the same error.

The full task failure:

FAILED - RETRYING: Create credentials for registry auth (3 retries left).Result was: {
    "attempts": 1, 
    "changed": false, 
    "invocation": {
        "module_args": {
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "path": "/var/lib/origin/.docker", 
            "proxy_vars": "  ", 
            "registry": "<redacted-external-non-redhat-registry>, 
            "test_image": "openshift3/ose-pod:v3.11.153", 
            "test_login": true, 
            "test_timeout": 60, 
            "tls_verify": true, 
            "username": "<username>"
        }
    }, 
    "msg": "time=\"2020-09-28T17:05:12-07:00\" level=fatal msg=\"Error parsing image name \\\"docker://<redacted>/openshift3/ose-pod:v3.11.153\\\": unable to retrieve auth token: invalid username/password: unauthorized: Invalid Username or Password\"\n", 
    "retries": 4, 
    "state": "unknown"
}


What we can see is that username shown in the error above, is actually the username which was used during the installation of the cluster.
However, that value was removed from the inventory.

My initial assumption is, that ansible takes this value from different location or setting (there is no mention of that value in openshift.facts).


Version:
OpenShift Container Platform 3.11.153
OpenShift Container Platform 3.11.248

Platform:
baremetal

Please specify:
#n/a

Comment 1 Vladislav Walek 2020-09-30 23:47:42 UTC
*** Bug 1884089 has been marked as a duplicate of this bug. ***

Comment 4 Russell Teague 2020-10-01 12:54:34 UTC
In reviewing the customer case I saw they don't use username/password to access their internal registry.  The module 'docker_creds' in that failed task will by default attempt to connect to the registry using a username/password to confirm the credentials provided are functional.  Since the username/password are commented out of the inventory file, this task is failing.  By setting oreg_test_login=False in the inventory, the credentials validation step in the module should be skipped.  The credentials validation step uses skopeo on the host so the credentials you are seeing are probably being pulled from configuration set by skopeo login, podman login, buildah login, or docker login on the host.


- name: Create credentials for registry auth
  docker_creds:
    path: "{{ oreg_auth_credentials_path }}"
    registry: "{{ oreg_host }}"
    username: "{{ oreg_auth_user }}"
    password: "{{ oreg_auth_password }}"
    # Test that we can actually connect with provided info   <==
    test_login: "{{ oreg_test_login | default(True) }}"      <==
    ...

Please set the following in the inventory:
oreg_test_login=False


Note You need to log in before you can comment on or make changes to this bug.