Bug 1652532

Summary: docker_image_availability" fails everytime on openshift 3.11
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: InstallerAssignee: Scott Dodson <sdodson>
Installer sub component: openshift-ansible QA Contact: Johnny Liu <jialiu>
Status: CLOSED NOTABUG Docs Contact:
Severity: medium    
Priority: medium CC: ajacocks, jkaur, mgugino
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-29 21:06:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Inventory file none

Description Jaspreet Kaur 2018-11-22 10:26:44 UTC
Created attachment 1507909 [details]
Inventory file

Description of problem: Installation fails with below error

  2. Hosts:    host1, host2, host3
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required container images are not available:
                   registry.redhat.io/openshift3/ose-control-plane:v3.11,
                   registry.redhat.io/openshift3/ose-deployer:v3.11,
                   registry.redhat.io/openshift3/ose-docker-registry:v3.11,
                   registry.redhat.io/openshift3/ose-haproxy-router:v3.11,
                   registry.redhat.io/openshift3/ose-pod:v3.11,
                   registry.redhat.io/openshift3/registry-console:v3.11,
                   registry.redhat.io/rhel7/etcd:3.2.22
               Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>




Version-Release number of the following components:

# rpm -q openshift-ansible
openshift-ansible-3.11.43-1.git.0.fa69a02.el7.noarch

# rpm -q ansible
ansible-2.6.7-1.el7ae.noarch

[quicklab@master-2 ~]$ ansible --version
ansible 2.6.7
  config file = /home/quicklab/ansible.cfg
  configured module search path = [u'/home/quicklab/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Steps to Reproduce:
1. Installed a new cluster with ocp 3.11
2.
3.

Actual results: Installation fails on above check

Expected results: shouldn't have failed on above check

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Scott Dodson 2018-11-26 21:31:46 UTC
Can you please gather complete logs at -vvv level? That will show us all the images it's trying to inspect and the output from those commands.

Comment 3 J. Alexander Jacocks 2018-11-27 20:53:15 UTC
I'm having the same issue, and I attached the logs, below.

Comment 4 J. Alexander Jacocks 2018-11-27 21:05:21 UTC
And never mind, I figured out the problem, based on the error:

CHECK [memory_availability : master.jajtest] **********************
fatal: [master.jajtest]: FAILED! => {
    "changed": true,
    "checks": {
        "disk_availability": {},
        "docker_image_availability": {
            "changed": true,
            "failed": true,
            "failures": [
                [
                    "OpenShiftCheckException",
                    "One or more required container images are not available:\n    registry.redhat.io/openshift3/ose-control-plane:v3.11,\n    registry.redhat.io/openshift3/ose-deployer:v3.11,\n    registry.redhat.io/openshift3/ose-docker-registry:v3.11,\n    registry.redhat.io/openshift3/ose-haproxy-router:v3.11,\n    registry.redhat.io/openshift3/ose-pod:v3.11,\n    registry.redhat.io/openshift3/registry-console:v3.11,\n    registry.redhat.io/rhel7/etcd:3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\n"
                ]
            ],

And from the inventory file:

$ grep oreg inventory_ose 
oreg_auth_user=${INSERT_VALUE_HERE}
oreg_auth_password=${INSERT_VALUE_HERE}

The issue is the new need for a Red Hat login and password to install OSE:

https://access.redhat.com/solutions/3661111

And:

https://docs.openshift.com/container-platform/3.11/release_notes/ocp_3_11_release_notes.html#ocp-311-registry-auth-credentials-required

Comment 5 Michael Gugino 2018-11-29 21:07:59 UTC
This should fail early if no oreg_auth_* variables are defined, which did go through QE.  Most likely inventory problem (eg, disable sanity checks).

Comment 6 Red Hat Bugzilla 2023-09-14 04:42:41 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days