Bug 1615504

Description of problem:

While running the openshift-ansible install command: 
source /home/cloud-user/keystonerc; ansible-playbook -vvv --user openshift -i inventory -i openshift-ansible/playbooks/openstack/inventory.py openshift-ansible/playbooks/openstack/openshift-cluster/install.yml 2>&1 >> /home/cloud-user/logs/openshift_install.log

The installer fails on:

TASK [openshift_cluster_monitoring_operator : Wait for the ServiceMonitor CRD to be created] ***
task path: /home/cloud-user/openshift-ansible/roles/openshift_cluster_monitoring_operator/tasks/install.yaml:115

The cluster-monitoring-operator-cf688f46c-94cnf pod is in an ImagePullBackOff state. According to the log the Deployment object specified the "image": "registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11.0"

I could not pull that image

# docker pull registry.redhat.io/openshift3/ose-cluster-monitoring-operator:v3.11.0
Trying to pull repository registry.redhat.io/openshift3/ose-cluster-monitoring-operator ... 
Get https://registry.redhat.io/v2/openshift3/ose-cluster-monitoring-operator/manifests/v3.11.0: unauthorized: invalid credentials provided when attempting to perform docker authentication

The install is configured to use oreg_url: registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}

I found the image at that url:
# docker pull registry.reg-aws.openshift.com:443/openshift3/ose-cluster-monitoring-operator:v3.11.0

I believe the deployment template for the cluster-monitoring-operator should use the oreg_url for the image location.


Version-Release number of the following components:
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jul 16 2018, 19:52:45) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
git describe: openshift-ansible-3.9.0-0.10.0-3147-g7ad2385

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

TASK [openshift_cluster_monitoring_operator : Wait for the ServiceMonitor CRD to be created] ***
task path: /home/cloud-user/openshift-ansible/roles/openshift_cluster_monitoring_operator/tasks/install.yaml:115
... 30 retries ...
<192.168.0.18> (1, '\n{"changed": true, "end": "2018-08-13 11:33:52.869678", "stdout": "", "cmd": ["oc", "get", "crd", "servicemonitors.monitoring.coreos.com", "-n", "openshift-monitoring", "--config=/tmp/openshift-cluster-monitoring-ansible-RtKcWm/admin.kubeconfig"], "failed": true, "delta": "0:00:00.159734", "stderr": "No resources found.\\nError from server (NotFound): customresourcedefinitions.apiextensions.k8s.io \\"servicemonitors.monitoring.coreos.com\\" not found", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "oc get crd servicemonitors.monitoring.coreos.com -n openshift-monitoring --config=/tmp/openshift-cluster-monitoring-ansible-RtKcWm/admin.kubeconfig", "removes": null, "argv": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2018-08-13 11:33:52.709944", "msg": "non-zero return code"}\n', '')
fatal: [master-0.scale-ci.example.com]: FAILED! => {
    "attempts": 30, 
    "changed": true, 
    "cmd": [
        "oc", 
        "get", 
        "crd", 
        "servicemonitors.monitoring.coreos.com", 
        "-n", 
        "openshift-monitoring", 
        "--config=/tmp/openshift-cluster-monitoring-ansible-RtKcWm/admin.kubeconfig"
    ], 
    "delta": "0:00:00.159734", 
    "end": "2018-08-13 11:33:52.869678", 
    "invocation": {
        "module_args": {
            "_raw_params": "oc get crd servicemonitors.monitoring.coreos.com -n openshift-monitoring --config=/tmp/openshift-cluster-monitoring-ansible-RtKcWm/admin.kubeconfig", 
            "_uses_shell": false, 
            "argv": null, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2018-08-13 11:33:52.709944", 
    "stderr": "No resources found.\nError from server (NotFound): customresourcedefinitions.apiextensions.k8s.io \"servicemonitors.monitoring.coreos.com\" not found", 
    "stderr_lines": [
        "No resources found.", 
        "Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io \"servicemonitors.monitoring.coreos.com\" not found"
    ], 
    "stdout": "", 
    "stdout_lines": []
}

...

Failure summary:


  1. Hosts:    master-0.scale-ci.example.com
     Play:     Configure Cluster Monitoring Operator
     Task:     Wait for the ServiceMonitor CRD to be created
     Message:  non-zero return code

Expected results: I expect the cluster-monitoring-operator image to use the oreg_url