Bug 1476195

Summary: Deploy metrics via ansible was failed due to clusterrole "hawkular-metrics" was not found
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: InstallerAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.6.0CC: aos-bugs, jokerman, mmccomas, xtian
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The necessary role for a rolebinding in openshift_metrics was missing due to being processed out of order in the role. Consequence: The rolebinding creation would fail and the role would fail to install Fix: Updated so role was created right away so that rolebinding would correctly create. Result: The rolebinding is able to be created during installation every time.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:06:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
ansilbe running log
inventory file none

Description Junqi Zhao 2017-07-28 08:59:38 UTC
Created attachment 1305845 [details]
ansilbe running log

Description of problem:
Deploy metrics 3.6 via ansible failed at error: role.authorization.openshift.io "hawkular-metrics" not found.
We have one similar CLI defect: https://bugzilla.redhat.com/show_bug.cgi?id=1476166
fatal: [host-8-174-222.host.centralci.eng.rdu2.redhat.com]: FAILED! => {
    "changed": false, 
    "cmd": [
    "delta": "0:00:00.200168", 
    "end": "2017-07-28 04:27:52.029236", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc --config=/tmp/openshift-metrics-ansible-UCPk2o/admin.kubeconfig apply -f /tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml -n openshift-infra", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        "module_name": "command"
    "rc": 1, 
    "start": "2017-07-28 04:27:51.829068", 
    "warnings": []


Error from server (NotFound): error when creating "/tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml": role.authorization.openshift.io "hawkular-metrics" not found
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.retry

Version-Release number of selected component (if applicable):
# rpm -qa | grep openshift-ansible*

# openshift version
openshift v3.
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:

Steps to Reproduce:
1. Deploy metrics 3.6 via ansible

Actual results:
Deployment was failed due to clusterrole "hawkular-metrics" was not found

Expected results:
Deployment should be successfully.

Additional info:
Attached ansible running log and inventory file.

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Junqi Zhao 2017-07-28 09:04:14 UTC
Created attachment 1305847 [details]
inventory file

Comment 2 Junqi Zhao 2017-07-28 09:08:17 UTC
Metrics cases are all blocked.

Comment 3 ewolinet 2017-07-28 15:14:30 UTC
It looks like this is failing from the underlying oc apply command when creating the role binding. It seems the role template isn't being `oc apply`'d before this rolebinding one

Comment 5 Junqi Zhao 2017-07-31 01:23:41 UTC
Tested with openshift-ansible playbooks, metrics can be deployed successfully now, please change the state to ON_QA, so we can close it. 
# rpm -qa | grep openshift-ansible

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-q14ls   1/1       Running   0          9m
hawkular-metrics-lj3xg       1/1       Running   0          9m
heapster-jsl3m               1/1       Running   0          9m

Comment 7 Junqi Zhao 2017-07-31 04:33:09 UTC
Set it to VERIFIED based on Comment 5

Comment 11 errata-xmlrpc 2017-11-28 22:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.