Bug 1476195

Summary: Deploy metrics via ansible was failed due to clusterrole "hawkular-metrics" was not found
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: InstallerAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.6.0CC: aos-bugs, jokerman, mmccomas, xtian
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The necessary role for a rolebinding in openshift_metrics was missing due to being processed out of order in the role. Consequence: The rolebinding creation would fail and the role would fail to install Fix: Updated so role was created right away so that rolebinding would correctly create. Result: The rolebinding is able to be created during installation every time.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:06:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ansilbe running log
none
inventory file none

Description Junqi Zhao 2017-07-28 08:59:38 UTC 
Created attachment 1305845 [details]
ansilbe running log

Description of problem:
Deploy metrics 3.6 via ansible failed at error: role.authorization.openshift.io "hawkular-metrics" not found.
We have one similar CLI defect: https://bugzilla.redhat.com/show_bug.cgi?id=1476166
 
fatal: [host-8-174-222.host.centralci.eng.rdu2.redhat.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-metrics-ansible-UCPk2o/admin.kubeconfig", 
        "apply", 
        "-f", 
        "/tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml", 
        "-n", 
        "openshift-infra"
    ], 
    "delta": "0:00:00.200168", 
    "end": "2017-07-28 04:27:52.029236", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc --config=/tmp/openshift-metrics-ansible-UCPk2o/admin.kubeconfig apply -f /tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml -n openshift-infra", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-07-28 04:27:51.829068", 
    "warnings": []
}

STDERR:

Error from server (NotFound): error when creating "/tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml": role.authorization.openshift.io "hawkular-metrics" not found
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.retry


Version-Release number of selected component (if applicable):
# rpm -qa | grep openshift-ansible*
openshift-ansible-filter-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-lookup-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-callback-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-playbooks-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-docs-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-roles-3.6.172.0.1-1.git.0.5bd2286.el7.noarch

# openshift version
openshift v3.6.172.0.0
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 via ansible
2. 
3.

Actual results:
Deployment was failed due to clusterrole "hawkular-metrics" was not found

Expected results:
Deployment should be successfully.

Additional info:
Attached ansible running log and inventory file.

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 1 Junqi Zhao 2017-07-28 09:04:14 UTC 
Created attachment 1305847 [details]
inventory file
Comment 2 Junqi Zhao 2017-07-28 09:08:17 UTC 
Metrics cases are all blocked.
Comment 3 ewolinet 2017-07-28 15:14:30 UTC 
It looks like this is failing from the underlying oc apply command when creating the role binding. It seems the role template isn't being `oc apply`'d before this rolebinding one
Comment 4 ewolinet 2017-07-28 16:04:03 UTC 
https://github.com/openshift/openshift-ansible/pull/4931
https://github.com/openshift/openshift-ansible/pull/4932
Comment 5 Junqi Zhao 2017-07-31 01:23:41 UTC 
Tested with 3.6.172.0.3-1 openshift-ansible playbooks, metrics can be deployed successfully now, please change the state to ON_QA, so we can close it. 
# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-docs-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-callback-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-lookup-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-playbooks-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-roles-3.6.172.0.3-1.git.0.8753f3b.el7.noarch

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-q14ls   1/1       Running   0          9m
hawkular-metrics-lj3xg       1/1       Running   0          9m
heapster-jsl3m               1/1       Running   0          9m
Comment 7 Junqi Zhao 2017-07-31 04:33:09 UTC 
Set it to VERIFIED based on Comment 5
Comment 11 errata-xmlrpc 2017-11-28 22:06:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188