Bug 1476195 - Deploy metrics via ansible was failed due to clusterrole "hawkular-metrics" was not found
Summary: Deploy metrics via ansible was failed due to clusterrole "hawkular-metrics" w...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: ewolinet
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-28 08:59 UTC by Junqi Zhao
Modified: 2017-11-28 22:06 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The necessary role for a rolebinding in openshift_metrics was missing due to being processed out of order in the role. Consequence: The rolebinding creation would fail and the role would fail to install Fix: Updated so role was created right away so that rolebinding would correctly create. Result: The rolebinding is able to be created during installation every time.
Clone Of:
Environment:
Last Closed: 2017-11-28 22:06:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
ansilbe running log (833.01 KB, text/plain)
2017-07-28 08:59 UTC, Junqi Zhao
no flags Details
inventory file (443 bytes, text/plain)
2017-07-28 09:04 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Junqi Zhao 2017-07-28 08:59:38 UTC
Created attachment 1305845 [details]
ansilbe running log

Description of problem:
Deploy metrics 3.6 via ansible failed at error: role.authorization.openshift.io "hawkular-metrics" not found.
We have one similar CLI defect: https://bugzilla.redhat.com/show_bug.cgi?id=1476166
 
fatal: [host-8-174-222.host.centralci.eng.rdu2.redhat.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-metrics-ansible-UCPk2o/admin.kubeconfig", 
        "apply", 
        "-f", 
        "/tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml", 
        "-n", 
        "openshift-infra"
    ], 
    "delta": "0:00:00.200168", 
    "end": "2017-07-28 04:27:52.029236", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc --config=/tmp/openshift-metrics-ansible-UCPk2o/admin.kubeconfig apply -f /tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml -n openshift-infra", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-07-28 04:27:51.829068", 
    "warnings": []
}

STDERR:

Error from server (NotFound): error when creating "/tmp/openshift-metrics-ansible-UCPk2o/templates/hawkular-cluster-rolebinding.yaml": role.authorization.openshift.io "hawkular-metrics" not found
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.retry


Version-Release number of selected component (if applicable):
# rpm -qa | grep openshift-ansible*
openshift-ansible-filter-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-lookup-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-callback-plugins-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-playbooks-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-docs-3.6.172.0.1-1.git.0.5bd2286.el7.noarch
openshift-ansible-roles-3.6.172.0.1-1.git.0.5bd2286.el7.noarch

# openshift version
openshift v3.6.172.0.0
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 via ansible
2. 
3.

Actual results:
Deployment was failed due to clusterrole "hawkular-metrics" was not found

Expected results:
Deployment should be successfully.

Additional info:
Attached ansible running log and inventory file.

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Junqi Zhao 2017-07-28 09:04:14 UTC
Created attachment 1305847 [details]
inventory file

Comment 2 Junqi Zhao 2017-07-28 09:08:17 UTC
Metrics cases are all blocked.

Comment 3 ewolinet 2017-07-28 15:14:30 UTC
It looks like this is failing from the underlying oc apply command when creating the role binding. It seems the role template isn't being `oc apply`'d before this rolebinding one

Comment 5 Junqi Zhao 2017-07-31 01:23:41 UTC
Tested with 3.6.172.0.3-1 openshift-ansible playbooks, metrics can be deployed successfully now, please change the state to ON_QA, so we can close it. 
# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-docs-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-callback-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-lookup-plugins-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-playbooks-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-3.6.172.0.3-1.git.0.8753f3b.el7.noarch
openshift-ansible-roles-3.6.172.0.3-1.git.0.8753f3b.el7.noarch

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-q14ls   1/1       Running   0          9m
hawkular-metrics-lj3xg       1/1       Running   0          9m
heapster-jsl3m               1/1       Running   0          9m

Comment 7 Junqi Zhao 2017-07-31 04:33:09 UTC
Set it to VERIFIED based on Comment 5

Comment 11 errata-xmlrpc 2017-11-28 22:06:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.