Bug 1653273

Summary: Metrics dashboard is throwing "Templating init failed" error after rerunning metrics playbook
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Uday kurundwade <ukurundw>
Component: Ceph-MetricsAssignee: Boris Ranto <branto>
Status: CLOSED ERRATA QA Contact: Uday kurundwade <ukurundw>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.2CC: ceph-eng-bugs, gmeno, hnallurv, ukurundw
Target Milestone: rcKeywords: Regression
Target Release: 3.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: grafana-5.2.4-1.el7cp grafana-container-3-8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-03 17:44:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screenshot of error message page none

Description Uday kurundwade 2018-11-26 12:26:53 UTC
Created attachment 1508564 [details]
Screenshot of error message page

Description of problem:
Metrics dashboard is throwing "Templating init failed" error after rerunning metrics playbook

Version-Release number of selected component (if applicable):
ceph-3.2-rhel-7-containers-candidate-38188-20181121222025
ceph-ansible-3.2.0-0.1.rc3.el7cp.noarch
cephmetrics-ansible-2.0.1-1.el7cp.x86_64
grafana version: 3-7

How reproducible:
Always

Steps to Reproduce:
1.Install containerized ceph cluster with LVM scenario
2.Install ceph metrics dashboard
3.Remove one mon from ceph cluster(re-size cluster)
4.re-run the ceph metrics playbook

Actual results:
Throwing error: "Templating init failed"

Expected results:
Should be able to access metrics dashboard without throwing any error

Additional info:

Comment 4 Christina Meno 2018-11-26 22:14:33 UTC
Uday,

The error suggest that you check the logs. We can't easily diagnose this error without them. Would you please attach the grafana logs mentioned here ? http://docs.grafana.org/installation/troubleshooting/#logging

Comment 6 Christina Meno 2018-11-27 16:19:55 UTC
Maybe blocker, we are investigating now

Comment 7 Boris Ranto 2018-11-27 23:06:10 UTC
I could reproduce locally and I do consider this a blocker. This is happening when you deploy 3.1 cephmetrics (based on grafana 5.0.0) and then you re-run the playbook with grafana 5.3.2 image that we want to ship in 3.2.

It looks like, the best bet for us will be to roll back the new image version. I could reproduce this with upstream images as well and it is an internal grafana issue (nil pointer dereference) somewhere in the grafana code. It is also reproducible with the latest version 5.3.4 image.

Comment 8 Boris Ranto 2018-11-28 00:18:13 UTC
My upstream testing shows that this is a regression in grafana 5.3.x. The upgrade to grafana 5.2.4 works fine for me. I will do a new downstream grafana build for 5.2.4. It should contain all the necessary security fixes while still being upgradeable from 5.0.0.

btw: Based on my testing, even the password prompt to change the default password that was broken in 5.3.2 (bz1652427) works fine in 5.2.4.

Comment 10 Boris Ranto 2018-11-28 01:54:28 UTC
The container build:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=19300192

Comment 13 errata-xmlrpc 2019-01-03 17:44:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0019