Bug 1733548
| Summary: | No data in grafana dashboard | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Rajnikant <rkant> |
| Component: | Monitoring | Assignee: | Lili Cosic <lcosic> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Junqi Zhao <juzhao> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.11.0 | CC: | alegrand, anpicker, erooth, fbranczy, fshaikh, jmalde, lcosic, mirollin, mloibl, openshift-bugs-escalate, pkrupa, prthakur, spasquie, srengan, surbania |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-13 07:15:01 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Rajnikant
2019-07-26 13:42:04 UTC
I tried to reproduce this but I can't. I spin up a v3.11 openshift cluster and I can access Grafana dashboard and see all the data. Can you provide additional information. Cluster events, logs from grafana container would be useful. How long was the cluster up and running? Anything else failing besides grafana? Also I noticed it says "- All pods are running no pod restart." But in the output of the ` oc get pods` command it seems like multiple pods were restarted. Can you clarify that. Thanks! Another question, it says "- Promethues installation done completely ", does this mean the installation was done manually? Thanks for the access to the logs. While having a look at the grafana logs, we noticed that grafana itself seems to be having problems with a locked database table. We suggest to delete the grafana pod and see if this problem occurs again. (There are many upstream issues open about this on grafana itself [1]. If data in grafana is still not visible after the pod is deleted and restarted, can you provide the logs and events again. Thank you. FYI: We could not reproduce this on a new cluster. [1] https://github.com/grafana/grafana/issues/9870 Lili, The pod has been deleted and still the issue persists. Infact the logs provided are after restarting the pod again. Hello, Can you double check you can see the data in both the Alertmanager and Prometheus dashboards and the only place you are not seeing it is in Grafana? That should help us pin-down the problem better. Also we noticed a lot of "unauthorised access" errors in the grafana proxy logs, were these just failed login attempts? I believe you haven't provided the events and other information we asked above, that would make it easier to debug. Thanks! Can you make sure that Grafana can reach prometheus, exec into the grafana container and try to reach https://prometheus-k8s.openshift-monitoring.svc:9091, ping binary seems like it's not available, but curl should work. The other thing we can try is to increase the logs verbosity if the above works. Thanks! Created attachment 1605901 [details]
UI screenshots of grafana and Prometheus target UI - grafana cluster-ui
|