Bug 1631926
Summary: | Should not show apiserver and kube-controllers data in etcd grafana page | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> |
Component: | Monitoring | Assignee: | Frederic Branczyk <fbranczy> |
Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 3.11.0 | CC: | mloibl, surbania, vwalek, wsun |
Target Milestone: | --- | ||
Target Release: | 4.1.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-06-04 10:40:35 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1670700 | ||
Bug Blocks: | |||
Attachments: |
Description
Junqi Zhao
2018-09-22 02:54:30 UTC
Created attachment 1485839 [details]
apiserver and kube-controllers also shown in etcd grafana page
Created attachment 1485840 [details]
take apiserver for example, the data is shown in etcd grafana page
Created attachment 1485841 [details]
cluster-monitoring-config and grafana-dashboard-etcd configmap output
The etcd grafana dashboard determines its data sources based on the `etcd_server_has_leader` [1] metric. As a lot of Golang projects use the global metrics registry and register them in the `init` function of a package, this results in faulty registrations in other projects, importing the initial one. In the long run this will be fixed with the Kubernetes metrics overhaul [2]. As a short term fix, we can adjust the dashboard upstream (etcd-repo) and trickle the changes down to cluster-monitoring-operator. In particular we can hide the faulty cluster options. Impact for customers: One will see a broken dashboard, when selecting `apiserver` or `kube-controllers` as a cluster. As of my knowledge this does not classify for a release blocker. [1] https://github.com/openshift/cluster-monitoring-operator/blob/75f539957f384c084f691d311114227f2a9a38d2/assets/grafana/dashboard-definitions.yaml#L1181 [2] https://github.com/kubernetes/kubernetes/pull/67476#issuecomment-413785762 We have now proposed a bug fix with etcd itself: https://github.com/etcd-io/etcd/pull/10116 PR [1] to fix the issue is merged into Prometheus Operator. This will propagate into the cluster-monitoring-operator soon and then make it into the Openshift 3.11.z release. Let me know if you need anything else here from my side. [1] https://github.com/coreos/prometheus-operator/pull/1959/ *** Bug 1634680 has been marked as a duplicate of this bug. *** Issue is not fixed, apiserver and kube-controllers data in etcd grafana page cluster monitoring image version: v3.11.36-1 Created attachment 1500351 [details]
apiserver and kube-controllers are still shown in etcd grafana page
Created attachment 1500353 [details]
cluster-monitoring-config and grafana-dashboard-etcd -v3.11.36-1
We're not going to fix this in 3.11.z as the required changes are risky to introduce so we're postponing it to the next non-patch release. move back to MODIFIED, bug 1670700 is not fixed move back to MODIFIED, bug 1670700 is not fixed ectd data is shown in etcd grafana page payload: 4.0.0-0.nightly-2019-04-20-175518 Created attachment 1557470 [details]
etcd grafana page
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758 |