Description of problem: enabled etcd monitoring, besides etcd data, apiserver and kube-controllers cluster data are shown in etcd grafana page Version-Release number of selected component (if applicable): ose-cluster-monitoring-operator-v3.11.12-1 How reproducible: always Steps to Reproduce: 1. Install cluster monitoring 2. Create Secret/kube-etcd-client-certs that the cluster-monitoring stack expects in your master ******************************************************************************** #!/usr/bin/env bash set -e set -x # only exit with zero if all commands of the pipeline exit successfully set -o pipefail oc create -f -<<EOF apiVersion: v1 data: etcd-client-ca.crt: "$(cat /etc/origin/master/master.etcd-ca.crt | base64 --wrap=0)" etcd-client.crt: "$(cat /etc/origin/master/master.etcd-client.crt | base64 --wrap=0)" etcd-client.key: "$(cat /etc/origin/master/master.etcd-client.key | base64 --wrap=0)" kind: Secret metadata: name: kube-etcd-client-certs namespace: openshift-monitoring type: Opaque EOF ******************************************************************************** Secret/kube-etcd-client-certs is created 3. # oc edit cm cluster-monitoring-config -n openshift-monitoring Enable etcd monitoring by adding the followings to cluster-monitoring-config configmap ******************************************************************************** etcd: enabled: true targets: selector: openshift.io/component: etcd openshift.io/control-plane: "true" ******************************************************************************** FYI: https://github.com/openshift/cluster-monitoring-operator/blob/master/manifests/cluster-monitoring-config.yaml#L22-L27 4. Check etcd grafana page. Actual results: apiserver and kube-controllers cluster data are shown in etcd grafana page Expected results: Should not show apiserver and kube-controllers data in etcd grafana page Additional info:
Created attachment 1485839 [details] apiserver and kube-controllers also shown in etcd grafana page
Created attachment 1485840 [details] take apiserver for example, the data is shown in etcd grafana page
Created attachment 1485841 [details] cluster-monitoring-config and grafana-dashboard-etcd configmap output
The etcd grafana dashboard determines its data sources based on the `etcd_server_has_leader` [1] metric. As a lot of Golang projects use the global metrics registry and register them in the `init` function of a package, this results in faulty registrations in other projects, importing the initial one. In the long run this will be fixed with the Kubernetes metrics overhaul [2]. As a short term fix, we can adjust the dashboard upstream (etcd-repo) and trickle the changes down to cluster-monitoring-operator. In particular we can hide the faulty cluster options. Impact for customers: One will see a broken dashboard, when selecting `apiserver` or `kube-controllers` as a cluster. As of my knowledge this does not classify for a release blocker. [1] https://github.com/openshift/cluster-monitoring-operator/blob/75f539957f384c084f691d311114227f2a9a38d2/assets/grafana/dashboard-definitions.yaml#L1181 [2] https://github.com/kubernetes/kubernetes/pull/67476#issuecomment-413785762
We have now proposed a bug fix with etcd itself: https://github.com/etcd-io/etcd/pull/10116
PR [1] to fix the issue is merged into Prometheus Operator. This will propagate into the cluster-monitoring-operator soon and then make it into the Openshift 3.11.z release. Let me know if you need anything else here from my side. [1] https://github.com/coreos/prometheus-operator/pull/1959/
*** Bug 1634680 has been marked as a duplicate of this bug. ***
Issue is not fixed, apiserver and kube-controllers data in etcd grafana page cluster monitoring image version: v3.11.36-1
Created attachment 1500351 [details] apiserver and kube-controllers are still shown in etcd grafana page
Created attachment 1500353 [details] cluster-monitoring-config and grafana-dashboard-etcd -v3.11.36-1
We're not going to fix this in 3.11.z as the required changes are risky to introduce so we're postponing it to the next non-patch release.
move back to MODIFIED, bug 1670700 is not fixed
ectd data is shown in etcd grafana page payload: 4.0.0-0.nightly-2019-04-20-175518
Created attachment 1557470 [details] etcd grafana page
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758