Fedora Account System
Red Hat Associate
Red Hat Customer
On skew tests that run 4.5 tests against a 4.6 cluster with 4.5 kubelets (I think), this prometheus test fails: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-origin-installer-old-rhcos-e2e-aws-4.6 https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.6/1313535555748761600 is one example. I cannot understand what the error is telling me. What's actually broken? fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected <map[string]error | len:1>: { "prometheus_rule_evaluation_failures_total >= 1": { s: "promQL query: prometheus_rule_evaluation_failures_total >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"prometheus_rule_evaluation_failures_total\",\"container\":\"prometheus-proxy\",\"endpoint\":\"web\",\"instance\":\"10.128.2.19:9091\",\"job\":\"prometheus-k8s\",\"namespace\":\"openshift-monitoring\",\"pod\":\"prometheus-k8s-1\",\"rule_group\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-k8s-rules.yaml;kubernetes.rules\",\"service\":\"prometheus-k8s\"},\"value\":[1602011841.24,\"10\"]}]", }, } to be empty
The purpose of this test is to check if we registered an invalid prometheus rule. We can have more info about the exact errors by looking into the prometheus container logs: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.6/1313535555748761600/artifacts/e2e-aws-upgrade/pods/openshift-monitoring_prometheus-k8s-1_prometheus.log The test failure is caused by the following invalid recording rules: - cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum - cluster:kubelet_volume_stats_used_bytes:provisioner:sum Those were recently added for telemetry purposes in https://github.com/openshift/cluster-monitoring-operator/pull/892. I am assigning this BZ to the original author of this PR for them to fix the invalid recording rules. Let us know if there is anything we can help with.
The issue happens because the kube-state-metrics pod is being recreated during the upgrade and at some point, there are 2 different timeseries for the kube_storageclass_info metric with different "instance" label values. The way to fix it is to aggregate the kube_storageclass_info metric over the joined dimensions (e.g. "provisioner") + use topk(1, ...) to eliminate remaining duplicates. sum by(provisioner) ( topk by (namespace, persistentvolumeclaim) ( 1, kubelet_volume_stats_used_bytes ) * on(namespace, persistentvolumeclaim) group_right() topk by(namespace, persistentvolumeclaim) ( 1, kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) topk by(storageclass) (1, max by(storageclass, provisioner) (kube_storageclass_info)) ) ) sum by(provisioner) ( topk by (namespace, persistentvolumeclaim) ( 1, kube_persistentvolumeclaim_resource_requests_storage_bytes ) * on(namespace, persistentvolumeclaim) group_right() topk by(namespace, persistentvolumeclaim) ( 1, kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) topk by(storageclass) (1, max by(storageclass, provisioner) (kube_storageclass_info)) ) )
*** This bug has been marked as a duplicate of bug 1879520 ***