Bug 1886177

Summary:	Prometheus when installed on the cluster shouldn't have failing rules evaluation fails on skew
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	David Eads <deads>
Component:	ocs-operator	Assignee:	Anmol Sachan <asachan>
Status:	CLOSED DUPLICATE	QA Contact:	Raz Tamir <ratamir>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.6	CC:	alegrand, anpicker, dgrisonn, erooth, kakkoyun, lcosic, madam, mloibl, ocs-bugs, pkrupa, sostapov, surbania
Target Milestone:	---
Target Release:	OCS 4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	[sig-instrumentation] Prometheus when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel]
Last Closed:	2020-10-09 11:17:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Eads 2020-10-07 20:00:43 UTC

On skew tests that run 4.5 tests against a 4.6 cluster with 4.5 kubelets (I think), this prometheus test fails: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-origin-installer-old-rhcos-e2e-aws-4.6

https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.6/1313535555748761600 is one example.  I cannot understand what the error is telling me.  What's actually broken?

fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected
    <map[string]error | len:1>: {
        "prometheus_rule_evaluation_failures_total >= 1": {
            s: "promQL query: prometheus_rule_evaluation_failures_total >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"prometheus_rule_evaluation_failures_total\",\"container\":\"prometheus-proxy\",\"endpoint\":\"web\",\"instance\":\"10.128.2.19:9091\",\"job\":\"prometheus-k8s\",\"namespace\":\"openshift-monitoring\",\"pod\":\"prometheus-k8s-1\",\"rule_group\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-prometheus-k8s-rules.yaml;kubernetes.rules\",\"service\":\"prometheus-k8s\"},\"value\":[1602011841.24,\"10\"]}]",
        },
    }
to be empty

Comment 1 Damien Grisonnet 2020-10-08 08:26:38 UTC

The purpose of this test is to check if we registered an invalid prometheus rule.

We can have more info about the exact errors by looking into the prometheus container logs: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-old-rhcos-e2e-aws-4.6/1313535555748761600/artifacts/e2e-aws-upgrade/pods/openshift-monitoring_prometheus-k8s-1_prometheus.log

The test failure is caused by the following invalid recording rules:
- cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum
- cluster:kubelet_volume_stats_used_bytes:provisioner:sum

Those were recently added for telemetry purposes in https://github.com/openshift/cluster-monitoring-operator/pull/892.

I am assigning this BZ to the original author of this PR for them to fix the invalid recording rules.

Let us know if there is anything we can help with.

Comment 3 Simon Pasquier 2020-10-08 09:26:40 UTC

The issue happens because the kube-state-metrics pod is being recreated during the upgrade and at some point, there are 2 different timeseries for the kube_storageclass_info metric with different "instance" label values.

The way to fix it is to aggregate the kube_storageclass_info metric over the joined dimensions (e.g. "provisioner") + use topk(1, ...) to eliminate remaining duplicates.

sum by(provisioner) (
  topk by (namespace, persistentvolumeclaim) (
    1, kubelet_volume_stats_used_bytes
  ) * on(namespace, persistentvolumeclaim) group_right()
  topk by(namespace, persistentvolumeclaim) (
      1, kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) topk by(storageclass) (1, max by(storageclass, provisioner) (kube_storageclass_info))
  )
)

sum by(provisioner) (
  topk by (namespace, persistentvolumeclaim) (
    1, kube_persistentvolumeclaim_resource_requests_storage_bytes
  ) * on(namespace, persistentvolumeclaim) group_right()
  topk by(namespace, persistentvolumeclaim) (
    1, kube_persistentvolumeclaim_info * on(storageclass) group_left(provisioner) topk by(storageclass) (1, max by(storageclass, provisioner) (kube_storageclass_info))
  )
)

Comment 4 Pawel Krupa 2020-10-09 11:17:47 UTC


*** This bug has been marked as a duplicate of bug 1879520 ***