test: [sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] is failing frequently in CI, see search results: https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-instrumentation%5C%5D+Prometheus+when+installed+on+the+cluster+shouldn%27t+report+any+alerts+in+firing+state+apart+from+Watchdog+and+AlertmanagerReceiversNotConfigured+%5C%5BEarly%5C%5D https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-4.5/1298666591172431872 ----- [sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Suite:openshift/conformance/parallel] expand_less 1m38s fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected <map[string]error | len:1>: { "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1": { s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubeAPIDown\",\"alertstate\":\"firing\",\"severity\":\"critical\"},\"value\":[1598466340.548,\"1\"]},{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubeControllerManagerDown\",\"alertstate\":\"firing\",\"severity\":\"critical\"},\"value\":[1598466340.548,\"1\"]},{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"KubeSchedulerDown\",\"alertstate\":\"firing\",\"severity\":\"critical\"},\"value\":[1598466340.548,\"1\"]}]", }, } to be empty
KubeAPIDown, KubeControllerManagerDown indicate issues with the control plane, hence reassigning to kube-apiserver.
This is not actionable. The query mixes many root causes already tracked elsewhere. Either give an analysis or point too some concrete issue (e.g. by platform, networking stack, component).
https://search.ci.openshift.org/?search=%5C%5Bsig-instrumentation%5C%5D+Prometheus+when+installed+on+the+cluster+shouldn%27t+report+any+alerts+in+firing+state+apart+from+Watchdog+and+AlertmanagerReceiversNotConfigured+%5C%5BEarly%5C%5D&maxAge=168h&context=1&type=junit&name=4.6&maxMatches=5&maxBytes=20971520&groupBy=job https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/307/pull-ci-openshift-ovn-kubernetes-release-4.6-e2e-vsphere-ovn/1315608803391049728
*** Bug 1891068 has been marked as a duplicate of this bug. ***