Description of problem: In 4.3 CI on aws, there are alerts firing for DELETECOLLECTION high latency. https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1373 https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1374 https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1384 https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1385 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Pasting in the example from [1] for CI-search/Sippy to find: [Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Suite:openshift/conformance/parallel/minimal] fail [github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:134]: Expected <map[string]error | len:1>: { "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|UsingDeprecatedAPIExtensionsV1Beta1\",alertstate=\"firing\"} >= 1": { s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|UsingDeprecatedAPIExtensionsV1Beta1\",alertstate=\"firing\"} >= 1 had reported incorrect results: ALERTS{alertname=\"KubeAPILatencyHigh\", alertstate=\"firing\", component=\"apiserver\", endpoint=\"https\", group=\"events.k8s.io\", job=\"apiserver\", namespace=\"default\", resource=\"events\", scope=\"namespace\", service=\"kubernetes\", severity=\"warning\", verb=\"DELETECOLLECTION\", version=\"v1beta1\"} => 1 @[1588883691.309]\nALERTS{alertname=\"KubeAPILatencyHigh\", alertstate=\"firing\", component=\"apiserver\", endpoint=\"https\", job=\"apiserver\", namespace=\"default\", resource=\"events\", scope=\"namespace\", service=\"kubernetes\", severity=\"warning\", verb=\"DELETECOLLECTION\", version=\"v1\"} => 1 @[1588883691.309]", }, } to be empty [1]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1373
The PR adding DELETECOLLECTION into long-running requests list was merged in March: https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/375 Moving over to monitoring team if this is something they can backport to 4.3.z ?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409
It was reopened automatically after closing a stale PR.