Bug 1835988

Summary: alert firing for KubeAPILatencyHigh for verb DELETECOLLECTION
Product: OpenShift Container Platform Reporter: Abhinav Dahiya <adahiya>
Component: MonitoringAssignee: Pawel Krupa <pkrupa>
Status: CLOSED NOTABUG QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: alegrand, anpicker, aos-bugs, erooth, kakkoyun, lcosic, mfojtik, mloibl, pkrupa, surbania, wking
Target Milestone: ---Keywords: Reopened
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-12 12:00:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Abhinav Dahiya 2020-05-14 20:48:35 UTC 
Description of problem:


In 4.3 CI on aws, there are alerts firing for DELETECOLLECTION high latency.

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1373
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1374
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1384
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1385

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 W. Trevor King 2020-05-14 22:55:54 UTC 
Pasting in the example from [1] for CI-search/Sippy to find:

[Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Suite:openshift/conformance/parallel/minimal]
fail [github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:134]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|UsingDeprecatedAPIExtensionsV1Beta1\",alertstate=\"firing\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|UsingDeprecatedAPIExtensionsV1Beta1\",alertstate=\"firing\"} >= 1 had reported incorrect results: ALERTS{alertname=\"KubeAPILatencyHigh\", alertstate=\"firing\", component=\"apiserver\", endpoint=\"https\", group=\"events.k8s.io\", job=\"apiserver\", namespace=\"default\", resource=\"events\", scope=\"namespace\", service=\"kubernetes\", severity=\"warning\", verb=\"DELETECOLLECTION\", version=\"v1beta1\"} => 1 @[1588883691.309]\nALERTS{alertname=\"KubeAPILatencyHigh\", alertstate=\"firing\", component=\"apiserver\", endpoint=\"https\", job=\"apiserver\", namespace=\"default\", resource=\"events\", scope=\"namespace\", service=\"kubernetes\", severity=\"warning\", verb=\"DELETECOLLECTION\", version=\"v1\"} => 1 @[1588883691.309]",
        },
    }
to be empty

[1]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/1373
Comment 2 Michal Fojtik 2020-05-19 07:51:54 UTC 
The PR adding DELETECOLLECTION into long-running requests list was merged in March: https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/375

Moving over to monitoring team if this is something they can backport to 4.3.z ?
Comment 7 errata-xmlrpc 2020-07-13 17:39:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409
Comment 9 Pawel Krupa 2020-10-12 12:00:56 UTC 
It was reopened automatically after closing a stale PR.