[flake]Alerts shouldn't report any alerts in firing or pending state apart from Watchdog and AlertmanagerReceiversNotConfigured and have no gaps in Watchdog firing
Description of problem:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-aws/1410150717401862144
[sig-instrumentation][Late] Alerts shouldn't report any alerts in firing or pending state apart from Watchdog and AlertmanagerReceiversNotConfigured and have no gaps in Watchdog firing [Suite:openshift/conformance/parallel] failed
Run #0: Failed expand_less 12s
flake: Unexpected alert behavior during test:
alert KubeAPIErrorBudgetBurn pending for 756.5079998970032 seconds with labels: {long="3d", severity="warning", short="6h"}
same for
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-aws/1410217313440894976
alert details
********************************
- alert: KubeAPIErrorBudgetBurn
annotations:
description: The API server is burning too much error budget. This alert fires
when too many requests are failing with high latency. Use the 'API Performance'
monitoring dashboards to narrow down the request states and latency. The 'etcd'
monitoring dashboards also provides metrics to help determine etcd stability
and performance.
summary: The API server is burning too much error budget.
expr: |
sum(apiserver_request:burnrate3d) > (1.00 * 0.01000)
and
sum(apiserver_request:burnrate6h) > (1.00 * 0.01000)
for: 3h
labels:
long: 3d
severity: warning
short: 6h
********************************
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
alert KubeAPIErrorBudgetBurn pending
Expected results:
Additional info:
Comment 1Prashant Balachandran
2021-07-01 11:16:18 UTC