Bug 1810947 - Alert PodDisruptionBudgetAtLimit evaluating rule failed
Summary: Alert PodDisruptionBudgetAtLimit evaluating rule failed
Keywords:
Status: CLOSED DUPLICATE of bug 1806640
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Maciej Szulik
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-06 09:34 UTC by Lili Cosic
Modified: 2020-03-10 13:48 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-10 13:48:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Lili Cosic 2020-03-06 09:34:35 UTC
Description of problem:

level=warn ts=2020-03-06T08:49:46.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetAtLimit\nexpr: kube_poddisruptionbudget_status_expected_pods == on(namespace, poddisruptionbudget,\n  service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: The pod disruption budget is preventing further disruption to pods because\n    it is at the minimum allowed level.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.20:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-7d5958fc5d-ggnq2\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, {__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.19:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-bb6dd868-nzhzp\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}];many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-06T08:49:46.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetLimit\nexpr: kube_poddisruptionbudget_status_expected_pods < on(namespace, poddisruptionbudget,\n  service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n  severity: critical\nannotations:\n  message: The pod disruption budget is below the minimum number allowed pods.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.20:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-7d5958fc5d-ggnq2\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, {__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.19:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-bb6dd868-nzhzp\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}];many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-06T08:50:16.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetAtLimit\nexpr: kube_poddisruptionbudget_status_expected_pods == on(namespace, poddisruptionbudget,\n  service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: The pod disruption budget is preventing further disruption to pods because\n    it is at the minimum allowed level.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.20:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-7d5958fc5d-ggnq2\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, 


Version-Release number of selected component (if applicable):
4.3.1


Steps to Reproduce:
1. Run two kube-state-metrics pods and check logs of promethus-k8s pods or alert that is firing about rule failed.


Note You need to log in before you can comment on or make changes to this bug.