Created attachment 1668934 [details] many-to-many error Description of problem: after upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427, there are "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod after upgrade, no such error before upgrade here is a stastic for the affected rules record rule occurrence number record: mixin_pod_workload 3 record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate 3 record: node_namespace_pod_container:container_memory_cache 3 record: node_namespace_pod_container:container_memory_rss 3 record: node_namespace_pod_container:container_memory_swap 3 record: node_namespace_pod_container:container_memory_working_set_bytes 3 record: node:node_num_cpu:sum 2 alert rule occurrence number alert: KubePodNotReady 3 alert: PodDisruptionBudgetAtLimit 3 alert: PodDisruptionBudgetLimit 3 full logs see the attached file # oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side" level=warn ts=2020-03-10T07:47:10.625Z caller=manager.go:525 component="rule manager" group=kubernetes-apps msg="Evaluating rule failed" rule="alert: KubePodNotReady\nexpr: sum by(namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\",namespace=~\"(openshift-.*|kube-.*|default|logging)\",phase=~\"Pending|Unknown\"})\n * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"}))\n > 0\nfor: 15m\nlabels:\n severity: critical\nannotations:\n message: Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state\n for longer than 15 minutes.\n" err="found duplicate series for the match group {namespace=\"openshift-kube-apiserver\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"} on the right hand-side of the operation: [{namespace=\"openshift-kube-apiserver\", owner_kind=\"Node\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}, {namespace=\"openshift-kube-apiserver\", owner_kind=\"<none>\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}];many-to-many matching not allowed: matching labels must be unique on one side" level=warn ts=2020-03-10T07:47:16.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetAtLimit\nexpr: kube_poddisruptionbudget_status_expected_pods == on(namespace, poddisruptionbudget,\n service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n severity: warning\nannotations:\n message: The pod disruption budget is preventing further disruption to pods because\n it is at the minimum allowed level.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.37:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-9j4fx\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, {__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.128.2.10:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-fs6gs\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}];many-to-many matching not allowed: matching labels must be unique on one side" ... Version-Release number of selected component (if applicable): upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427 How reproducible: sometimes Steps to Reproduce: 1. oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side" 2. 3. Actual results: Expected results: Additional info:
upgraded from 4.4.3 to 4.5.0-0.nightly-2020-05-13-144407, no such issue now # oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side" no result # oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side" no result
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409