Bug 1834913

+++ This bug was initially created as a clone of Bug #1812006 +++

Description of problem:
after upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427, there are "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod after upgrade, no such error before upgrade

here is a stastic for the affected rules
record rule							occurrence number
record: mixin_pod_workload					3
record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate	3
record: node_namespace_pod_container:container_memory_cache	3
record: node_namespace_pod_container:container_memory_rss	3
record: node_namespace_pod_container:container_memory_swap	3
record: node_namespace_pod_container:container_memory_working_set_bytes	3
record: node:node_num_cpu:sum						2
	
	
alert rule				occurrence number	
alert: KubePodNotReady			3
alert: PodDisruptionBudgetAtLimit	3
alert: PodDisruptionBudgetLimit		3

full logs see the attached file
# oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-10T07:47:10.625Z caller=manager.go:525 component="rule manager" group=kubernetes-apps msg="Evaluating rule failed" rule="alert: KubePodNotReady\nexpr: sum by(namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\",namespace=~\"(openshift-.*|kube-.*|default|logging)\",phase=~\"Pending|Unknown\"})\n  * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"}))\n  > 0\nfor: 15m\nlabels:\n  severity: critical\nannotations:\n  message: Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state\n    for longer than 15 minutes.\n" err="found duplicate series for the match group {namespace=\"openshift-kube-apiserver\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"} on the right hand-side of the operation: [{namespace=\"openshift-kube-apiserver\", owner_kind=\"Node\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}, {namespace=\"openshift-kube-apiserver\", owner_kind=\"<none>\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}];many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-10T07:47:16.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetAtLimit\nexpr: kube_poddisruptionbudget_status_expected_pods == on(namespace, poddisruptionbudget,\n  service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: The pod disruption budget is preventing further disruption to pods because\n    it is at the minimum allowed level.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.37:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-9j4fx\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, {__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.128.2.10:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-fs6gs\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}];many-to-many matching not allowed: matching labels must be unique on one side"
...


Version-Release number of selected component (if applicable):
upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427

How reproducible:
sometimes

Steps to Reproduce:
1. oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Simon Pasquier on 2020-03-10 13:49:48 UTC ---

PodDisruptionBudgetAtLimit and PodDisruptionBudgetLimit alerts are tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1806640

--- Additional comment from Junqi Zhao on 2020-04-15 03:23:25 UTC ---

met the same issue after upgrading from 4.3.12 to 4.4.0-0.nightly-2020-04-13-113747

--- Additional comment from Simon Pasquier on 2020-05-05 09:15:06 UTC ---

deferred due to higher priority tasks.

--- Additional comment from errata-xmlrpc on 2020-05-12 14:59:22 UTC ---

This bug has been added to advisory RHBA-2020:51809 by OpenShift Release Team Bot (ocp-build/buildvm.openshift.eng.bos.redhat.com)

--- Additional comment from errata-xmlrpc on 2020-05-12 14:59:35 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2020:51809-02
https://errata.devel.redhat.com/advisory/51809