Bug 1812006 - "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod after upgrade
Summary: "many-to-many matching not allowed: matching labels must be unique on one sid...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.4
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.5.0
Assignee: Simon Pasquier
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1834913
TreeView+ depends on / blocked
 
Reported: 2020-03-10 11:28 UTC by Junqi Zhao
Modified: 2020-07-13 17:19 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1834913 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:19:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
many-to-many error (30.49 KB, text/plain)
2020-03-10 11:28 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes-monitoring kubernetes-mixin pull 417 0 None closed Fix potential 'many-to-many matching' errors 2021-02-18 05:12:07 UTC
Github openshift cluster-monitoring-operator pull 785 0 None closed Bug 1812006: avoid "many-to-many matching not allowed" errors 2021-02-18 05:12:07 UTC

Description Junqi Zhao 2020-03-10 11:28:51 UTC
Created attachment 1668934 [details]
many-to-many error

Description of problem:
after upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427, there are "many-to-many matching not allowed: matching labels must be unique on one side" warn info in prometheus-k8s pod after upgrade, no such error before upgrade

here is a stastic for the affected rules
record rule							occurrence number
record: mixin_pod_workload					3
record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate	3
record: node_namespace_pod_container:container_memory_cache	3
record: node_namespace_pod_container:container_memory_rss	3
record: node_namespace_pod_container:container_memory_swap	3
record: node_namespace_pod_container:container_memory_working_set_bytes	3
record: node:node_num_cpu:sum						2
	
	
alert rule				occurrence number	
alert: KubePodNotReady			3
alert: PodDisruptionBudgetAtLimit	3
alert: PodDisruptionBudgetLimit		3

full logs see the attached file
# oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-10T07:47:10.625Z caller=manager.go:525 component="rule manager" group=kubernetes-apps msg="Evaluating rule failed" rule="alert: KubePodNotReady\nexpr: sum by(namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{job=\"kube-state-metrics\",namespace=~\"(openshift-.*|kube-.*|default|logging)\",phase=~\"Pending|Unknown\"})\n  * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=\"Job\"}))\n  > 0\nfor: 15m\nlabels:\n  severity: critical\nannotations:\n  message: Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state\n    for longer than 15 minutes.\n" err="found duplicate series for the match group {namespace=\"openshift-kube-apiserver\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"} on the right hand-side of the operation: [{namespace=\"openshift-kube-apiserver\", owner_kind=\"Node\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}, {namespace=\"openshift-kube-apiserver\", owner_kind=\"<none>\", pod=\"kube-apiserver-zhsun-qqrhz-m-0.c.openshift-qe.internal\"}];many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2020-03-10T07:47:16.084Z caller=manager.go:525 component="rule manager" group=cluster-version msg="Evaluating rule failed" rule="alert: PodDisruptionBudgetAtLimit\nexpr: kube_poddisruptionbudget_status_expected_pods == on(namespace, poddisruptionbudget,\n  service) kube_poddisruptionbudget_status_desired_healthy\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  message: The pod disruption budget is preventing further disruption to pods because\n    it is at the minimum allowed level.\n" err="found duplicate series for the match group {namespace=\"openshift-machine-config-operator\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"} on the right hand-side of the operation: [{__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.131.0.37:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-9j4fx\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}, {__name__=\"kube_poddisruptionbudget_status_desired_healthy\", endpoint=\"https-main\", instance=\"10.128.2.10:8443\", job=\"kube-state-metrics\", namespace=\"openshift-machine-config-operator\", pod=\"kube-state-metrics-d6c94499b-fs6gs\", poddisruptionbudget=\"etcd-quorum-guard\", service=\"kube-state-metrics\"}];many-to-many matching not allowed: matching labels must be unique on one side"
...


Version-Release number of selected component (if applicable):
upgrade from 4.3.5 to 4.4.0-0.nightly-2020-03-10-042427

How reproducible:
sometimes

Steps to Reproduce:
1. oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
2.
3.

Actual results:


Expected results:


Additional info:

Comment 7 Junqi Zhao 2020-05-14 11:59:48 UTC
upgraded from 4.4.3 to 4.5.0-0.nightly-2020-05-13-144407, no such issue now
# oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
no result

# oc -n openshift-monitoring logs prometheus-k8s-1 -c prometheus | grep "many-to-many matching not allowed: matching labels must be unique on one side"
no result

Comment 9 errata-xmlrpc 2020-07-13 17:19:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.