Bug 1922053

upgrade from 4.5.30 to 4.6.0-0.nightly-2021-01-28-234643, meet the same issue
# oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0
level=warn ts=2021-01-29T04:35:05.316Z caller=manager.go:598 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(cluster, node) (sum by(node, cpu) (node_cpu_seconds_total{job=\"node-exporter\"} * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="found duplicate series for the match group {namespace=\"openshift-user-workload-monitoring\", pod=\"prometheus-user-workload-1\"} on the right hand-side of the operation: [{__name__=\"node_namespace_pod:kube_pod_info:\", namespace=\"openshift-user-workload-monitoring\", node=\"ip-10-0-193-105.us-east-2.compute.internal\", pod=\"prometheus-user-workload-1\"}, {__name__=\"node_namespace_pod:kube_pod_info:\", namespace=\"openshift-user-workload-monitoring\", node=\"ip-10-0-177-99.us-east-2.compute.internal\", pod=\"prometheus-user-workload-1\"}];many-to-many matching not allowed: matching labels must be unique on one side"
level=warn ts=2021-01-29T04:35:35.309Z caller=manager.go:598 component="rule manager" group=node.rules msg="Evaluating rule failed" rule="record: node:node_num_cpu:sum\nexpr: count by(cluster, node) (sum by(node, cpu) (node_cpu_seconds_total{job=\"node-exporter\"} * on(namespace, pod) group_left(node) node_namespace_pod:kube_pod_info:))\n" err="found duplicate series for the match group {namespace=\"openshift-user-workload-monitoring\", pod=\"prometheus-user-workload-1\"} on the right hand-side of the operation: [{__name__=\"node_namespace_pod:kube_pod_info:\", namespace=\"openshift-user-workload-monitoring\", node=\"ip-10-0-193-105.us-east-2.compute.internal\", pod=\"prometheus-user-workload-1\"}, {__name__=\"node_namespace_pod:kube_pod_info:\", namespace=\"openshift-user-workload-monitoring\", node=\"ip-10-0-177-99.us-east-2.compute.internal\", pod=\"prometheus-user-workload-1\"}];many-to-many matching not allowed: matching labels must be unique on one side"

Created attachment 1751953 [details]
prometheus container logs

Created attachment 1751954 [details]
openshift-user-workload-monitoring events

prometheus-user-workload-1 pod changed from ip-10-0-193-105.us-east-2.compute.internal to ip-10-0-177-99.us-east-2.compute.internal

Since rule evaluation are instant queries, the error

found duplicate series for the match group {namespace="openshift-monitoring", pod="alertmanager-main-2"} on the right hand-side of the operation: [{__name__="node_namespace_pod:kube_pod_info:", namespace="openshift-monitoring", node="ip-10-0-70-152.us-east-2.compute.internal", pod="alertmanager-main-2"}, {__name__="node_namespace_pod:kube_pod_info:", namespace="openshift-monitoring", node="ip-10-0-67-241.us-east-2.compute.internal", pod="alertmanager-main-2"}]; many-to-many matching not allowed: matching labels must be unique on one side

is a bit weird as the pod "alertmanager-main-2" in namespace "openshift-monitoring" couldn't be possibly running on two nodes at once.

I am just guessing but one possible cause of this can be staleness behavior of Prometheus. If the rule evaluation happens just after a pod gets scheduled on a different node, Prometheus won't consider the old timeseries as stale yet and return both (with the new node label and with the old one) timeseries. This will cause the above error.

I doubt this has anything specific to do with user-workload monitoring though.

A quick summary would be, if a pod is scheduled on a different node, until the old timeseries gets marked as stale, the rule "record: node:node_num_cpu:sum" will fail.

@Prem your explanation is correct and it's been already fixed in the CMO master branch (see https://github.com/openshift/cluster-monitoring-operator/pull/1044 and https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/553). This bug is a clone of bug 1908655 created by Junqi, meaning that he wanted to backport it to 4.7 potentially. Given the bug's severity, I'm not sure that it deserves a backport though?

Junqi, do you feel strongly that it needs to be backported? If not we should close this bug.

(In reply to Simon Pasquier from comment #7)
> @Prem your explanation is correct and it's been already fixed in the CMO
> master branch (see
> https://github.com/openshift/cluster-monitoring-operator/pull/1044 and
> https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/553). This
> bug is a clone of bug 1908655 created by Junqi, meaning that he wanted to
> backport it to 4.7 potentially. Given the bug's severity, I'm not sure that
> it deserves a backport though?
> 
> Junqi, do you feel strongly that it needs to be backported? If not we should
> close this bug.

from Comment 0, the warning info is also shown for prometheus-k8s-0, so I think we should backport to 4.6

(In reply to Junqi Zhao from comment #8)
> from Comment 0, the warning info is also shown for prometheus-k8s-0, so I
> think we should backport to 4.6

also shown for alertmanager-main-2 pod

The warnings are only transient (e.g. during at most 5 minutes) and the situation goes back to normal after that. I'd be in favor of not backporting the fix knowing that 4.6 is in maintenance mode since March 24, 2021.

I have limited context about when to backport and when not to, but I also agree with Simon that given it's transient nature and rare reproducibility, we can skip backporting this to 4.6.

As discussed in the last sprint review, I am going ahead and closing this.