Bug 1920569 - "duplicate series for match group" on PVC scrape
Summary: "duplicate series for match group" on PVC scrape
Keywords:
Status: CLOSED DUPLICATE of bug 1903464
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-26 15:59 UTC by Sri Ramanujam
Modified: 2021-01-26 16:20 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-26 16:20:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
example warning log of rule failure (1.21 KB, text/plain)
2021-01-26 15:59 UTC, Sri Ramanujam
no flags Details

Description Sri Ramanujam 2021-01-26 15:59:11 UTC
Created attachment 1750919 [details]
example warning log of rule failure

Description of problem:

It seems that I have metrics for the same PVC from multiple worker nodes, and this is causing Prometheus to fail processing the rule due to "many-to-many matching not allowed: matching labels must be unique on one side". This, in turn, is causing an alert to fire. 

It only seems that one particular PVC is causing this alert. Perhaps it is something about how it is being deployed or used?

Version-Release number of selected component (if applicable): 4.6


How reproducible:

I'm not sure how reproducible it is, as I'm not sure how it got into this situation to begin with. I deployed a fresh cluster with fresh workloads, and a few minutes later, this alert began to fire.

Steps to Reproduce:
1. Deploy workload with PVC to cluster

Actual results: 
Alert begins firing that Prometheus failed to evaluate rules


Expected results: 
Prometheus successfully evaluates all rules

Comment 1 Damien Grisonnet 2021-01-26 16:20:30 UTC
Thank you for the report, this seems to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1903464.

*** This bug has been marked as a duplicate of bug 1903464 ***


Note You need to log in before you can comment on or make changes to this bug.