Bug 1819612

Summary: deduplication in thanos querier for rules coming from thanos ruler is not working
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Lili Cosic <lcosic>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: low    
Version: 4.5CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, surbania
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:24:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2020-04-01 07:50:32 UTC 
Description of problem:
create PrometheusRule under user namespace, and search ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3"} in thanos-querier, there is not difference if deduplication and partial response are selected or don't selected.

Steps: enable techPreviewUserWorkload and do the followings
***********************************************
# oc new-project test3
# oc create -f - << EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test3.rules
spec:
  groups:
  - name: alerting rules
    rules:
    - alert: Watchdog
      expr: vector(1)
      labels:
        severity: none
      message:
        This is an alert meant to ensure that the entire alerting pipeline is functional.
EOF
***********************************************

search in thanos-querier(deduplication and partial response are selected)
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3"}
Element 	Value
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-0"}	1
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-1"}	1

don't select deduplication and partial response, the result is the same with above
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3"}
Element 	Value
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-0"}	1
ALERTS{alertname="Watchdog",alertstate="firing",namespace="test3",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-1"}	1


Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-03-31-203533

How reproducible:
Always

Steps to Reproduce:
1. See the description
2.
3.

Actual results:
thanos_ruler_replica label is not dropped

Expected results:
should drop thanos_ruler_replica label

Additional info:
Comment 3 Junqi Zhao 2020-04-03 01:17:22 UTC 
issue is fixed with 4.5.0-0.nightly-2020-04-02-195956

# oc -n openshift-monitoring get deploy thanos-querier -oyaml | grep "thanos_ruler_replica"
        - --query.replica-label=thanos_ruler_replica

search in thanos-querier
ALERTS{alertname="Watchdog",alertstate="firing",namespace="load"}

selected deduplication, result is
Element 	Value
ALERTS{alertname="Watchdog",alertstate="firing",namespace="load",severity="none"}	1

don't select deduplication, result is
Element 	Value
ALERTS{alertname="Watchdog",alertstate="firing",namespace="load",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-0"}	1
ALERTS{alertname="Watchdog",alertstate="firing",namespace="load",severity="none",thanos_ruler_replica="thanos-ruler-user-workload-1"}	1
Comment 5 errata-xmlrpc 2020-07-13 17:24:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409