Bug 2065682

Summary:	Remote write relabel config adds label __tmp_openshift_cluster_id__ to all metrics
Product:	OpenShift Container Platform	Reporter:	Jan Fajerski <jfajersk>
Component:	Monitoring	Assignee:	Jan Fajerski <jfajersk>
Status:	CLOSED ERRATA	QA Contact:	hongyan li <hongyli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.11	CC:	amandalawrancee, amuller, anpicker, aos-bugs
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:54:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Fajerski 2022-03-18 13:24:55 UTC

Description of problem:
For the implementation https://issues.redhat.com/browse/MON-2245 we add a label called  __tmp_openshift_cluster_id__ with the value of the clusters id to all remote_write configurations. This is intended to be used as an input for users to add the cluster_id as a label of teir choice.
For write_relabel_configs (as opposed to relabel_configs) labels with a name starting with __ are not automatically dropped, so we end up with the __tmp_openshift_cluster_id__  on all metrics send via remote_write.

Version-Release number of selected component (if applicable):
4.11

How reproducible:
Add a remote_write specification to cluster-monitoring-config.

Actual results:
All metrics that are send vial remote_write have the __tmp_openshift_cluster_id__ label.

Expected results:
No additional label.

Comment 3 hongyan li 2022-03-21 04:59:24 UTC

Test with payload 4.11.0-0.nightly-2022-03-20-160505

oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://remote-write.endpoint"
        writeRelabelConfigs:
        - sourceLabels:
          - __tmp_openshift_cluster_id__
          targetLabel: "cluster_id"
EOF

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- cat /etc/prometheus/config_out/prometheus.env.yaml|grep -i -A15 remote_write 
remote_write:
- url: https://remote-write.endpoint
  remote_timeout: 30s
  write_relabel_configs:
  - target_label: __tmp_openshift_cluster_id__
    replacement: 996af312-a993-4066-9c66-975faeba18bc
    action: replace
  - source_labels:
    - __tmp_openshift_cluster_id__
    target_label: cluster_id
    action: replace
  - regex: __tmp_openshift_cluster_id__
    action: labeldrop

Comment 7 errata-xmlrpc 2022-08-10 10:54:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Comment 8 amandalawrance 2022-08-29 21:38:56 UTC

This post is informative, however I noticed that the one on https://www.tmcnet.com/topics/articles/2022/02/24/451626-best-essay-writing-services-2021-everything-should-know.htm had much more helpful details, so you might want to check that out as well. College students who are looking for advice on selecting a writing service will find it useful. A former version of myself was a student just like me a few years ago.