Bug 1997028

Summary:	prometheus-operator update removes env var support for thanos-sidecar
Product:	OpenShift Container Platform	Reporter:	Jan Fajerski <jfajersk>
Component:	Monitoring	Assignee:	Jan Fajerski <jfajersk>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	medium	Docs Contact:
Priority:	low
Version:	4.9	CC:	amuller, anpicker, aos-bugs, brad.ison, erooth
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-12 04:37:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Fajerski 2021-08-24 09:13:03 UTC

Description of problem:
The prometheus pod never starts due to thanos-sidecar. https://github.com/prometheus-operator/prometheus-operator/pull/4038 removed the env var replacement that we rely on in CMO.
We need to remove [$(POD_IP)].


Version-Release number of selected component (if applicable):
current CMO with prometheus-operator 0.50.0 (https://github.com/openshift/prometheus-operator/pull/133)


How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 6 Junqi Zhao 2021-09-13 06:35:54 UTC

tested with 4.10.0-0.nightly-2021-09-10-083647, prometheus pods are normal
# oc -n openshift-monitoring get prometheus k8s -oyaml | grep grpc-address
no result

# oc -n openshift-user-workload-monitoring get prometheus user-workload -oyaml | grep grpc-address
no result

# oc -n openshift-monitoring get pod | grep prometheus-k8s
prometheus-k8s-0                              7/7     Running   0               3h14m
prometheus-k8s-1                              7/7     Running   0               3h14m

# oc -n openshift-user-workload-monitoring get pod | grep prometheus-user-workload
prometheus-user-workload-0             5/5     Running   0          9m40s
prometheus-user-workload-1             5/5     Running   0          9m40s

Comment 7 Junqi Zhao 2021-09-13 07:18:12 UTC

address now is 0.0.0.0, instead of pod ip
# oc -n openshift-monitoring logs -c  thanos-sidecar prometheus-k8s-0 | grep sidecar | grep "listening for serving gRPC"
level=info ts=2021-09-13T03:20:35.200956589Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
# oc -n openshift-user-workload-monitoring logs -c  thanos-sidecar prometheus-user-workload-0 | grep sidecar | grep "listening for serving gRPC"
level=info ts=2021-09-13T06:24:52.919399744Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901

Comment 12 errata-xmlrpc 2022-03-12 04:37:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056