Bug 1997028 - prometheus-operator update removes env var support for thanos-sidecar
Summary: prometheus-operator update removes env var support for thanos-sidecar
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.9
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.10.0
Assignee: Jan Fajerski
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-24 09:13 UTC by Jan Fajerski
Modified: 2022-03-12 04:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:37:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1367 0 None None None 2021-09-07 12:17:13 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:37:52 UTC

Description Jan Fajerski 2021-08-24 09:13:03 UTC
Description of problem:
The prometheus pod never starts due to thanos-sidecar. https://github.com/prometheus-operator/prometheus-operator/pull/4038 removed the env var replacement that we rely on in CMO.
We need to remove [$(POD_IP)].


Version-Release number of selected component (if applicable):
current CMO with prometheus-operator 0.50.0 (https://github.com/openshift/prometheus-operator/pull/133)


How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 6 Junqi Zhao 2021-09-13 06:35:54 UTC
tested with 4.10.0-0.nightly-2021-09-10-083647, prometheus pods are normal
# oc -n openshift-monitoring get prometheus k8s -oyaml | grep grpc-address
no result

# oc -n openshift-user-workload-monitoring get prometheus user-workload -oyaml | grep grpc-address
no result

# oc -n openshift-monitoring get pod | grep prometheus-k8s
prometheus-k8s-0                              7/7     Running   0               3h14m
prometheus-k8s-1                              7/7     Running   0               3h14m

# oc -n openshift-user-workload-monitoring get pod | grep prometheus-user-workload
prometheus-user-workload-0             5/5     Running   0          9m40s
prometheus-user-workload-1             5/5     Running   0          9m40s

Comment 7 Junqi Zhao 2021-09-13 07:18:12 UTC
address now is 0.0.0.0, instead of pod ip
# oc -n openshift-monitoring logs -c  thanos-sidecar prometheus-k8s-0 | grep sidecar | grep "listening for serving gRPC"
level=info ts=2021-09-13T03:20:35.200956589Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
# oc -n openshift-user-workload-monitoring logs -c  thanos-sidecar prometheus-user-workload-0 | grep sidecar | grep "listening for serving gRPC"
level=info ts=2021-09-13T06:24:52.919399744Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901

Comment 12 errata-xmlrpc 2022-03-12 04:37:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.