Description of problem: The prometheus pod never starts due to thanos-sidecar. https://github.com/prometheus-operator/prometheus-operator/pull/4038 removed the env var replacement that we rely on in CMO. We need to remove [$(POD_IP)]. Version-Release number of selected component (if applicable): current CMO with prometheus-operator 0.50.0 (https://github.com/openshift/prometheus-operator/pull/133) How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
tested with 4.10.0-0.nightly-2021-09-10-083647, prometheus pods are normal # oc -n openshift-monitoring get prometheus k8s -oyaml | grep grpc-address no result # oc -n openshift-user-workload-monitoring get prometheus user-workload -oyaml | grep grpc-address no result # oc -n openshift-monitoring get pod | grep prometheus-k8s prometheus-k8s-0 7/7 Running 0 3h14m prometheus-k8s-1 7/7 Running 0 3h14m # oc -n openshift-user-workload-monitoring get pod | grep prometheus-user-workload prometheus-user-workload-0 5/5 Running 0 9m40s prometheus-user-workload-1 5/5 Running 0 9m40s
address now is 0.0.0.0, instead of pod ip # oc -n openshift-monitoring logs -c thanos-sidecar prometheus-k8s-0 | grep sidecar | grep "listening for serving gRPC" level=info ts=2021-09-13T03:20:35.200956589Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901 # oc -n openshift-user-workload-monitoring logs -c thanos-sidecar prometheus-user-workload-0 | grep sidecar | grep "listening for serving gRPC" level=info ts=2021-09-13T06:24:52.919399744Z caller=grpc.go:123 service=gRPC/server component=sidecar msg="listening for serving gRPC" address=0.0.0.0:10901
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056