Bug 1639097

Created attachment 1493909 [details]
server returned HTTP status 403 Forbidden for alert-proxy container endpoint

+++ This bug was initially created as a clone of Bug #1638663 +++

Description of problem:
Deploy prometheus v3.9.45-1

# oc -n openshift-metrics get pod -o wide
NAME                             READY     STATUS    RESTARTS   AGE       IP               NODE
prometheus-0                     6/6       Running   0          3h        10.2.2.4         share3-wmengr76o39-master-etcd-2
prometheus-node-exporter-25v67   1/1       Running   0          3h        192.168.100.14   share3-wmengr76o39-nrri-1
prometheus-node-exporter-9v6gs   1/1       Running   0          3h        192.168.100.12   share3-wmengr76o39-master-etcd-3
prometheus-node-exporter-bkn67   1/1       Running   0          3h        192.168.100.20   share3-wmengr76o39-node-primary-3
prometheus-node-exporter-d9wfc   1/1       Running   0          3h        192.168.100.8    share3-wmengr76o39-node-primary-1
prometheus-node-exporter-fnngw   1/1       Running   0          3h        192.168.100.9    share3-wmengr76o39-nrri-2
prometheus-node-exporter-g7km9   1/1       Running   0          3h        192.168.100.4    share3-wmengr76o39-master-etcd-1
prometheus-node-exporter-jlf2v   1/1       Running   0          3h        192.168.100.16   share3-wmengr76o39-node-primary-2
prometheus-node-exporter-k986p   1/1       Running   0          3h        192.168.100.7    share3-wmengr76o39-master-etcd-2


Checked the targets, server returned HTTP status 403 Forbidden for alert-proxy container endpoint

alert-proxy container endpoint could be accssed by the following
get prometheus token first
# oc sa get-token prometheus -n openshift-metrics

# oc -n openshift-metrics rsh prometheus-0
sh-4.2$ curl -k -H "Authorization: Bearer ${prometheus_token}" curl -k https://10.2.2.4:9443/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
................................................................................


Version-Release number of selected component (if applicable):
prometheus v3.9.45-1

How reproducible:
Always

Steps to Reproduce:
1. Deploy prometheus v3.9.45-1 and check /targets page
2.
3.

Actual results:
server returned HTTP status 403 Forbidden for alert-proxy container endpoint

Expected results:
endpoints should are in UP state

Additional info: