Created attachment 1493170 [details] server returned HTTP status 403 Forbidden for alert-proxy container endpoint Description of problem: Deploy prometheus v3.9.45-1 # oc -n openshift-metrics get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE prometheus-0 6/6 Running 0 3h 10.2.2.4 share3-wmengr76o39-master-etcd-2 prometheus-node-exporter-25v67 1/1 Running 0 3h 192.168.100.14 share3-wmengr76o39-nrri-1 prometheus-node-exporter-9v6gs 1/1 Running 0 3h 192.168.100.12 share3-wmengr76o39-master-etcd-3 prometheus-node-exporter-bkn67 1/1 Running 0 3h 192.168.100.20 share3-wmengr76o39-node-primary-3 prometheus-node-exporter-d9wfc 1/1 Running 0 3h 192.168.100.8 share3-wmengr76o39-node-primary-1 prometheus-node-exporter-fnngw 1/1 Running 0 3h 192.168.100.9 share3-wmengr76o39-nrri-2 prometheus-node-exporter-g7km9 1/1 Running 0 3h 192.168.100.4 share3-wmengr76o39-master-etcd-1 prometheus-node-exporter-jlf2v 1/1 Running 0 3h 192.168.100.16 share3-wmengr76o39-node-primary-2 prometheus-node-exporter-k986p 1/1 Running 0 3h 192.168.100.7 share3-wmengr76o39-master-etcd-2 Checked the targets, server returned HTTP status 403 Forbidden for alert-proxy container endpoint alert-proxy container endpoint could be accssed by the following get prometheus token first # oc sa get-token prometheus -n openshift-metrics # oc -n openshift-metrics rsh prometheus-0 sh-4.2$ curl -k -H "Authorization: Bearer ${prometheus_token}" curl -k https://10.2.2.4:9443/metrics # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0 go_gc_duration_seconds{quantile="0.25"} 0 go_gc_duration_seconds{quantile="0.5"} 0 go_gc_duration_seconds{quantile="0.75"} 0 go_gc_duration_seconds{quantile="1"} 0 go_gc_duration_seconds_sum 0 go_gc_duration_seconds_count 0 # HELP go_goroutines Number of goroutines that currently exist. ................................................................................ Version-Release number of selected component (if applicable): prometheus v3.9.45-1 How reproducible: Always Steps to Reproduce: 1. Deploy prometheus v3.9.45-1 and check /targets page 2. 3. Actual results: server returned HTTP status 403 Forbidden for alert-proxy container endpoint Expected results: endpoints should are in UP state Additional info:
This issue only happen with prometheus 3.9
This will be fixed by PR: https://github.com/openshift/openshift-ansible/pull/10424
Fixed in 3.9.49
Issue is fixed, alert-proxy container endpoint is UP now openshift-ansible:openshift-ansible-3.10.69-1
Created attachment 1501687 [details] alert-proxy container endpoint is UP -3.9
(In reply to Junqi Zhao from comment #4) > openshift-ansible:openshift-ansible-3.10.69-1 Correction: openshift-ansible:openshift-ansible-3.9.50-1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3748