Bug 1638663

Summary: [3.9] server returned HTTP status 403 Forbidden for alert-proxy container endpoint
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Paul Gier <pgier>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.9.0CC: minden
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of:
: 1639097 (view as bug list) Environment:
Last Closed: 2018-12-13 19:27:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1639097    
Attachments:
Description Flags
server returned HTTP status 403 Forbidden for alert-proxy container endpoint
none
alert-proxy container endpoint is UP -3.9 none

Description Junqi Zhao 2018-10-12 08:17:54 UTC 
Created attachment 1493170 [details]
server returned HTTP status 403 Forbidden for alert-proxy container endpoint

Description of problem:
Deploy prometheus v3.9.45-1

# oc -n openshift-metrics get pod -o wide
NAME                             READY     STATUS    RESTARTS   AGE       IP               NODE
prometheus-0                     6/6       Running   0          3h        10.2.2.4         share3-wmengr76o39-master-etcd-2
prometheus-node-exporter-25v67   1/1       Running   0          3h        192.168.100.14   share3-wmengr76o39-nrri-1
prometheus-node-exporter-9v6gs   1/1       Running   0          3h        192.168.100.12   share3-wmengr76o39-master-etcd-3
prometheus-node-exporter-bkn67   1/1       Running   0          3h        192.168.100.20   share3-wmengr76o39-node-primary-3
prometheus-node-exporter-d9wfc   1/1       Running   0          3h        192.168.100.8    share3-wmengr76o39-node-primary-1
prometheus-node-exporter-fnngw   1/1       Running   0          3h        192.168.100.9    share3-wmengr76o39-nrri-2
prometheus-node-exporter-g7km9   1/1       Running   0          3h        192.168.100.4    share3-wmengr76o39-master-etcd-1
prometheus-node-exporter-jlf2v   1/1       Running   0          3h        192.168.100.16   share3-wmengr76o39-node-primary-2
prometheus-node-exporter-k986p   1/1       Running   0          3h        192.168.100.7    share3-wmengr76o39-master-etcd-2


Checked the targets, server returned HTTP status 403 Forbidden for alert-proxy container endpoint

alert-proxy container endpoint could be accssed by the following
get prometheus token first
# oc sa get-token prometheus -n openshift-metrics

# oc -n openshift-metrics rsh prometheus-0
sh-4.2$ curl -k -H "Authorization: Bearer ${prometheus_token}" curl -k https://10.2.2.4:9443/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
................................................................................


Version-Release number of selected component (if applicable):
prometheus v3.9.45-1

How reproducible:
Always

Steps to Reproduce:
1. Deploy prometheus v3.9.45-1 and check /targets page
2.
3.

Actual results:
server returned HTTP status 403 Forbidden for alert-proxy container endpoint

Expected results:
endpoints should are in UP state

Additional info:
Comment 1 Junqi Zhao 2018-10-12 08:18:34 UTC 
This issue only happen with prometheus 3.9
Comment 2 Paul Gier 2018-10-17 16:12:56 UTC 
This will be fixed by PR: https://github.com/openshift/openshift-ansible/pull/10424
Comment 3 Paul Gier 2018-11-02 14:40:58 UTC 
Fixed in 3.9.49
Comment 4 Junqi Zhao 2018-11-05 08:53:11 UTC 
Issue is fixed, alert-proxy container endpoint is UP now
openshift-ansible:openshift-ansible-3.10.69-1
Comment 5 Junqi Zhao 2018-11-05 08:54:48 UTC 
Created attachment 1501687 [details]
alert-proxy container endpoint is UP -3.9
Comment 6 Junqi Zhao 2018-11-05 08:55:44 UTC 
(In reply to Junqi Zhao from comment #4)
> openshift-ansible:openshift-ansible-3.10.69-1

Correction:
openshift-ansible:openshift-ansible-3.9.50-1
Comment 9 errata-xmlrpc 2018-12-13 19:27:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3748