Bug 1731006

Summary: Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Product: OpenShift Container Platform Reporter: Jeff Cantrill <jcantril>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: anli, aos-bugs, rmeggins
Target Milestone: ---   
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The authentication class is instantiated multiple times with and without the desired configuration Consequence: User's are denied access to metrics Fix: Add SAR config to the authentication_backend Result: Multi-tenant plugin executes SARs and allows access to metrics
Story Points: ---
Clone Of: 1729297 Environment:
Last Closed: 2019-07-31 02:44:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1729297    
Bug Blocks: 1728856    

Description Jeff Cantrill 2019-07-18 06:36:48 UTC
+++ This bug was initially created as a clone of Bug #1729297 +++

+++ This bug was initially created as a clone of Bug #1728856 +++

Description of problem:
Updating to the latest version of the multi-tenant plugin blocks viewing by metrics unless you have cluster admin rights

Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Deploy logging
2. Set up an SA that can scrape metrics
3. curl -kv https://172.30.59.79/_prometheus/metrics -H"Authorization: Bearer $(oc serviceaccounts get-token metrics-test)"


Actual results: 403


Expected results: 200


Additional info: https://github.com/openshift/origin-aggregated-logging/issues/1686

Comment 2 Anping Li 2019-07-25 05:01:42 UTC
The sa prometheus-k8s can fetch metrics from es. 
sh-4.2# curl -k -H "Authorization: Bearer $token" -H "Content-type: application/json"  https://172.30.119.95:60000/_prometheus/metrics
# HELP es_index_segments_memory_bytes Memory used by segments
# TYPE es_index_segments_memory_bytes gauge
es_index_segments_memory_bytes{cluster="elasticsearch",type="points",index=".kibana",context="total",} 0.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="norms",index=".operations.2019.07.25",context="primaries",} 47552.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="indexwriter",index=".kibana",context="primaries",} 0.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="docvalues",index=".searchguard",context="primaries",} 636.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="termvectors",index=".kibana",context="total",} 0.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="versionmap",index=".operations.2019.07.25",context="total",} 63240.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="terms",index=".operations.2019.07.25",context="primaries",} 537727.0
es_index_segments_memory_bytes{cluster="elasticsearch",type="indexwriter",index=".kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac

Comment 4 Anping Li 2019-07-30 03:01:00 UTC
Also verified using registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.1.4-201906271212.

Comment 6 errata-xmlrpc 2019-07-31 02:44:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1866