Bug 1729297

Summary: Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Product: OpenShift Container Platform Reporter: Jeff Cantrill <jcantril>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: anli, aos-bugs, qitang, rmeggins
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The authentication class is instantiated multiple times with and without the desired configuration Consequence: User's are denied access to metrics Fix: Add SAR config to the authentication_backend Result: Multi-tenant plugin executes SARs and allows access to metrics
Story Points: ---
Clone Of: 1728856
: 1731006 (view as bug list) Environment:
Last Closed: 2019-10-16 06:33:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1728856, 1731006    

Description Jeff Cantrill 2019-07-11 19:57:58 UTC
+++ This bug was initially created as a clone of Bug #1728856 +++

Description of problem:
Updating to the latest version of the multi-tenant plugin blocks viewing by metrics unless you have cluster admin rights

Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Deploy logging
2. Set up an SA that can scrape metrics
3. curl -kv https://172.30.59.79/_prometheus/metrics -H"Authorization: Bearer $(oc serviceaccounts get-token metrics-test)"


Actual results: 403


Expected results: 200


Additional info: https://github.com/openshift/origin-aggregated-logging/issues/1686

Comment 2 Qiaoling Tang 2019-07-22 07:44:33 UTC
Verified in ose-logging-elasticsearch5-v4.2.0-201907211357

$ oc exec fluentd-mfmkb -- curl -k -H "Authorization: Bearer `oc sa get-token prometheus-k8s -n openshift-monitoring`"   -H "Content-type: application/json" https://172.30.66.253:60000/_prometheus/metrics
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP es_ingest_pipeline_total_count Ingestion total number
# TYPE es_ingest_pipeline_total_count gauge
# HELP es_indices_refresh_listeners_number Number of refresh listeners
# TYPE es_indices_refresh_listeners_number gauge
es_indices_refresh_listeners_number{cluster="elasticsearch",node="elasticsearch-cdm-b65gkd2u-1",nodeid="YlZKKtQZTHy8XtXZmCPwAg",} 0.0
# HELP es_indices_merges_current_number Current rate of merges
# TYPE es_indices_merges_current_number gauge
es_indices_merges_current_number{cluster="elasticsearch",node="elasticsearch-cdm-b65gkd2u-1",nodeid="YlZKKtQZTHy8XtXZmCPwAg",} 0.0
# HELP es_index_requestcache_hit_count Number of hits in request cache
# TYPE es_index_requestcache_hit_count gauge
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.fn6ob.8c200c02-ac50-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="primaries",} 0.0
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.xiu.b69dd8d2-ac47-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="primaries",} 0.0
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.xiaocwan-t.77ec7730-ac2f-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="total",} 0.0

Comment 3 errata-xmlrpc 2019-10-16 06:33:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922