Bug 1729297 - Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Summary: Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.2.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks: 1728856 1731006
TreeView+ depends on / blocked
 
Reported: 2019-07-11 19:57 UTC by Jeff Cantrill
Modified: 2019-10-16 06:33 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The authentication class is instantiated multiple times with and without the desired configuration Consequence: User's are denied access to metrics Fix: Add SAR config to the authentication_backend Result: Multi-tenant plugin executes SARs and allows access to metrics
Clone Of: 1728856
: 1731006 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:33:27 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1695 None None None 2019-07-11 20:00:38 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:33:39 UTC

Description Jeff Cantrill 2019-07-11 19:57:58 UTC
+++ This bug was initially created as a clone of Bug #1728856 +++

Description of problem:
Updating to the latest version of the multi-tenant plugin blocks viewing by metrics unless you have cluster admin rights

Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Deploy logging
2. Set up an SA that can scrape metrics
3. curl -kv https://172.30.59.79/_prometheus/metrics -H"Authorization: Bearer $(oc serviceaccounts get-token metrics-test)"


Actual results: 403


Expected results: 200


Additional info: https://github.com/openshift/origin-aggregated-logging/issues/1686

Comment 2 Qiaoling Tang 2019-07-22 07:44:33 UTC
Verified in ose-logging-elasticsearch5-v4.2.0-201907211357

$ oc exec fluentd-mfmkb -- curl -k -H "Authorization: Bearer `oc sa get-token prometheus-k8s -n openshift-monitoring`"   -H "Content-type: application/json" https://172.30.66.253:60000/_prometheus/metrics
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP es_ingest_pipeline_total_count Ingestion total number
# TYPE es_ingest_pipeline_total_count gauge
# HELP es_indices_refresh_listeners_number Number of refresh listeners
# TYPE es_indices_refresh_listeners_number gauge
es_indices_refresh_listeners_number{cluster="elasticsearch",node="elasticsearch-cdm-b65gkd2u-1",nodeid="YlZKKtQZTHy8XtXZmCPwAg",} 0.0
# HELP es_indices_merges_current_number Current rate of merges
# TYPE es_indices_merges_current_number gauge
es_indices_merges_current_number{cluster="elasticsearch",node="elasticsearch-cdm-b65gkd2u-1",nodeid="YlZKKtQZTHy8XtXZmCPwAg",} 0.0
# HELP es_index_requestcache_hit_count Number of hits in request cache
# TYPE es_index_requestcache_hit_count gauge
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.fn6ob.8c200c02-ac50-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="primaries",} 0.0
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.xiu.b69dd8d2-ac47-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="primaries",} 0.0
es_index_requestcache_hit_count{cluster="elasticsearch",index="project.xiaocwan-t.77ec7730-ac2f-11e9-b0af-02f6fb8d3d9a.2019.07.22",context="total",} 0.0

Comment 3 errata-xmlrpc 2019-10-16 06:33:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.