Bug 1728856 - Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Summary: Prometheus metrics for ES are unavailable after plugin update to 5.6.13.6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.11.z
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1729297 1731006
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-10 19:18 UTC by Jeff Cantrill
Modified: 2020-02-19 19:53 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The authentication class is instantiated multiple times with and without the desired configuration Consequence: User's are denied access to metrics Fix: Add SAR config to the authentication_backend Result: Multi-tenant plugin executes SARs and allows access to metrics
Clone Of:
: 1729297 (view as bug list)
Environment:
Last Closed: 2020-02-19 19:53:43 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1691 None closed Bug 1728856: Fix prometheus metrics visibility 2020-11-27 06:29:00 UTC
Red Hat Product Errata RHBA-2020:0402 None None None 2020-02-19 19:53:57 UTC

Description Jeff Cantrill 2019-07-10 19:18:03 UTC
Description of problem:
Updating to the latest version of the multi-tenant plugin blocks viewing by metrics unless you have cluster admin rights

Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Deploy logging
2. Set up an SA that can scrape metrics
3. curl -kv https://172.30.59.79/_prometheus/metrics -H"Authorization: Bearer $(oc serviceaccounts get-token metrics-test)"


Actual results: 403


Expected results: 200


Additional info: https://github.com/openshift/origin-aggregated-logging/issues/1686

Comment 3 Anping Li 2019-08-07 07:34:01 UTC
The prometheus-k8s couldn't access the svc logging-es-prometheus. the sa with cluster-monitoring-view could't access the svc logging-es-prometheus.   



oc get svc
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
logging-es              ClusterIP   172.31.31.229    <none>        9200/TCP   5h
logging-es-cluster      ClusterIP   None             <none>        9300/TCP   1h
logging-es-prometheus   ClusterIP   172.31.215.247   <none>        443/TCP    5h
logging-kibana          ClusterIP   172.31.120.102   <none>        443/TCP    5h

oc get sa -n openshift-monitoring
NAME                          SECRETS   AGE
alertmanager-main             2         3h
builder                       2         3h
cluster-monitoring-operator   2         3h
default                       2         3h
deployer                      2         3h
grafana                       2         3h
kube-state-metrics            2         3h
node-exporter                 2         3h
prometheus-k8s                2         3h
prometheus-operator           2         3h

Comment 5 Jeff Cantrill 2020-02-02 01:25:21 UTC
(In reply to Anping Li from comment #3)
> The prometheus-k8s couldn't access the svc logging-es-prometheus. the sa
> with cluster-monitoring-view could't access the svc logging-es-prometheus.   
> 

Moving back to ON_QA.  Can the serviceaccount satisfy this SubjectAccessReview: https://github.com/openshift/origin-aggregated-logging/pull/1691/files#diff-84bb2a16fa453407379d6048d001a362R19-R22

Comment 6 Anping Li 2020-02-04 08:54:27 UTC
Verified in v3.11.169
1) oc create rolebinding --role=prometheus-metrics-viewer metrics-test-reader-1 --serviceaccount openshift-monitoring:prometheus-k8s -n openshift-logging
2) get token from prometheus pod
oc exec  prometheus-k8s-0 -n  openshift-monitoring -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
3) oc get svc logging-es-prometheus -n openshift-logging
4) access metrics via token
curl -kv https://$svc_ip/_prometheus/metrics -H"Authorization: Bearer $token"

Comment 9 errata-xmlrpc 2020-02-19 19:53:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0402


Note You need to log in before you can comment on or make changes to this bug.