Bug 1692281 - Prometheus is not showing metrics for ASB
Summary: Prometheus is not showing metrics for ASB
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.2.0
Assignee: Shawn Hurley
QA Contact: Zhang Cheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-25 09:18 UTC by Zhang Cheng
Modified: 2019-10-16 06:28 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: operator did not create the permissions to get metrics working Consequence: Users had to add the permissions themselves and we added a doc's section for this Fix: Should happen transparently and we should be able to remove that section of the docs Result:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:27:58 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:28:10 UTC

Description Zhang Cheng 2019-03-25 09:18:40 UTC
Description of problem:
Prometheus is not showing metrics for ASB, although we can get correct metrics in backend by curl api service.
Theis a similar bug from service-catalog, for you reference: https://bugzilla.redhat.com/show_bug.cgi?id=1690937


Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-03-23-222829   True        False         117m      Cluster version is 4.0.0-0.nightly-2019-03-23-222829

$ oc get csv
NAME                              DISPLAY                      VERSION   REPLACES   PHASE
automationbrokeroperator.v0.2.0   Automation Broker Operator   0.2.0                Succeeded


How reproducible:
Always


Steps to Reproduce:
1. Provision a serviceinstance
2. Check metrics from backend and Prometheus console


Actual results:
Can get metrics by curl api service, but Prometheus is not showing metrics.
curl -k -H "Authorization: Bearer fRWliDbmPOpqAfwQ18pM1zzVL0T_kcc-7RCswUIjups" https://${svc}:1338/metrics | grep -E "asb|bundlelib" | grep -v \#


Expected results:
Prometheus can show correct metrics for ASB.


Additional info:
None

Comment 1 Fabian von Feilitzsch 2019-04-03 16:16:44 UTC
Added ServiceMonitor to asb deployment: https://github.com/openshift/ansible-service-broker/pull/1204

Comment 3 Zhang Cheng 2019-04-09 03:09:12 UTC
PR: https://github.com/openshift/ansible-service-broker/pull/1204 was not in latest downstream image.
Moving status to MODIFIED since not ready for test.

Comment 4 Fabian von Feilitzsch 2019-04-09 17:26:34 UTC
Original change was incorrect strategy, updated here: https://github.com/openshift/ansible-service-broker/pull/1207

Comment 7 Zhang Cheng 2019-05-10 08:45:04 UTC
PR https://github.com/openshift/ansible-service-broker/pull/1207 was not merged in asb 1.4.4 and 1.4.5, not ready for test. 
Could you change the target release to 4.2? Thanks.

Comment 8 Zihan Tang 2019-05-10 09:29:05 UTC
test with latest operator: ose-ansible-service-broker-operator:v4.1.0-201905091018
still not show metrics in monitoring console.

Besides, does the operator support --version command ?

Comment 10 Zihan Tang 2019-05-13 07:14:55 UTC
I found the doc update in PR: https://github.com/openshift/openshift-docs/pull/14814/files#diff-ae005c183b6939f0f58d3de43ceef1adR18, 
so add label `openshift.io/cluster-monitoring=true` in project `openshift-ansible-service-broker`

but asb still not shown in prometheus 'service discovery' page.

in Configuration page, it has found ansible-service-broker monitorservice:
- job_name: openshift-ansible-service-broker/ansible-service-broker/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - openshift-ansible-service-broker
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: ansible-service-broker

checking the monitoring token and asb api: 
$ curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://asb-1338-openshift-ansible-service-broker.apps.zitang-5131.qe.devcluster.openshift.com/metrics | grep -E "asb|bundlelib" | grep -v \#
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11870  100 11870    0     0  19028      0 --:--:-- --:--:-- --:--:-- 19022
asb_deprovision_jobs 0
asb_provision_jobs 0
asb_sandbox 0
asb_specs_deleted 0
asb_specs_total{source="marked_for_deletion"} 1
asb_specs_total{source="rhcc"} 10
asb_update_jobs 0
bundlelib_sandbox 0

$ oc get servicemonitor -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    creationTimestamp: 2019-05-13T06:23:07Z
    generation: 1
    name: ansible-service-broker
    namespace: openshift-ansible-service-broker
    ownerReferences:
    - apiVersion: osb.openshift.io/v1
      kind: AutomationBroker
      name: ansible-service-broker
      uid: 0c4b353a-7546-11e9-b156-060df9331786
    resourceVersion: "51900"
    selfLink: /apis/monitoring.coreos.com/v1/namespaces/openshift-ansible-service-broker/servicemonitors/ansible-service-broker
    uid: 92a5bfd2-7547-11e9-af1f-0e9fa933bc6a
  spec:
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 30s
      port: port-1338
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
    namespaceSelector:
      matchNames:
      - openshift-ansible-service-broker
    selector:
      matchLabels:
        app: ansible-service-broker
        service: asb
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 20 Zihan Tang 2019-06-27 03:11:06 UTC
Verified:
ASB: ose-ansible-service-broker:v4.2.0-201906240232 , version 1.4.5
ASB-operator: ose-ansible-service-broker-operator:v4.2.0-201906221251

If enable Prometheus, should add label  `openshift.io/cluster-monitoring=true`  in ns 'openshift-ansible-service-broker'

Comment 21 errata-xmlrpc 2019-10-16 06:27:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.