Description of problem: Prometheus is not showing metrics for ASB, although we can get correct metrics in backend by curl api service. Theis a similar bug from service-catalog, for you reference: https://bugzilla.redhat.com/show_bug.cgi?id=1690937 Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-23-222829 True False 117m Cluster version is 4.0.0-0.nightly-2019-03-23-222829 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE automationbrokeroperator.v0.2.0 Automation Broker Operator 0.2.0 Succeeded How reproducible: Always Steps to Reproduce: 1. Provision a serviceinstance 2. Check metrics from backend and Prometheus console Actual results: Can get metrics by curl api service, but Prometheus is not showing metrics. curl -k -H "Authorization: Bearer fRWliDbmPOpqAfwQ18pM1zzVL0T_kcc-7RCswUIjups" https://${svc}:1338/metrics | grep -E "asb|bundlelib" | grep -v \# Expected results: Prometheus can show correct metrics for ASB. Additional info: None
Added ServiceMonitor to asb deployment: https://github.com/openshift/ansible-service-broker/pull/1204
PR: https://github.com/openshift/ansible-service-broker/pull/1204 was not in latest downstream image. Moving status to MODIFIED since not ready for test.
Original change was incorrect strategy, updated here: https://github.com/openshift/ansible-service-broker/pull/1207
PR https://github.com/openshift/ansible-service-broker/pull/1207 was not merged in asb 1.4.4 and 1.4.5, not ready for test. Could you change the target release to 4.2? Thanks.
test with latest operator: ose-ansible-service-broker-operator:v4.1.0-201905091018 still not show metrics in monitoring console. Besides, does the operator support --version command ?
I found the doc update in PR: https://github.com/openshift/openshift-docs/pull/14814/files#diff-ae005c183b6939f0f58d3de43ceef1adR18, so add label `openshift.io/cluster-monitoring=true` in project `openshift-ansible-service-broker` but asb still not shown in prometheus 'service discovery' page. in Configuration page, it has found ansible-service-broker monitorservice: - job_name: openshift-ansible-service-broker/ansible-service-broker/0 scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: endpoints namespaces: names: - openshift-ansible-service-broker bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: ansible-service-broker checking the monitoring token and asb api: $ curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://asb-1338-openshift-ansible-service-broker.apps.zitang-5131.qe.devcluster.openshift.com/metrics | grep -E "asb|bundlelib" | grep -v \# % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11870 100 11870 0 0 19028 0 --:--:-- --:--:-- --:--:-- 19022 asb_deprovision_jobs 0 asb_provision_jobs 0 asb_sandbox 0 asb_specs_deleted 0 asb_specs_total{source="marked_for_deletion"} 1 asb_specs_total{source="rhcc"} 10 asb_update_jobs 0 bundlelib_sandbox 0 $ oc get servicemonitor -o yaml apiVersion: v1 items: - apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: creationTimestamp: 2019-05-13T06:23:07Z generation: 1 name: ansible-service-broker namespace: openshift-ansible-service-broker ownerReferences: - apiVersion: osb.openshift.io/v1 kind: AutomationBroker name: ansible-service-broker uid: 0c4b353a-7546-11e9-b156-060df9331786 resourceVersion: "51900" selfLink: /apis/monitoring.coreos.com/v1/namespaces/openshift-ansible-service-broker/servicemonitors/ansible-service-broker uid: 92a5bfd2-7547-11e9-af1f-0e9fa933bc6a spec: endpoints: - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token interval: 30s port: port-1338 scheme: https tlsConfig: insecureSkipVerify: true namespaceSelector: matchNames: - openshift-ansible-service-broker selector: matchLabels: app: ansible-service-broker service: asb kind: List metadata: resourceVersion: "" selfLink: ""
PR: https://github.com/openshift/ansible-service-broker/pull/1227
Verified: ASB: ose-ansible-service-broker:v4.2.0-201906240232 , version 1.4.5 ASB-operator: ose-ansible-service-broker-operator:v4.2.0-201906221251 If enable Prometheus, should add label `openshift.io/cluster-monitoring=true` in ns 'openshift-ansible-service-broker'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922