Bug 1782061 - template-service-broker-operator doesn't notify users and admins via alerts in prometheus
Summary: template-service-broker-operator doesn't notify users and admins via alerts i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.4.z
Assignee: Jesus M. Rodriguez
QA Contact: Cuiping HUO
URL:
Whiteboard:
Depends On: 1800645
Blocks: 1776133
TreeView+ depends on / blocked
 
Reported: 2019-12-11 05:19 UTC by Jesus M. Rodriguez
Modified: 2020-06-02 11:18 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1776133
Environment:
Last Closed: 2020-06-02 11:17:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift template-service-broker-operator pull 101 0 None closed Bug 1782061: Allow the PrometheusAlert to fire. 2020-06-18 14:14:24 UTC
Github openshift template-service-broker-operator pull 63 0 None closed Bug 1782061: Allow TSBO to access endpoints 2020-06-18 14:14:24 UTC
Red Hat Product Errata RHBA-2020:2310 0 None None None 2020-06-02 11:18:14 UTC

Comment 2 Cuiping HUO 2019-12-17 06:45:40 UTC
No packagemanifest is ready for 4.4 verification.

Comment 3 Jesus M. Rodriguez 2020-02-07 16:12:43 UTC
This bug depends on PR https://github.com/openshift/template-service-broker-operator/pull/68 to re-enabled TSBO

Comment 4 Jesus M. Rodriguez 2020-02-25 21:17:29 UTC
Now that PR 68 merged, this should be ready to test.

Comment 6 Cuiping HUO 2020-03-05 09:46:19 UTC
Verification failed.
cluster version:4.4.0-0.nightly-2020-03-04-230603
templateservicebroker_info metrics works, but 
1.Alert: TemplateServiceBrokerEnabled is not firing.
2.prometheus rule with alert name  TemplateServiceBrokerEnabled should have a message field.

$ $ oc get ep -n openshift-template-service-broker
NAME                                                 ENDPOINTS                           AGE
apiserver                                            10.128.2.12:8443                    79m
openshift-template-service-broker-operator-metrics   10.131.0.32:8383,10.131.0.32:8686   79m

$ token=`oc -n openshift-monitoring sa get-token prometheus-k8s`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" 'https://10.128.2.12:8443/metrics' | grep template_service_broker
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17482    0 17482    0     0   116k      0 --:--:-- --:--:-- --:--:--  116k
# HELP template_service_broker_build_info A metric with a constant '1' value labeled by major, minor, git commit & git version from which Template Service Broker was built.
# TYPE template_service_broker_build_info gauge
template_service_broker_build_info{gitCommit="58203d1a763764c7eec567473304e831946ab8c3",gitVersion="v0.0.0-alpha.0-5-g58203d1a",major="",minor=""} 1

$ oc get csv -n openshift-template-service-broker
NAME                                                        DISPLAY                                      VERSION              REPLACES   PHASE
openshifttemplateservicebrokeroperator.4.4.0-202003021217   OpenShift Template Service Broker Operator   4.4.0-202003021217              Succeeded

Comment 8 Jesus M. Rodriguez 2020-03-06 00:53:08 UTC
I initially thought it was searching for the wrong metric, but it doesn't look like the metric is being exposed from the operator properly. The one seen is the build-info which wasn't the one I expected. This was not the quick fix I was hoping for so punting for now.

Comment 10 Jesus M. Rodriguez 2020-05-15 03:33:32 UTC
Bug has been reviewed this sprint.

Comment 13 Fan Jia 2020-05-28 11:06:34 UTC
test env:
cv: 4.4.5
template broker version:openshifttemplateservicebrokeroperator.4.4.0-202005252114

test result:
1)$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" http://10xxxxx6:8686/metrics | grep templateservicebroker_info
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   259  100   259    0     0  20734      0 --:--:-- --:--:-- --:--:-- 21583
# HELP templateservicebroker_info Information about the TemplateServiceBroker custom resource.
# TYPE templateservicebroker_info gauge
templateservicebroker_info{namespace="openshift-template-service-broker",templateservicebroker="template-service-broker"} 1

2)$ oc -n openshift-template-service-broker get prometheusrules
NAME                      AGE
template-service-broker   42m

3)TemplateServiceBrokerEnabled is firing in the Alerting page. But the alert doesn't have "message" to describe the content about the alert. Will open one more bug to track this problem.

Comment 15 errata-xmlrpc 2020-06-02 11:17:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2310


Note You need to log in before you can comment on or make changes to this bug.