Bug 1508496

Created attachment 1346560 [details]
metrics pods info

The ansible scripts set the ENABLE_PROMETHEUS_ENDPOINT variable to a value of "True". The cassandra-docker.sh script which checks to see if the variable is set looks for a value of "true". The same is true for hawkular-metrics in the standalone.conf script. You can work around this by do the following:

1) `oc edit rc hawkular-cassandra-1`  and set value of ENABLE_PROMETHEUS_ENDPOINT to true. Save the changes.

2) `oc edit rc hawkular-metrics` and set the value of ENABLE_PROMETHEUS_ENDPOINT to true. Save the changes.

3) `oc scale --replicas=0 rc hawkular-cassandra-1`

4) `oc scale --replicas=0 rc hawkular-metrics`

5) `oc scale --replicas=1 rc hawkular-cassandra-1`

6) `oc scale --replicas=1 rc hawkular-metrics`


For the permanent fix, I will need to update the cassandra-docker.sh and standalone.conf scripts.

Moving target release to 3.8 since there is a work around that I described in comment 2.

(In reply to John Sanda from comment #2)
> The ansible scripts set the ENABLE_PROMETHEUS_ENDPOINT variable to a value
> of "True". The cassandra-docker.sh script which checks to see if the
> variable is set looks for a value of "true". The same is true for
> hawkular-metrics in the standalone.conf script. You can work around this by
> do the following:
> 
> 1) `oc edit rc hawkular-cassandra-1`  and set value of
> ENABLE_PROMETHEUS_ENDPOINT to true. Save the changes.
> 
> 2) `oc edit rc hawkular-metrics` and set the value of
> ENABLE_PROMETHEUS_ENDPOINT to true. Save the changes.
> 
> 3) `oc scale --replicas=0 rc hawkular-cassandra-1`
> 
> 4) `oc scale --replicas=0 rc hawkular-metrics`
> 
> 5) `oc scale --replicas=1 rc hawkular-cassandra-1`
> 
> 6) `oc scale --replicas=1 rc hawkular-metrics`
> 
> 
> For the permanent fix, I will need to update the cassandra-docker.sh and
> standalone.conf scripts.

another workaround is set the following parameters in inventory file
openshift_metrics_cassandra_enable_prometheus_endpoint=true
openshift_metrics_hawkular_enable_prometheus_endpoint=true

I have created https://github.com/openshift/origin-metrics/pull/404 to fix this.

Issue is not fixed, it is changed to ON_QA by errata, change back to MODIFIED

# oc get po -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP            NODE
hawkular-cassandra-1-pdnb5   1/1       Running   0          36m       10.129.0.13   172.16.120.17
hawkular-metrics-2cjcx       1/1       Running   2          36m       10.129.0.12   172.16.120.17
heapster-mhw94               1/1       Running   2          36m       10.128.0.13   172.16.120.59
# curl http://10.129.0.13:7575/metrics
curl: (7) Failed connect to 10.129.0.13:7575; Connection refused
# curl http://10.129.0.12:7575/metrics
curl: (7) Failed connect to 10.129.0.12:7575; Connection refused

Images:
metrics-cassandra-v3.9.2-1
metrics-hawkular-metrics-v3.9.2-1
metrics-heapster-v3.9.2-1

I saw the PR for this fix and it seems like those changes are not on the latest build. I'll do a new build with those changes.

Could get hawkular-cassandra and hawkular-metrics prometheus metrics by command now, the output see the attached file
# curl http://${POD_IP}:7575/metrics

Images
metrics-cassandra-v3.9.4-1
metrics-hawkular-metrics-v3.9.4-1
metrics-heapster-v3.9.4-1

# openshift version
openshift v3.9.3
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16

Created attachment 1406065 [details]
issue is fixed

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489