Red Hat Bugzilla – Bug 1459877
Hawkular-metrics fails when one instance Cassandra is disabled
Last modified: 2017-06-08 14:55:12 EDT
Description of problem:
The metrics are running with two node of Cassandra with each replication controller and pv. However, when you scale down one node, the hawkular will fail with following error:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.12.1.5:9042
Where ip 10.12.1.5 is the ip of the pod which was scaled down.
I found that hawkular when starting is attaching the cassandra nodes like this:
08:02:44,344 INFO [com.datastax.driver.core.Cluster] (ServerService Thread Pool -- 78) New Cassandra host /10.12.1.5:9042 added
08:02:44,344 INFO [com.datastax.driver.core.Cluster] (ServerService Thread Pool -- 78) New Cassandra host hawkular-cassandra/172.30.57.143:9042 added
Which one os the pod's ip and one svc's ip. However, if the one pod is scaled down, it will show the error above.
Just adding that the replication factor is set to 1.
(com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
(cf. output included)
I know that there is no HA on Cassandra or load balancing, but the issue is hawkular, why it is taking the ip of the pod and not the ip of the service as primary source?
Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.4.0
Steps to Reproduce:
I am marking this as a duplicate of 1459968
*** This bug has been marked as a duplicate of bug 1459968 ***