Description of problem:

The metrics are running with two node of Cassandra with each replication controller and pv. However, when you scale down one node, the hawkular will fail with following error:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.12.1.5:9042 

Where ip 10.12.1.5 is the ip of the pod which was scaled down.
I found that hawkular when starting is attaching the cassandra nodes like this:

08:02:44,344 INFO  [com.datastax.driver.core.Cluster] (ServerService Thread Pool -- 78) New Cassandra host /10.12.1.5:9042 added
08:02:44,344 INFO  [com.datastax.driver.core.Cluster] (ServerService Thread Pool -- 78) New Cassandra host hawkular-cassandra/172.30.57.143:9042 added

Which one os the pod's ip and one svc's ip. However, if the one pod is scaled down, it will show the error above.
Just adding that the replication factor is set to 1.
(com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
(cf. output included)

I know that there is no HA on Cassandra or load balancing, but the issue is hawkular, why it is taking the ip of the pod and not the ip of the service as primary source?

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.4.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: