Hide Forgot
Description of problem: Hawkular-metrics pod is repeating [0] over and over (dozens of times every second) and rather than signaling that the pod is not healthy and needs to restart it just sits there retrying over and over again. [0] ERROR [org.hawkular.metrics.api.jaxrs.util.ApiUtils] (RxComputationScheduler-4) HAWKMETRICS200010: Failed to process request: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) Additional Information: Node the pod was deployed to ran out of Docker Storage if that might introduce an additional factor
We should probably decrease the amount of error messages being generated in this case. I created a JIRA for this here: https://issues.jboss.org/browse/HWKMETRICS-513) The Hawkular Metrics pod should not be restarted in this situation. The problem is it is not able to connect to the Cassandra instance as as such restarting the Hawkular Metrics pod is not going to resolve that. When the Cassandra connection is valid again, Hawkular Metrics should just be able to reconnect and continue to function. If this is not the case (eg restarting the Hawkular Metrics pod did fix this issue) then we will need to fix that sitation. When you say "Node the pod was deployed to ran out of Docker Storage" are you talking about the Cassandra Pod or the Hawkular Metrics pod?
I am requesting more information about which pod ran out of storage. The Hawkular Metrics pod not restarting is expected and desired behaviour, it should automatically restart once the Cassandra instance is available again. The contestant logging is a more pressing concern that needs to be looked into.
Lowing the priority of this issue as its more of a problem with logging in specific error conditions than with reduced functionality. It is something we need to handle in a better fashion though.
My apologies for the delay. The storage I referenced was not the the storage provided to the pod but the docker storage setup for the node that the cassandra and hawkular metrics pods were running on. Unfortunately the associated case is closed but I will be attaching the logs, provided at the start of the case, shortly.
Upstream tracking https://issues.jboss.org/browse/HWKMETRICS-513