New feature has been added to support Ceilometer coordination. New parameters have been added for the feature: CONFIG_CEILOMETER_COORDINATION_BACKEND, CONFIG_REDIS_MASTER_HOST, CONFIG_REDIS_PORT, CONFIG_REDIS_PORT, CONFIG_REDIS_SLAVE_HOSTS, CONFIG_REDIS_SENTINEL_HOSTS, CONFIG_REDIS_SENTINEL_CONTACT_HOST, CONFIG_REDIS_SENTINEL_PORT, CONFIG_REDIS_SENTINEL_QUORUM and CONFIG_REDIS_MASTER_NAME. For details check documentation.
Ceilometer now uses the tooz coordination library to allow workload partitioning across scaled out central agents (previously a SPoF).
For consistency, the ceilometer alarm evaluator can also optionally be scaled out in that way.
At a minimum this should allow a single redis instance on the controller plane to be selected as the coordination.backend_url in the ceilometer.conf.
It appears that while simple single server redis-based coordination with tooz will work fine, there are issues when using redis sentinel to support HA redis.
I discovered this while writing up some notes on how to verify that it works. In typical fashion, that led to discovering that it doesn't work. Those notes are here:
https://tank.peermore.com/tanks/cdent-rhat/TestCeiloRedisPackstack
My apologies for the roughness, it's become a bit late in the day.
The gist is that there are at least three problems:
* a newer version of tooz is required in the juno-based packstack (0.9.0 is in there, we need at least 0.11.0)
* the default timeouts for sentinel driven failover may be a bit slow for some situations, or there is some other kind of timing issue which is making it longer than expected for a new master to be available
* the ceilometer code is not properly handling failure to retry contacting the tooz server
More detail in the link above. More detailed bugs will be created up and downstream after discussion with eglynn and yprokule tomorrow.
Comment 6Yurii Prokulevych
2015-03-19 17:03:58 UTC
"Redis HA" mode cannot be verified due to bug 1203706 "ceilometer group partitioning coordination with tooz+redis+sentinel fails to failover to new master".
Comment 7Yurii Prokulevych
2015-03-19 17:05:02 UTC
The fix for this on upstream tooz was merge from https://review.openstack.org/#/c/166291/
There has not yet been a tooz release, the most recent was a few days prior to that merge.