Description of problem: Ceilometer redis IP address is not showing on all controllers in HA deployment with 3 controllers. Version-Release number of selected component (if applicable): openstack-heat-engine-2015.1.0-3.el7ost.noarch openstack-heat-api-cfn-2015.1.0-3.el7ost.noarch openstack-heat-templates-0-0.6.20150605git.el7ost.noarch openstack-heat-api-2015.1.0-3.el7ost.noarch heat-cfntools-1.2.8-2.el7.noarch openstack-heat-common-2015.1.0-3.el7ost.noarch python-heatclient-0.6.0-1.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.0-3.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-9.el7ost.noarch [stack@instack ~]$ rpm -qa | grep puppet openstack-puppet-modules-2015.1.6-1.el7ost.noarch openstack-tripleo-puppet-elements-0.0.1-2.el7ost.noarch puppet-3.6.2-2.el7.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with 3 controller nodes 2. 3. Actual results: Redis backend url in /etc/ceilometer/ceilometer.conf is set only one of the controller nodes. Expected results: All the controller nodes have the redis backend url. Additional info: [heat-admin@overcloud-controller-0 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf backend_url=redis://192.0.2.13:6379 [heat-admin@overcloud-controller-1 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf backend_url=redis://:6379 [heat-admin@overcloud-controller-2 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf backend_url=redis://:6379
This might need updates to make it more similar to what we have in quickstack: https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/pacemaker/ceilometer.pp#L35 https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/pacemaker/redis.pp#L77 https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/load_balancer/redis.pp
neutron port-list | grep redis_virtual_ip | b6240296-0968-4c3d-a6be-2fb2e2371c90 | redis_virtual_ip | fa:16:3e:10:2e:26 | {"subnet_id": "34f46be5-855f-42d7-96cb-0533d916def0", "ip_address": "192.0.2.13"} | The Redis VIP seems to be set in the ceilometer conf file only on one of the controllers. Nevertheles the node where the Redis VIP is set is running Redis on the local IP address not on the VIP: [stack@instack ~]$ ssh heat-admin.2.13 'sudo lsof -i :6379 -n' COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME redis-ser 23585 redis 4u IPv4 194002 0t0 TCP 192.0.2.14:6379 (LISTEN) redis-ser 23585 redis 6u IPv4 197905 0t0 TCP 192.0.2.14:38999->192.0.2.15:6379 (ESTABLISHED)
hi Marisu, redis is meant to bind on the local ip address, the clients have to reach it via the VIP can you please attach log from ceilometer-central ?
Created attachment 1040023 [details] central.log Attached, it's showing connection refused for the VIP address.
Balancing for Redis is false by default [1] we miss to enable it! 1. https://github.com/openstack/puppet-tripleo/blob/master/manifests/loadbalancer.pp#L217
On the non-0 controllers backend_url appear to be misconfigured as well: backend_url=redis://:6379
waiting for envs to come up but setting [1] is trivial may just start a review for that while waiting and then dig at the other issue giulio mentions in #8 above. [1] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller_pacemaker.pp#L58
https://review.openstack.org/#/c/193048/ Enable loadbalancing of the Redis VIP, defaults to False upstream review out as a placeholder fow now to see if it fixes the problem and what else might be needed.
so confirmed by gfidente and also my latest run with downstream tht + the review above shows correct backend_url on controller-1 for example: [root@overcloud-controller-1 heat-admin]# sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf backend_url=redis://192.0.2.14:6379
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1549