Bug 1232797 - Ceilometer redis url not showing up on all controllers in HA deployment
Summary: Ceilometer redis url not showing up on all controllers in HA deployment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: Director
Assignee: Giulio Fidente
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-17 13:48 UTC by Marius Cornea
Modified: 2015-08-05 13:54 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.6-11.el7ost
Doc Type: Bug Fix
Doc Text:
Ceilometer uses an incorrect redis VIP for its backend_url. This fix sets the Ceilometer backend_url to one that Heat provides and not constructed during deployment. Ceilometer now uses the correct IP address.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:54:18 UTC
Target Upstream Version:


Attachments (Terms of Use)
central.log (5.68 KB, text/plain)
2015-06-17 16:42 UTC, Marius Cornea
no flags Details


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 193048 None None None Never
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Marius Cornea 2015-06-17 13:48:26 UTC
Description of problem:
Ceilometer redis IP address is not showing on all controllers in HA deployment with 3 controllers.

Version-Release number of selected component (if applicable):
openstack-heat-engine-2015.1.0-3.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-3.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-heat-api-2015.1.0-3.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.0-3.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-3.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-9.el7ost.noarch
[stack@instack ~]$ rpm -qa | grep puppet
openstack-puppet-modules-2015.1.6-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-2.el7ost.noarch
puppet-3.6.2-2.el7.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with 3 controller nodes
2.
3.

Actual results:
Redis backend url in /etc/ceilometer/ceilometer.conf is set only one of the controller nodes. 

Expected results:
All the controller nodes have the redis backend url.

Additional info:
[heat-admin@overcloud-controller-0 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf
backend_url=redis://192.0.2.13:6379
[heat-admin@overcloud-controller-1 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf
backend_url=redis://:6379
[heat-admin@overcloud-controller-2 ~]$ sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf
backend_url=redis://:6379

Comment 4 Marius Cornea 2015-06-17 16:10:15 UTC
neutron port-list | grep redis_virtual_ip 
| b6240296-0968-4c3d-a6be-2fb2e2371c90 | redis_virtual_ip   | fa:16:3e:10:2e:26 | {"subnet_id": "34f46be5-855f-42d7-96cb-0533d916def0", "ip_address": "192.0.2.13"} |

The Redis VIP seems to be set in the ceilometer conf file only on one of the controllers. Nevertheles the node where the Redis VIP is set is running Redis on the local IP address not on the VIP:

[stack@instack ~]$ ssh heat-admin@192.0.2.13 'sudo lsof -i :6379 -n'
COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 23585 redis    4u  IPv4 194002      0t0  TCP 192.0.2.14:6379 (LISTEN)
redis-ser 23585 redis    6u  IPv4 197905      0t0  TCP 192.0.2.14:38999->192.0.2.15:6379 (ESTABLISHED)

Comment 5 Giulio Fidente 2015-06-17 16:35:15 UTC
hi Marisu,

redis is meant to bind on the local ip address, the clients have to reach it via the VIP

can you please attach log from ceilometer-central ?

Comment 6 Marius Cornea 2015-06-17 16:42:40 UTC
Created attachment 1040023 [details]
central.log

Attached, it's showing connection refused for the VIP address.

Comment 7 Giulio Fidente 2015-06-17 16:44:47 UTC
Balancing for Redis is false by default [1] we miss to enable it!

1. https://github.com/openstack/puppet-tripleo/blob/master/manifests/loadbalancer.pp#L217

Comment 8 Giulio Fidente 2015-06-17 16:53:54 UTC
On the non-0 controllers backend_url appear to be misconfigured as well:

 backend_url=redis://:6379

Comment 9 Marios Andreou 2015-06-18 09:15:13 UTC
waiting for envs to come up but setting [1] is trivial may just start a review for that while waiting and then dig at the other issue giulio mentions in #8 above.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller_pacemaker.pp#L58

Comment 10 Marios Andreou 2015-06-18 10:20:06 UTC
https://review.openstack.org/#/c/193048/ Enable loadbalancing of the Redis VIP, defaults to False


upstream review out as a placeholder fow now to see if it fixes the problem and what else might be needed.

Comment 11 Marios Andreou 2015-06-18 14:02:36 UTC
so confirmed by gfidente and also my latest run with downstream tht + the review above shows correct backend_url on controller-1 for example:

[root@overcloud-controller-1 heat-admin]# sudo grep backend_url=redis /etc/ceilometer/ceilometer.conf
backend_url=redis://192.0.2.14:6379

Comment 16 errata-xmlrpc 2015-08-05 13:54:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.