Bug 1374326

Summary: Auth directive missing in haproxy.cfg for redis
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED DUPLICATE QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: bperkins, dbecker, mburns, morazi, pkilambi, rhel-osp-director-maint
Target Milestone: ga   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-08 13:03:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Yurii Prokulevych 2016-09-08 12:50:56 UTC
Description of problem:
-----------------------
Redis' section in haproxy.conf missing authentication step, though authentication in redis is configured. This causes a telemtery* services to fail.

Excerpt from ceilometer/central.log:
------------------------------------
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination [-] Error connecting to coordination backend.
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination Traceback (most recent call last):
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/ceilometer/coordination.py", line 102, in start
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     self._coordinator.start()
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 383, in start
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     self._start()
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 439, in _start
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     self._server_info = self._client.info()
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     self.gen.throw(type, value, traceback)
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 50, in _translate_failures
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     cause=e)
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination     raise value
2016-09-08 12:04:04.186 29129 ERROR ceilometer.coordination ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)

Excerpt from gnocchi/metricd.log:
---------------------------------
2016-09-08 12:23:15.000 18260 ERROR cotyledon Traceback (most recent call last):
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/cotyledon/__init__.py", line 62, in _exit_on_exception
2016-09-08 12:23:15.000 18260 ERROR cotyledon     yield
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/cotyledon/__init__.py", line 404, in _start_service
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self._current_process = config.service(worker_id, *args, **kwargs)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 155, in __init__
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self._enable_coordination(conf)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 149, in _enable_coordination
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self._coord.start(start_heart=True)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 383, in start
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self._start()
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 439, in _start
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self._server_info = self._client.info()
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2016-09-08 12:23:15.000 18260 ERROR cotyledon     self.gen.throw(type, value, traceback)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 50, in _translate_failures
2016-09-08 12:23:15.000 18260 ERROR cotyledon     cause=e)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2016-09-08 12:23:15.000 18260 ERROR cotyledon     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2016-09-08 12:23:15.000 18260 ERROR cotyledon     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2016-09-08 12:23:15.000 18260 ERROR cotyledon   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2016-09-08 12:23:15.000 18260 ERROR cotyledon     raise value
2016-09-08 12:23:15.000 18260 ERROR cotyledon ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)
2016-09-08 12:23:15.000 18260 ERROR cotyledon 


Adding next snippet to haproxy.conf[redis] resolved issue:

tcp-check send AUTH\ 4DHUCCwufYFK8FZwAahtDWtM3\r\n

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-heat-templates-5.0.0-0.20160823140311.72404b.1.el7ost.noarch
openstack-tripleo-common-5.0.0-0.20160823132420.be5b0d3.el7ost.noarch
openstack-tripleo-0.0.1-0.20160817001035.04d3c80.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20160809165922.703eeea.el7ost.noarch
puppet-tripleo-5.0.0-0.20160823145616.c16ae9c.el7ost.noarch
python-tripleoclient-5.0.0-0.20160823121648.2a9a168.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-0.20160810171527.98b9c6a.el7ost.noarch


Steps to Reproduce:
-------------------
Overcloud deployed with next command:
openstack overcloud deploy --ntp-server clock.redhat.com \
--templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 \
--libvirt-type qemu -e puppet-pacemaker.yaml -e storage-environment.yaml

Comment 1 Pradeep Kilambi 2016-09-08 13:03:37 UTC

*** This bug has been marked as a duplicate of bug 1371657 ***