Bug 1482198

Summary:	gnocchi-statsd fails to start with cradox conf_read_file error
Product:	Red Hat OpenStack	Reporter:	Pradeep Kilambi <pkilambi>
Component:	openstack-tripleo-heat-templates	Assignee:	Pradeep Kilambi <pkilambi>
Status:	CLOSED ERRATA	QA Contact:	Sasha Smolyak <ssmolyak>
Severity:	high	Docs Contact:
Priority:	high
Version:	12.0 (Pike)	CC:	apevec, jjoyce, jschluet, lhh, mburns, rhel-osp-director-maint, sasha, tvignaud
Target Milestone:	rc	Keywords:	Triaged
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-13 21:52:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pradeep Kilambi 2017-08-16 17:09:18 UTC

When using ceph backend gnocchi-statsd daemon keeps restarting with the following traceback

2017-08-16 16:53:52.512 1 CRITICAL gnocchi [-] Unhandled error: Error: error calling conf_read_file: err
or code 22
2017-08-16 16:53:52.512 1 ERROR gnocchi Traceback (most recent call last):
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/bin/gnocchi-statsd", line 10, in <module>
2017-08-16 16:53:52.512 1 ERROR gnocchi     sys.exit(statsd())
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 1
02, in statsd
2017-08-16 16:53:52.512 1 ERROR gnocchi     statsd_service.start()
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", lin
e 171, in start
2017-08-16 16:53:52.512 1 ERROR gnocchi     stats = Stats(conf)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", lin
e 38, in __init__
2017-08-16 16:53:52.512 1 ERROR gnocchi     self.storage = storage.get_driver(self.conf)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init_
_.py", line 162, in get_driver
2017-08-16 16:53:52.512 1 ERROR gnocchi     conf.storage, incoming, coord)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py
", line 46, in __init__
2017-08-16 16:53:52.512 1 ERROR gnocchi     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/
ceph.py", line 66, in create_rados_connection
2017-08-16 16:53:52.512 1 ERROR gnocchi     conf=options)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "cradox.pyx", line 545, in cradox.Rados.__init__ (cradox.c:6512)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "cradox.pyx", line 445, in cradox.requires.wrapper.validate_func (cradox.c:4724)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "cradox.pyx", line 588, in cradox.Rados.__setup (cradox.c:7481)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "cradox.pyx", line 445, in cradox.requires.wrapper.validate_func (cradox.c:4724)
2017-08-16 16:53:52.512 1 ERROR gnocchi   File "cradox.pyx", line 651, in cradox.Rados.conf_read_file (cradox.c:8658)
2017-08-16 16:53:52.512 1 ERROR gnocchi Error: error calling conf_read_file: error code 22


Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Alexander Chuzhoy 2017-08-16 17:11:40 UTC

Environment:
python-gnocchi-4.0.0-0.20170612072714.a852437.el7ost.noarch
python-gnocchiclient-3.3.1-0.20170721134935.004bf71.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-4.0.0-0.20170612072714.a852437.el7ost.noarch
openstack-gnocchi-statsd-4.0.0-0.20170612072714.a852437.el7ost.noarch
puppet-gnocchi-11.3.0-0.20170805095843.a7b430b.el7ost.noarch
openstack-gnocchi-common-4.0.0-0.20170612072714.a852437.el7ost.noarch
openstack-gnocchi-metricd-4.0.0-0.20170612072714.a852437.el7ost.noarch
openstack-gnocchi-api-4.0.0-0.20170612072714.a852437.el7ost.noarch

instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch

Comment 2 Pradeep Kilambi 2017-08-16 17:29:01 UTC

we are seeing this in upstream gate as well:

http://logs.openstack.org/26/491826/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/b26a86b/logs/subnode-2/var/log/containers/gnocchi/gnocchi-statsd.log.txt.gz#_2017-08-11_23_27_14_928

Comment 6 Sasha Smolyak 2017-11-09 12:37:52 UTC

logs are clean, 
/var/log/containers/gnocchi/gnocchi-statsd.log:

2017-11-09 11:28:19,153 [1] INFO     gnocchi.storage.common.ceph: Ceph storage backend use 'cradox' python library
2017-11-09 11:28:19,667 [1] INFO     gnocchi.statsd: Created resource 0a8b55df-f90f-491c-8cb9-7cdecec6fc26
2017-11-09 11:28:19,672 [1] INFO     gnocchi.statsd: Started on 0.0.0.0:8125
2017-11-09 11:28:19,672 [1] INFO     gnocchi.statsd: Flush delay: 10 seconds

Comment 9 errata-xmlrpc 2017-12-13 21:52:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462