Bug 1338954 - openstack-gnocchi-statsd fails to start
Summary: openstack-gnocchi-statsd fails to start
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Pradeep Kilambi
QA Contact: Yurii Prokulevych
Depends On:
TreeView+ depends on / blocked
Reported: 2016-05-23 17:40 UTC by Marius Cornea
Modified: 2016-08-11 11:31 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-heat-templates-2.0.0-11.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-08-11 11:31:18 UTC
Target Upstream Version:

Attachments (Terms of Use)
GnocchiUpdate_To_CEPH.log (43.58 KB, text/plain)
2016-07-22 06:26 UTC, Yurii Prokulevych
no flags Details

System ID Priority Status Summary Last Updated
OpenStack gerrit 321133 None MERGED Fix ceph keyring setting for gnocchi 2020-05-20 20:53:36 UTC
Red Hat Product Errata RHEA-2016:1599 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 director Release Candidate Advisory 2016-08-11 15:25:37 UTC

Description Marius Cornea 2016-05-23 17:40:17 UTC
Description of problem:
openstack-gnocchi-statsd pacemaker resources fail to start

/var/log/gnocchi/statsd.log shows:

INFO gnocchi.storage.ceph [-] Ceph storage backend use 'rados' python library
CRITICAL gnocchi [-] ObjectNotFound: error connecting to the cluster
ERROR gnocchi Traceback (most recent call last):
ERROR gnocchi   File "/usr/bin/gnocchi-statsd", line 10, in <module>
ERROR gnocchi     sys.exit(statsd())
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 61, in statsd
ERROR gnocchi     statsd_service.start()
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 174, in start
ERROR gnocchi     stats = Stats(conf)
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__
ERROR gnocchi     self.storage = storage.get_driver(self.conf)
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 155, in get_driver
ERROR gnocchi     return get_driver_class(conf)(conf.storage)
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 80, in __init__
ERROR gnocchi     self.rados.connect()
ERROR gnocchi   File "/usr/lib/python2.7/site-packages/rados.py", line 429, in connect
ERROR gnocchi     raise make_ex(ret, "error connecting to the cluster")
ERROR gnocchi ObjectNotFound: error connecting to the cluster
ERROR gnocchi 
INFO gnocchi.storage.ceph [-] Ceph storage backend use 'rados' python library
CRITICAL gnocchi [-] ObjectNotFound: error connecting to the cluster

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy environment with 3 ctrls, 1 compute and 1 ceph storage node

Additional info:

[root@overcloud-controller-0 heat-admin]# grep -v ^# /etc/gnocchi/gnocchi.conf  | grep -v ^$
log_dir = /var/log/gnocchi
port = 8041
host =
workers = 4
url = mysql+pymysql://gnocchi:xrTTwbaPqbaHTKZxbxmNeRetV@
auth_uri =
identity_uri =
admin_user = gnocchi
admin_password = xrTTwbaPqbaHTKZxbxmNeRetV
admin_tenant_name = service
resource_id = 0a8b55df-f90f-491c-8cb9-7cdecec6fc26
user_id = 27c0d3f8-e7ee-42f0-8317-72237d1c5ae3
project_id = 6c38cd8d-099a-4cb2-aecf-17be688e8616
archive_policy_name = low
driver = ceph
ceph_pool = metrics
ceph_username = openstack
ceph_keyring = client.openstack
ceph_conffile = /etc/ceph/ceph.conf

When I try to use the client.openstack keyring I get an authentication error (22) Invalid argument error:

[root@overcloud-controller-0 ~]# ceph -k /etc/ceph/ceph.client.openstack.keyring  -c /etc/ceph/ceph.conf osd lspools
2016-05-23 17:39:50.778675 7faf00713700  0 librados: client.admin authentication error (22) Invalid argument
Error connecting to cluster: Error

Comment 3 Marius Cornea 2016-05-25 08:41:18 UTC
After setting ceph_keyring = /etc/ceph/ceph.client.openstack.keyring in /etc/gnocchi/gnocchi.conf it started ok. 

The value configured by the installer is: ceph_keyring = client.openstack

Comment 7 Jay Dobies 2016-06-08 20:08:21 UTC
Upstream backport still needs to merge: https://review.openstack.org/#/c/322312/

Comment 10 Alexander Chuzhoy 2016-06-30 22:13:43 UTC
Reproduced upon upgrade from 8.0 to 9.0:

[root@overcloud-controller-0 ~]# pcs status|grep -B1 -i stop
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Comment 16 Yurii Prokulevych 2016-07-22 06:26:23 UTC
Created attachment 1182715 [details]

Comment 17 Yurii Prokulevych 2016-07-22 06:26:44 UTC
Overcloud deployed with next command:
openstack  overcloud deploy --libvirt-type qemu \
--ntp-server clock.redhat.com --templates \
--control-scale 3 --compute-scale 1 --ceph-storage-scale 1

Gnocchi* resources up and running.

Reconfigured gnocchi to use ceph backend(from default file) via stack update:
openstack  overcloud deploy --libvirt-type qemu \
--ntp-server clock.redhat.com --templates \
--control-scale 3 --compute-scale 1 --ceph-storage-scale 1 \
-e gnocchi_ceph.yaml

$ cat gnocchi_ceph.yaml 
    GnocchiBackend: 'rbd'

Gnocchi resources up and running.


Comment 19 errata-xmlrpc 2016-08-11 11:31:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.