The Ceph puppet module (puppet-ceph) did not update CephX keyrings when the caps parameter for a key were changed. This caused overcloud upgrades to fail, as caps to operate on the new 'metrics' pool were not added to the CephX keyring.
With this update, puppet-ceph regenerates the virsh secret or updates its key if either 'rbd_keyring' or 'linvirt_rbd_secret_uuid' change. This ensures that the CephX keyring is updated as required when secrets or caps change.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHEA-2016-1597.html
Description of problem: Gnocchi manually configured to use ceph backend. Package python-cradox-1.1.8-2.el7ost.x86_64 installed manually. Attemtp to add measure to metric fails: gnocchi measures add -m '2016-07-07T14:45:01@1' 8d4c4821-e69d-497a-a9c5-db88f78aaba9 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500) Excerpt from gnocchi/metricd.log 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage [-] Unable to expunge metric f9633d99-eb1c-400a-b271-b4d82dd83386 from storage 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage Traceback (most recent call last): 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 204, in expunge_metrics 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage self.delete_metric(m, sync) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 250, in delete_metric 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage self._delete_metric_archives(metric) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 360, in _delete_metric_archives 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage str("gnocchi_%s_%s" % (metric.id, aggregation))) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "cradox.pyx", line 2352, in cradox.Ioctx.remove_object (cradox.c:29337) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage PermissionError: Failed to remove 'gnocchi_f9633d99-eb1c-400a-b271-b4d82dd83386_max' Excerpt from httpd/gnocchi_wsgi_error.log [Fri Jul 08 13:44:50.061870 2016] [:error] [pid 16248] [remote 192.0.2.20:4] mod_wsgi (pid=16248): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'. [Fri Jul 08 13:44:50.061919 2016] [:error] [pid 16248] [remote 192.0.2.20:4] Traceback (most recent call last): [Fri Jul 08 13:44:50.061936 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__ [Fri Jul 08 13:44:50.062065 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return app(environ, start_response) [Fri Jul 08 13:44:50.062075 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ [Fri Jul 08 13:44:50.062089 2016] [:error] [pid 16248] [remote 192.0.2.20:4] resp = self.call_func(req, *args, **self.kwargs) [Fri Jul 08 13:44:50.062096 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func [Fri Jul 08 13:44:50.062106 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.func(req, *args, **kwargs) [Fri Jul 08 13:44:50.062113 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__ [Fri Jul 08 13:44:50.062159 2016] [:error] [pid 16248] [remote 192.0.2.20:4] response = req.get_response(self._app) [Fri Jul 08 13:44:50.062168 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send [Fri Jul 08 13:44:50.062180 2016] [:error] [pid 16248] [remote 192.0.2.20:4] application, catch_exc_info=False) [Fri Jul 08 13:44:50.062187 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application [Fri Jul 08 13:44:50.062210 2016] [:error] [pid 16248] [remote 192.0.2.20:4] app_iter = application(self.environ, start_response) [Fri Jul 08 13:44:50.062218 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__ [Fri Jul 08 13:44:50.062228 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.application(environ, start_response) [Fri Jul 08 13:44:50.062235 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__ [Fri Jul 08 13:44:50.062247 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.app(environ, start_response) [Fri Jul 08 13:44:50.062254 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__ [Fri Jul 08 13:44:50.062266 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.application(environ, start_response) [Fri Jul 08 13:44:50.062272 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 829, in __call__ [Fri Jul 08 13:44:50.062282 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return super(Pecan, self).__call__(environ, start_response) [Fri Jul 08 13:44:50.062289 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__ [Fri Jul 08 13:44:50.062308 2016] [:error] [pid 16248] [remote 192.0.2.20:4] self.invoke_controller(controller, args, kwargs, state) [Fri Jul 08 13:44:50.062317 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 572, in invoke_controller [Fri Jul 08 13:44:50.062328 2016] [:error] [pid 16248] [remote 192.0.2.20:4] result = controller(*args, **kwargs) [Fri Jul 08 13:44:50.062335 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 550, in get_measures [Fri Jul 08 13:44:50.062346 2016] [:error] [pid 16248] [remote 192.0.2.20:4] float(granularity) if granularity is not None else None) [Fri Jul 08 13:44:50.062353 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 125, in get_measures [Fri Jul 08 13:44:50.062363 2016] [:error] [pid 16248] [remote 192.0.2.20:4] from_timestamp, to_timestamp)] [Fri Jul 08 13:44:50.062370 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 156, in _get_measures_timeserie [Fri Jul 08 13:44:50.062380 2016] [:error] [pid 16248] [remote 192.0.2.20:4] metric, aggregation, granularity) [Fri Jul 08 13:44:50.062386 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 304, in _list_split_keys_for_metric [Fri Jul 08 13:44:50.062397 2016] [:error] [pid 16248] [remote 192.0.2.20:4] xattrs = ioctx.get_xattrs("gnocchi_%s_container" % metric.id) [Fri Jul 08 13:44:50.062403 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188) [Fri Jul 08 13:44:50.062515 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 2463, in cradox.Ioctx.get_xattrs (cradox.c:30504) [Fri Jul 08 13:44:50.062595 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 1341, in cradox.XattrIterator.__cinit__ (cradox.c:16886) [Fri Jul 08 13:44:50.062675 2016] [:error] [pid 16248] [remote 192.0.2.20:4] PermissionError: Failed to get rados xattrs for object 'gnocchi_4b77de95-893d-4e62-b99f-b2121697aa36_container' Version-Release number of selected component (if applicable): ------------------------------------------------------------- openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch python-gnocchi-2.1.3-2.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch python-gnocchiclient-2.2.0-1.el7ost.noarch openstack-gnocchi-common-2.1.3-2.el7ost.noarch openstack-gnocchi-api-2.1.3-2.el7ost.noarch python-cradox-1.1.8-2.el7ost.x86_64 Expected results: ----------------- Gnocchi is handling measures for metrics without errors Additional info: Virtual environment: 3controllers + 2computes + 1ceph, after upgrade.