| Summary: | 'Failed to expunge metric: PermissionError: Failed to remove metric' with ceph backend | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yurii Prokulevych <yprokule> |
| Component: | openstack-puppet-modules | Assignee: | Giulio Fidente <gfidente> |
| Status: | CLOSED ERRATA | QA Contact: | Yurii Prokulevych <yprokule> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.0 (Mitaka) | CC: | apevec, ddomingo, eglynn, fbaudin, gfidente, jjoyce, johfulto, jschluet, lbezdick, lhh, mburns, pkilambi, srevivo, tvignaud |
| Target Milestone: | ga | Keywords: | Reopened |
| Target Release: | 9.0 (Mitaka) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-puppet-modules-8.1.7-2.el7ost | Doc Type: | Bug Fix |
| Doc Text: |
The Ceph puppet module (puppet-ceph) did not update CephX keyrings when the caps parameter for a key were changed. This caused overcloud upgrades to fail, as caps to operate on the new 'metrics' pool were not added to the CephX keyring.
With this update, puppet-ceph regenerates the virsh secret or updates its key if either 'rbd_keyring' or 'linvirt_rbd_secret_uuid' change. This ensures that the CephX keyring is updated as required when secrets or caps change.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-08-15 07:19:01 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | |||
| Bug Blocks: | 1356107 | ||
Turns out the issue is in puppet-ceph where if the secret is already there we dont update the acls and hence the key. See https://github.com/openstack/puppet-ceph/blob/master/manifests/key.pp#L145 gfidente will push a fix upstream. The puppet-ceph submission is now merged, the puppet-nova submission is still failing on unrelated CI issues. Gnocchi is working with ceph backend after the upgrade. Package: openstack-puppet-modules-8.1.7-2.el7ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1597.html |
Description of problem: Gnocchi manually configured to use ceph backend. Package python-cradox-1.1.8-2.el7ost.x86_64 installed manually. Attemtp to add measure to metric fails: gnocchi measures add -m '2016-07-07T14:45:01@1' 8d4c4821-e69d-497a-a9c5-db88f78aaba9 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> (HTTP 500) Excerpt from gnocchi/metricd.log 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage [-] Unable to expunge metric f9633d99-eb1c-400a-b271-b4d82dd83386 from storage 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage Traceback (most recent call last): 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 204, in expunge_metrics 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage self.delete_metric(m, sync) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 250, in delete_metric 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage self._delete_metric_archives(metric) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 360, in _delete_metric_archives 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage str("gnocchi_%s_%s" % (metric.id, aggregation))) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage File "cradox.pyx", line 2352, in cradox.Ioctx.remove_object (cradox.c:29337) 2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage PermissionError: Failed to remove 'gnocchi_f9633d99-eb1c-400a-b271-b4d82dd83386_max' Excerpt from httpd/gnocchi_wsgi_error.log [Fri Jul 08 13:44:50.061870 2016] [:error] [pid 16248] [remote 192.0.2.20:4] mod_wsgi (pid=16248): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'. [Fri Jul 08 13:44:50.061919 2016] [:error] [pid 16248] [remote 192.0.2.20:4] Traceback (most recent call last): [Fri Jul 08 13:44:50.061936 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__ [Fri Jul 08 13:44:50.062065 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return app(environ, start_response) [Fri Jul 08 13:44:50.062075 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ [Fri Jul 08 13:44:50.062089 2016] [:error] [pid 16248] [remote 192.0.2.20:4] resp = self.call_func(req, *args, **self.kwargs) [Fri Jul 08 13:44:50.062096 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func [Fri Jul 08 13:44:50.062106 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.func(req, *args, **kwargs) [Fri Jul 08 13:44:50.062113 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__ [Fri Jul 08 13:44:50.062159 2016] [:error] [pid 16248] [remote 192.0.2.20:4] response = req.get_response(self._app) [Fri Jul 08 13:44:50.062168 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send [Fri Jul 08 13:44:50.062180 2016] [:error] [pid 16248] [remote 192.0.2.20:4] application, catch_exc_info=False) [Fri Jul 08 13:44:50.062187 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application [Fri Jul 08 13:44:50.062210 2016] [:error] [pid 16248] [remote 192.0.2.20:4] app_iter = application(self.environ, start_response) [Fri Jul 08 13:44:50.062218 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__ [Fri Jul 08 13:44:50.062228 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.application(environ, start_response) [Fri Jul 08 13:44:50.062235 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__ [Fri Jul 08 13:44:50.062247 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.app(environ, start_response) [Fri Jul 08 13:44:50.062254 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__ [Fri Jul 08 13:44:50.062266 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return self.application(environ, start_response) [Fri Jul 08 13:44:50.062272 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 829, in __call__ [Fri Jul 08 13:44:50.062282 2016] [:error] [pid 16248] [remote 192.0.2.20:4] return super(Pecan, self).__call__(environ, start_response) [Fri Jul 08 13:44:50.062289 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__ [Fri Jul 08 13:44:50.062308 2016] [:error] [pid 16248] [remote 192.0.2.20:4] self.invoke_controller(controller, args, kwargs, state) [Fri Jul 08 13:44:50.062317 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 572, in invoke_controller [Fri Jul 08 13:44:50.062328 2016] [:error] [pid 16248] [remote 192.0.2.20:4] result = controller(*args, **kwargs) [Fri Jul 08 13:44:50.062335 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 550, in get_measures [Fri Jul 08 13:44:50.062346 2016] [:error] [pid 16248] [remote 192.0.2.20:4] float(granularity) if granularity is not None else None) [Fri Jul 08 13:44:50.062353 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 125, in get_measures [Fri Jul 08 13:44:50.062363 2016] [:error] [pid 16248] [remote 192.0.2.20:4] from_timestamp, to_timestamp)] [Fri Jul 08 13:44:50.062370 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 156, in _get_measures_timeserie [Fri Jul 08 13:44:50.062380 2016] [:error] [pid 16248] [remote 192.0.2.20:4] metric, aggregation, granularity) [Fri Jul 08 13:44:50.062386 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 304, in _list_split_keys_for_metric [Fri Jul 08 13:44:50.062397 2016] [:error] [pid 16248] [remote 192.0.2.20:4] xattrs = ioctx.get_xattrs("gnocchi_%s_container" % metric.id) [Fri Jul 08 13:44:50.062403 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188) [Fri Jul 08 13:44:50.062515 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 2463, in cradox.Ioctx.get_xattrs (cradox.c:30504) [Fri Jul 08 13:44:50.062595 2016] [:error] [pid 16248] [remote 192.0.2.20:4] File "cradox.pyx", line 1341, in cradox.XattrIterator.__cinit__ (cradox.c:16886) [Fri Jul 08 13:44:50.062675 2016] [:error] [pid 16248] [remote 192.0.2.20:4] PermissionError: Failed to get rados xattrs for object 'gnocchi_4b77de95-893d-4e62-b99f-b2121697aa36_container' Version-Release number of selected component (if applicable): ------------------------------------------------------------- openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch python-gnocchi-2.1.3-2.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch python-gnocchiclient-2.2.0-1.el7ost.noarch openstack-gnocchi-common-2.1.3-2.el7ost.noarch openstack-gnocchi-api-2.1.3-2.el7ost.noarch python-cradox-1.1.8-2.el7ost.x86_64 Expected results: ----------------- Gnocchi is handling measures for metrics without errors Additional info: Virtual environment: 3controllers + 2computes + 1ceph, after upgrade.