Bug 1353971 - 'Failed to expunge metric: PermissionError: Failed to remove metric' with ceph backend
Summary: 'Failed to expunge metric: PermissionError: Failed to remove metric' with ce...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Giulio Fidente
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks: 1356107
TreeView+ depends on / blocked
 
Reported: 2016-07-08 14:49 UTC by Yurii Prokulevych
Modified: 2016-08-18 04:00 UTC (History)
14 users (show)

Fixed In Version: openstack-puppet-modules-8.1.7-2.el7ost
Doc Type: Bug Fix
Doc Text:
The Ceph puppet module (puppet-ceph) did not update CephX keyrings when the caps parameter for a key were changed. This caused overcloud upgrades to fail, as caps to operate on the new 'metrics' pool were not added to the CephX keyring. With this update, puppet-ceph regenerates the virsh secret or updates its key if either 'rbd_keyring' or 'linvirt_rbd_secret_uuid' change. This ensures that the CephX keyring is updated as required when secrets or caps change.
Clone Of:
Environment:
Last Closed: 2016-08-15 07:19:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1606918 None None None 2016-07-27 13:06:31 UTC
OpenStack gerrit 232369 None None None 2016-07-29 14:12:26 UTC
OpenStack gerrit 350512 None None None 2016-08-03 10:48:45 UTC
Red Hat Product Errata RHEA-2016:1597 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 Release Candidate Advisory 2016-08-11 16:06:52 UTC

Description Yurii Prokulevych 2016-07-08 14:49:51 UTC
Description of problem:
Gnocchi manually configured to use ceph backend.
Package python-cradox-1.1.8-2.el7ost.x86_64 installed manually.

Attemtp to add measure to metric fails:
gnocchi  measures add -m '2016-07-07T14:45:01@1' 8d4c4821-e69d-497a-a9c5-db88f78aaba9
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
 (HTTP 500)

Excerpt from gnocchi/metricd.log

2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage [-] Unable to expunge metric f9633d99-eb1c-400a-b271-b4d82dd83386 from storage
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage Traceback (most recent call last):
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 204, in expunge_metrics
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     self.delete_metric(m, sync)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 250, in delete_metric
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     self._delete_metric_archives(metric)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 360, in _delete_metric_archives
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage     str("gnocchi_%s_%s" % (metric.id, aggregation)))
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage   File "cradox.pyx", line 2352, in cradox.Ioctx.remove_object (cradox.c:29337)
2016-07-08 14:46:42.292 15330 ERROR gnocchi.storage PermissionError: Failed to remove 'gnocchi_f9633d99-eb1c-400a-b271-b4d82dd83386_max'

Excerpt from httpd/gnocchi_wsgi_error.log
[Fri Jul 08 13:44:50.061870 2016] [:error] [pid 16248] [remote 192.0.2.20:4] mod_wsgi (pid=16248): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'.
[Fri Jul 08 13:44:50.061919 2016] [:error] [pid 16248] [remote 192.0.2.20:4] Traceback (most recent call last):
[Fri Jul 08 13:44:50.061936 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
[Fri Jul 08 13:44:50.062065 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return app(environ, start_response)
[Fri Jul 08 13:44:50.062075 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
[Fri Jul 08 13:44:50.062089 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     resp = self.call_func(req, *args, **self.kwargs)
[Fri Jul 08 13:44:50.062096 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
[Fri Jul 08 13:44:50.062106 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.func(req, *args, **kwargs)
[Fri Jul 08 13:44:50.062113 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
[Fri Jul 08 13:44:50.062159 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     response = req.get_response(self._app)
[Fri Jul 08 13:44:50.062168 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
[Fri Jul 08 13:44:50.062180 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     application, catch_exc_info=False)
[Fri Jul 08 13:44:50.062187 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
[Fri Jul 08 13:44:50.062210 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     app_iter = application(self.environ, start_response)
[Fri Jul 08 13:44:50.062218 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__
[Fri Jul 08 13:44:50.062228 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.application(environ, start_response)
[Fri Jul 08 13:44:50.062235 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__
[Fri Jul 08 13:44:50.062247 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.app(environ, start_response)
[Fri Jul 08 13:44:50.062254 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Fri Jul 08 13:44:50.062266 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return self.application(environ, start_response)
[Fri Jul 08 13:44:50.062272 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 829, in __call__
[Fri Jul 08 13:44:50.062282 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     return super(Pecan, self).__call__(environ, start_response)
[Fri Jul 08 13:44:50.062289 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
[Fri Jul 08 13:44:50.062308 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     self.invoke_controller(controller, args, kwargs, state)
[Fri Jul 08 13:44:50.062317 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 572, in invoke_controller
[Fri Jul 08 13:44:50.062328 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     result = controller(*args, **kwargs)
[Fri Jul 08 13:44:50.062335 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 550, in get_measures
[Fri Jul 08 13:44:50.062346 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     float(granularity) if granularity is not None else None)
[Fri Jul 08 13:44:50.062353 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 125, in get_measures
[Fri Jul 08 13:44:50.062363 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     from_timestamp, to_timestamp)]
[Fri Jul 08 13:44:50.062370 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 156, in _get_measures_timeserie
[Fri Jul 08 13:44:50.062380 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     metric, aggregation, granularity)
[Fri Jul 08 13:44:50.062386 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 304, in _list_split_keys_for_metric
[Fri Jul 08 13:44:50.062397 2016] [:error] [pid 16248] [remote 192.0.2.20:4]     xattrs = ioctx.get_xattrs("gnocchi_%s_container" % metric.id)
[Fri Jul 08 13:44:50.062403 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188)
[Fri Jul 08 13:44:50.062515 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 2463, in cradox.Ioctx.get_xattrs (cradox.c:30504)
[Fri Jul 08 13:44:50.062595 2016] [:error] [pid 16248] [remote 192.0.2.20:4]   File "cradox.pyx", line 1341, in cradox.XattrIterator.__cinit__ (cradox.c:16886)
[Fri Jul 08 13:44:50.062675 2016] [:error] [pid 16248] [remote 192.0.2.20:4] PermissionError: Failed to get rados xattrs for object 'gnocchi_4b77de95-893d-4e62-b99f-b2121697aa36_container'


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch
openstack-gnocchi-api-2.1.3-2.el7ost.noarch

python-cradox-1.1.8-2.el7ost.x86_64

Expected results:
-----------------
Gnocchi is handling measures for metrics without errors

Additional info:
Virtual environment: 3controllers + 2computes + 1ceph, after upgrade.

Comment 5 Pradeep Kilambi 2016-07-27 13:05:59 UTC
Turns out the issue is in puppet-ceph where if the secret is already there we dont update the acls and hence the key. See https://github.com/openstack/puppet-ceph/blob/master/manifests/key.pp#L145

gfidente will push a fix upstream.

Comment 6 Giulio Fidente 2016-08-02 22:59:28 UTC
The puppet-ceph submission is now merged, the puppet-nova submission is still failing on unrelated CI issues.

Comment 13 Yurii Prokulevych 2016-08-05 15:08:39 UTC
Gnocchi is working with ceph backend after the upgrade.

Package: openstack-puppet-modules-8.1.7-2.el7ost.noarch

Comment 15 errata-xmlrpc 2016-08-15 07:19:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html


Note You need to log in before you can comment on or make changes to this bug.