| Summary: | Gnocchi 'failed to write measure_1f..error code 36' with ceph backend | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yurii Prokulevych <yprokule> |
| Component: | openstack-gnocchi | Assignee: | Mehdi ABAAKOUK <mabaakou> |
| Status: | CLOSED NOTABUG | QA Contact: | Yurii Prokulevych <yprokule> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | apevec, jdanjou, jschluet, lhh, pkilambi |
| Target Milestone: | ga | ||
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-12 07:48:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
In this case, the ceph setup was not working, I have proposed to set in ceph.conf "osd max object name len = 64" to workaround the issue. But the workaround provokes this error. This error occurs when the object name is too long for Ceph. Gnocchi object name looks like gnocchi_<uuid>_<timestamp>_<aggregation>_<granularity> and can each the limit of 64. So I suggest to set "osd max object name len = 1024" instead as workaround, that should work for ceph on ext4 filesystem too and we should not run in this issue. But the real issue is why ceph doesn't start anymore by default with ospd on OSP10 Looks like the maximun for ext4 is 256 The ceph issue: https://bugzilla.redhat.com/show_bug.cgi?id=1372804 |
Description of problem: ----------------------- Attempts to store measure for metric fails [Fri Sep 09 13:00:46.194965 2016] [:error] [pid 19000] [remote 192.0.2.6:24] mod_wsgi (pid=19000): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'. [Fri Sep 09 13:00:46.195017 2016] [:error] [pid 19000] [remote 192.0.2.6:24] Traceback (most recent call last): [Fri Sep 09 13:00:46.195034 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__ [Fri Sep 09 13:00:46.195059 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return app(environ, start_response) [Fri Sep 09 13:00:46.195069 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ [Fri Sep 09 13:00:46.195084 2016] [:error] [pid 19000] [remote 192.0.2.6:24] resp = self.call_func(req, *args, **self.kwargs) [Fri Sep 09 13:00:46.195093 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func [Fri Sep 09 13:00:46.195106 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return self.func(req, *args, **kwargs) [Fri Sep 09 13:00:46.195115 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ [Fri Sep 09 13:00:46.195129 2016] [:error] [pid 19000] [remote 192.0.2.6:24] response = req.get_response(self._app) [Fri Sep 09 13:00:46.195138 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send [Fri Sep 09 13:00:46.195151 2016] [:error] [pid 19000] [remote 192.0.2.6:24] application, catch_exc_info=False) [Fri Sep 09 13:00:46.195160 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application [Fri Sep 09 13:00:46.195172 2016] [:error] [pid 19000] [remote 192.0.2.6:24] app_iter = application(self.environ, start_response) [Fri Sep 09 13:00:46.195181 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ [Fri Sep 09 13:00:46.195192 2016] [:error] [pid 19000] [remote 192.0.2.6:24] resp = self.call_func(req, *args, **self.kwargs) [Fri Sep 09 13:00:46.195200 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func [Fri Sep 09 13:00:46.195212 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return self.func(req, *args, **kwargs) [Fri Sep 09 13:00:46.195220 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 126, in __call__ [Fri Sep 09 13:00:46.195233 2016] [:error] [pid 19000] [remote 192.0.2.6:24] response = req.get_response(self.application) [Fri Sep 09 13:00:46.195241 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send [Fri Sep 09 13:00:46.195253 2016] [:error] [pid 19000] [remote 192.0.2.6:24] application, catch_exc_info=False) [Fri Sep 09 13:00:46.195261 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application [Fri Sep 09 13:00:46.195272 2016] [:error] [pid 19000] [remote 192.0.2.6:24] app_iter = application(self.environ, start_response) [Fri Sep 09 13:00:46.195293 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__ [Fri Sep 09 13:00:46.195307 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return self.application(environ, start_response) [Fri Sep 09 13:00:46.195316 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__ [Fri Sep 09 13:00:46.195330 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return self.app(environ, start_response) [Fri Sep 09 13:00:46.195338 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__ [Fri Sep 09 13:00:46.195351 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return self.application(environ, start_response) [Fri Sep 09 13:00:46.195359 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 835, in __call__ [Fri Sep 09 13:00:46.195371 2016] [:error] [pid 19000] [remote 192.0.2.6:24] return super(Pecan, self).__call__(environ, start_response) [Fri Sep 09 13:00:46.195379 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__ [Fri Sep 09 13:00:46.195390 2016] [:error] [pid 19000] [remote 192.0.2.6:24] self.invoke_controller(controller, args, kwargs, state) [Fri Sep 09 13:00:46.195399 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 569, in invoke_controller [Fri Sep 09 13:00:46.195410 2016] [:error] [pid 19000] [remote 192.0.2.6:24] result = controller(*args, **kwargs) [Fri Sep 09 13:00:46.195418 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 1204, in post [Fri Sep 09 13:00:46.195430 2016] [:error] [pid 19000] [remote 192.0.2.6:24] pecan.request.storage.add_measures(metric, measures) [Fri Sep 09 13:00:46.195438 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 204, in add_measures [Fri Sep 09 13:00:46.195451 2016] [:error] [pid 19000] [remote 192.0.2.6:24] list(map(tuple, measures)))) [Fri Sep 09 13:00:46.195459 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 151, in _store_new_measures [Fri Sep 09 13:00:46.195471 2016] [:error] [pid 19000] [remote 192.0.2.6:24] self.ioctx.write_full(name, data) [Fri Sep 09 13:00:46.195479 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "cradox.pyx", line 413, in cradox.requires.wrapper.validate_func (cradox.c:4188) [Fri Sep 09 13:00:46.195608 2016] [:error] [pid 19000] [remote 192.0.2.6:24] File "cradox.pyx", line 2142, in cradox.Ioctx.write_full (cradox.c:27304) [Fri Sep 09 13:00:46.195698 2016] [:error] [pid 19000] [remote 192.0.2.6:24] Error: Ioctx.write_full(metrics): failed to write measure_1f600e5f-45ef-4537-a024-1bd48636667a_073edd26-90ba-4b47-b29b-6d09b4ee337d_20160909_13:00:46: error code 36 Version-Release number of selected component (if applicable): ------------------------------------------------------------- openstack-gnocchi-indexer-sqlalchemy-2.2.1-0.20160821103314.7a40269.el7ost.noarch openstack-gnocchi-metricd-2.2.1-0.20160821103314.7a40269.el7ost.noarch puppet-gnocchi-9.1.0-0.20160815142605.7aa0c60.el7ost.noarch python-gnocchi-2.2.1-0.20160821103314.7a40269.el7ost.noarch openstack-gnocchi-api-2.2.1-0.20160821103314.7a40269.el7ost.noarch python-gnocchiclient-2.5.1-0.20160823092719.a1ec8aa.el7ost.noarch openstack-gnocchi-carbonara-2.2.1-0.20160821103314.7a40269.el7ost.noarch openstack-gnocchi-common-2.2.1-0.20160821103314.7a40269.el7ost.noarch openstack-gnocchi-statsd-2.2.1-0.20160821103314.7a40269.el7ost.noarch python-cradox-1.1.8-2.el7ost.x86_64 Steps to Reproduce: 1. Deploy overcloud with gnocchi using ceph as a backend: openstack overcloud deploy \ --ntp-server clock.redhat.com \ --templates \ --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 \ --libvirt-type qemu -e puppet-pacemaker.yaml \ -e storage-environment.yaml 2. Gnocchi now is configured as a default backend so wait for next polling cycle to happen or try to add measure manually: gnocchi measures add -m '2016-09-09T13:57:05@3' $METRIC_ID