Description of problem: There are no gnocchi measures of any metric in the system. The folders of measures on /var/log/gnocchi are empty, the log shows that "Permission is denied": 2018-10-17 07:15:13,796 [277055] ERROR gnocchi.chef: Error processing new measures Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gnocchi/chef.py", line 165, in process_new_measures_for_sack for metric in metrics File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 727, in add_measures_to_metrics self._update_metric_splits(splits_to_update) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 460, in _update_metric_splits return self._store_metric_splits(metrics_splits_to_store) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 236, in _store_metric_splits in six.iteritems(metrics_keys_aggregations_data_offset) File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 316, in parallel_map return list(executor.map(lambda args: fn(*args), list_of_args)) File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator yield future.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 316, in <lambda> return list(executor.map(lambda args: fn(*args), list_of_args)) File "/usr/lib/python2.7/site-packages/gnocchi/storage/file.py", line 228, in _store_metric_splits_unbatched data) File "/usr/lib/python2.7/site-packages/gnocchi/storage/file.py", line 114, in _atomic_file_store delete=False) File "/usr/lib64/python2.7/tempfile.py", line 458, in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 13] Permission denied: '/var/lib/gnocchi/tmp/gnocchicxNwtv' Version-Release number of selected component (if applicable): puppet-gnocchi-13.3.1-0.20180901135430.edb2394.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy telemetry with file backend: custom templates: parameter defaults: GnocchiBackend: file 2. Deploy an instance, try to observe it's metrics 3. Go to /var/lib/gnocchi/, observe many folders for the metrics and the tmp folder. [heat-admin@controller-0 ~]$ ls -la /var/lib/gnocchi/tmp/ total 4 drwxr-xr-x. 2 root root 36 Oct 16 11:20 . drwxr-xr-x. 29 42416 42416 277 Oct 16 17:01 .. All the folders are under user 42416, and the tmp one is under root Actual results: No measures in the folders, permission to tmp denied, from gnocchi-metricd.log: 2018-10-17 07:15:03,255 [277052] INFO gnocchi.cli.metricd: 4777 measurements bundles across 29 metrics wait to be processed. Expected results: Measures in the folders, no measurements to be processed Additional info: python-gnocchiclient and python-pankoclient are not in the package and still need to be installed manually
Workaround: Add manually before overcloud deploy to openstack/tripleo-heat-templates/docker/services/gnocchi-api.yaml: - path: list_join: - "/" - - {get_param: GnocchiFileBasePath} - "tmp" owner: gnocchi:gnocchi perm: '0600' recurse: true https://review.openstack.org/#/c/611827/4/docker/services/gnocchi-api.yaml
patch to master was merged, adding link to rocky backport.
When deploying with GnocchiBackend: file, undercloud installation gets stuck. Had to stop it manually after 3 hours. Deployment logs attached
Created attachment 1508439 [details] Artifacts of failed deployment Undercloud installation was manually stopped after 3 hours
[heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi total 4 drwxr-xr-x. 39 gnocchi gnocchi 4096 Nov 29 08:58 . drwxr-xr-x. 1 root root 19 Nov 28 15:21 .. drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 09 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:42 0e drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:53 13 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:46 21 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 22 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 26 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 28 drwxr-x---. 4 gnocchi gnocchi 26 Nov 29 08:48 39 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 46 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 4b drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 4c drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 52 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 5c drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 61 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:46 69 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:53 7b drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 86 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 8a drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 9e drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 a3 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 a7 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 a8 drwxr-x---. 5 gnocchi gnocchi 36 Nov 29 09:03 a9 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 ac drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:45 b0 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 b2 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 b5 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:49 bc drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 c0 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 ce drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 cf drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 d2 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 d5 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 e3 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 ea drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 f1 drw-------. 2 gnocchi gnocchi 36 Nov 28 15:19 tmp [heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi/tmp/ total 8 drw-------. 2 gnocchi gnocchi 36 Nov 28 15:19 . drwxr-xr-x. 39 gnocchi gnocchi 4096 Nov 29 08:58 .. -rw-------. 1 gnocchi gnocchi 17 Nov 28 15:19 gnocchi-storage-config The permissions look wrong for tmp folder, +x is missing Failed
https://review.openstack.org/#/c/618089/ finally merged, it's already downstream.
Fixed [heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi/ total 4 drwxr-xr-x. 33 gnocchi gnocchi 4096 Dec 9 11:55 . drwxr-xr-x. 1 root root 19 Dec 9 10:04 .. drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 03 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 10 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 1a drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:43 2b drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 37 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 39 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 3e drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 40 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 43 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 46 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 47 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 49 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 4f drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 50 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 56 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:55 58 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 69 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 80 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 8a drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 9d drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 a0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 af drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 d0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:36 d8 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 de drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 e0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 e9 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 ef drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 fd drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 fe drwxr-x---. 2 gnocchi gnocchi 36 Dec 9 15:42 tmp
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045