Bug 1640021
| Summary: | Permission denied to /var/lib/gnocchi/ folders when working with GnocchiBackend: file | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sasha Smolyak <ssmolyak> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Julien Danjou <jdanjou> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 14.0 (Rocky) | CC: | apevec, jjoyce, jschluet, lhh, mariel, mburns, mmagr, mrunge, sclewis, ssmolyak | ||||
| Target Milestone: | beta | Keywords: | Triaged | ||||
| Target Release: | 14.0 (Rocky) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-9.0.1-0.20181013060898.el7ost | Doc Type: | Known Issue | ||||
| Doc Text: |
Deploying the overcloud with a Gnocchi file backend might fail due to access permissions to the /var/lib/gnocchi/ directory.
Workaround: Before you deploy the overcloud, set the permissions to the directory in the openstack/tripleo-heat-templates/docker/services/gnocchi-api.yaml file as follows:
- path:
list_join:
- "/"
- - {get_param: GnocchiFileBasePath}
- "tmp"
owner: gnocchi:gnocchi
perm: '0600'
recurse: true
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-01-11 11:54:06 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1653577, 1655886 | ||||||
| Bug Blocks: | 1511597, 1665165 | ||||||
| Attachments: |
|
||||||
Workaround:
Add manually before overcloud deploy to openstack/tripleo-heat-templates/docker/services/gnocchi-api.yaml:
- path:
list_join:
- "/"
- - {get_param: GnocchiFileBasePath}
- "tmp"
owner: gnocchi:gnocchi
perm: '0600'
recurse: true
https://review.openstack.org/#/c/611827/4/docker/services/gnocchi-api.yaml
patch to master was merged, adding link to rocky backport. When deploying with GnocchiBackend: file, undercloud installation gets stuck. Had to stop it manually after 3 hours. Deployment logs attached Created attachment 1508439 [details]
Artifacts of failed deployment
Undercloud installation was manually stopped after 3 hours
[heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi total 4 drwxr-xr-x. 39 gnocchi gnocchi 4096 Nov 29 08:58 . drwxr-xr-x. 1 root root 19 Nov 28 15:21 .. drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 09 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:42 0e drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:53 13 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:46 21 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 22 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 26 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 28 drwxr-x---. 4 gnocchi gnocchi 26 Nov 29 08:48 39 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 46 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 4b drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 4c drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 52 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 5c drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 61 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:46 69 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:53 7b drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 86 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 8a drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 9e drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 a3 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 a7 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 a8 drwxr-x---. 5 gnocchi gnocchi 36 Nov 29 09:03 a9 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 ac drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:45 b0 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:33 b2 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 b5 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:49 bc drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:24 c0 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 ce drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 cf drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:38 d2 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:58 d5 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 e3 drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 07:28 ea drwxr-x---. 3 gnocchi gnocchi 16 Nov 29 08:48 f1 drw-------. 2 gnocchi gnocchi 36 Nov 28 15:19 tmp [heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi/tmp/ total 8 drw-------. 2 gnocchi gnocchi 36 Nov 28 15:19 . drwxr-xr-x. 39 gnocchi gnocchi 4096 Nov 29 08:58 .. -rw-------. 1 gnocchi gnocchi 17 Nov 28 15:19 gnocchi-storage-config The permissions look wrong for tmp folder, +x is missing Failed https://review.openstack.org/#/c/618089/ finally merged, it's already downstream. Fixed [heat-admin@controller-0 ~]$ sudo docker exec -ti gnocchi_api ls -la /var/lib/gnocchi/ total 4 drwxr-xr-x. 33 gnocchi gnocchi 4096 Dec 9 11:55 . drwxr-xr-x. 1 root root 19 Dec 9 10:04 .. drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 03 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 10 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 1a drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:43 2b drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 37 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 39 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 3e drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 40 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 43 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 46 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 47 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 49 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 4f drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 50 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 56 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:55 58 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 69 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 80 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 8a drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 9d drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 a0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 af drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 d0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:36 d8 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 de drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 e0 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 e9 drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 ef drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:42 fd drwxr-x---. 3 gnocchi gnocchi 16 Dec 9 11:37 fe drwxr-x---. 2 gnocchi gnocchi 36 Dec 9 15:42 tmp Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Description of problem: There are no gnocchi measures of any metric in the system. The folders of measures on /var/log/gnocchi are empty, the log shows that "Permission is denied": 2018-10-17 07:15:13,796 [277055] ERROR gnocchi.chef: Error processing new measures Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/gnocchi/chef.py", line 165, in process_new_measures_for_sack for metric in metrics File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 727, in add_measures_to_metrics self._update_metric_splits(splits_to_update) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 460, in _update_metric_splits return self._store_metric_splits(metrics_splits_to_store) File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 236, in _store_metric_splits in six.iteritems(metrics_keys_aggregations_data_offset) File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 316, in parallel_map return list(executor.map(lambda args: fn(*args), list_of_args)) File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator yield future.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 316, in <lambda> return list(executor.map(lambda args: fn(*args), list_of_args)) File "/usr/lib/python2.7/site-packages/gnocchi/storage/file.py", line 228, in _store_metric_splits_unbatched data) File "/usr/lib/python2.7/site-packages/gnocchi/storage/file.py", line 114, in _atomic_file_store delete=False) File "/usr/lib64/python2.7/tempfile.py", line 458, in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner fd = _os.open(file, flags, 0600) OSError: [Errno 13] Permission denied: '/var/lib/gnocchi/tmp/gnocchicxNwtv' Version-Release number of selected component (if applicable): puppet-gnocchi-13.3.1-0.20180901135430.edb2394.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy telemetry with file backend: custom templates: parameter defaults: GnocchiBackend: file 2. Deploy an instance, try to observe it's metrics 3. Go to /var/lib/gnocchi/, observe many folders for the metrics and the tmp folder. [heat-admin@controller-0 ~]$ ls -la /var/lib/gnocchi/tmp/ total 4 drwxr-xr-x. 2 root root 36 Oct 16 11:20 . drwxr-xr-x. 29 42416 42416 277 Oct 16 17:01 .. All the folders are under user 42416, and the tmp one is under root Actual results: No measures in the folders, permission to tmp denied, from gnocchi-metricd.log: 2018-10-17 07:15:03,255 [277052] INFO gnocchi.cli.metricd: 4777 measurements bundles across 29 metrics wait to be processed. Expected results: Measures in the folders, no measurements to be processed Additional info: python-gnocchiclient and python-pankoclient are not in the package and still need to be installed manually