Bug 1789710
| Summary: | "openstack image create" fails on multiple NFS store directories. | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ryo Hayakawa <rhayakaw> | |
| Component: | openstack-selinux | Assignee: | Julie Pichon <jpichon> | |
| Status: | CLOSED ERRATA | QA Contact: | Julie Pichon <jpichon> | |
| Severity: | high | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 10.0 (Newton) | CC: | jpichon, knoha, lhh, lvrabec, mfuruta, zcaplovi | |
| Target Milestone: | async | Keywords: | OtherQA, Triaged, ZStream | |
| Target Release: | 10.0 (Newton) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-selinux-0.8.18-2.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
When the filesystem_store_datadirs parameter in glance-api.conf is set to multiple NFS mount points, the "openstack image create" command was failing with "Permission denied" due to SELinux denials. A new SELinux rule to allow this was added.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1802366 (view as bug list) | Environment: | ||
| Last Closed: | 2020-04-02 10:25:38 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
Thank you for the detailed bug report. In the future it may also be useful to set SELinux to permissive and reproduce the error, then attach the audit.log to the bug to make sure the first AVC denial didn't hide other ones. From the description, it seems like there is confidence this one rule should resolve the issue by itself so I proposed a patch with it. Thanks! Ran unit test sanity check. $ rpm -qa openstack-selinux openstack-selinux-0.8.18-2.el7ost.noarch $ cat /usr/share/openstack-selinux/0.8.18/tests/bz1789710 type=AVC msg=audit(1578626829.434:88494): avc: denied { getattr } for pid=981968 comm="glance-api" name="/" dev="0:46" ino=289429476 scontext=system_u:system_r:glance_api_t:s0 tcontext=system_u:object_r:glance_var_lib_t:s0 tclass=filesystem permissive=0 $ /usr/share/openstack-selinux/0.8.18/tests/check_all Results: 798 total, 0 failed Overall result: PASS Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1303 |
Description of problem: When the filesystem_store_datadirs parameter in glance-api.conf is set to multiple NFS mount points, the "openstack image create" command fails with "Permission denied" in /var/log/glance/api.log and the audit.log shows "denied { getattr }" message (please see Actual results). As you can see /var/log/glance/api.log, "Permission denied" occurred at the os.statvfs(). Then, I ran "sealert -a /var/log/audit/audit.log" and got the following workaround: # ausearch -c 'glance-api' --raw | audit2allow -M my-glanceapi # semodule -i my-glanceapi.pp The above audit2allow generated my-glanceapi.te. The contents are as follows: module my-glanceapi 1.0; require { type glance_api_t; type glance_var_lib_t; class filesystem getattr; } #============= glance_api_t ============== allow glance_api_t glance_var_lib_t:filesystem getattr; This workaround can fix this problem, but I want a proper solution. I think this problem should be fixed in openstack-selinux package, what do you think? Our environment is as follows: == /etc/glance/glance-api.conf : (snip) [glance_store] : (snip) stores = glance.store.filesystem.Store,glance.store.http.Store default_store = filesystem filesystem_store_datadir = filesystem_store_datadirs = /var/lib/glance/images:1 filesystem_store_datadirs = /var/lib/glance/images2:2 : (snip) == # ls -lZd /var/lib/glance/images* drwxrwxrwx. root root system_u:object_r:glance_var_lib_t:s0 /var/lib/glance/images drwxrwxrwx. root root system_u:object_r:glance_var_lib_t:s0 /var/lib/glance/images2 == # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 16355356 0 16355356 0% /dev : (snip) 192.168.24.1:/exports/images 157274112 22625280 134648832 15% /var/lib/glance/images 192.168.24.1:/exports/images2 157274112 22625280 134648832 15% /var/lib/glance/images2 Version-Release number of selected component (if applicable): Red Hat OpenStack Platform release 10.0.14 (Newton) How reproducible: always Steps to Reproduce: 1. Mount NFS filesystem on the controller node to /var/lib/glance/images and /var/lib/glance/images2. 2. Configure the /etc/glance/glance-api.conf as follows: : (snip) [glance_store] : (snip) stores = glance.store.filesystem.Store,glance.store.http.Store default_store = filesystem filesystem_store_datadir = filesystem_store_datadirs = /var/lib/glance/images:1 filesystem_store_datadirs = /var/lib/glance/images2:2 : (snip) 3. Restart the glance-api service. 4. Run "openstack image create ..." command. Actual results: $ openstack image create --disk-format qcow2 --public --file ./rhel-server-7.8-beta-1-x86_64-kvm.qcow2 rhel-7.8-23 500 Internal Server Error The server has either erred or is incapable of performing the requested operation. (HTTP 500) === At the above time, /var/log/glance/api.log shows the following stack trace: 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data [req-82bb7b49-22fc-4ef3-ac64-bf912b83e1d2 612c35b39fc64ed3b37a308a735ac7be f362eab3275941b1a461879c7a85ed75 - default default] Failed to upload image data due to internal error 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data Traceback (most recent call last): 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data image.set_data(data, size) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data self.base.set_data(data, size) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data self.force_reraise() 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data self.repo.set_data(data, size) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 187, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data self.image.set_data(data, size=size) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data verifier=verifier) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 452, in add_to_backend 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data verifier) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 425, in store_add_to_backend 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data verifier=verifier) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/_drivers/filesystem.py", line 671, in add 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data datadir = self._find_best_datadir(image_size) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/_drivers/filesystem.py", line 629, in _find_best_datadir 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data free_space = self._get_capacity_info(datadir) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/_drivers/filesystem.py", line 606, in _get_capacity_info 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data stvfs_result = os.statvfs(mount_point) 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data OSError: [Errno 13] Permission denied: '/var/lib/glance/images2' 2020-01-10 11:31:34.664 852461 ERROR glance.api.v2.image_data === - /var/log/audit/audit.log shows the "denied { getattr }" type=AVC msg=audit(1578626829.434:88494): avc: denied { getattr } for pid=981968 comm="glance-api" name="/" dev="0:46" ino=289429476 scontext=system_u:system_r:glance_api_t:s0 tcontext=system_u:object_r:glance_var_lib_t:s0 tclass=filesystem permissive=0 type=SYSCALL msg=audit(1578626829.434:88494): arch=c000003e syscall=137 success=no exit=-13 a0=7f8ffa057d54 a1=7ffe71cf7400 a2=1 a3=7ffe71cf7100 items=0 ppid=981607 pid=981968 auid=4294967295 uid=161 gid=161 euid=161 suid=161 fsuid=161 egid=161 sgid=161 fsgid=161 tty=(none) ses=4294967295 comm="glance-api" exe="/usr/bin/python2.7" subj=system_u:system_r:glance_api_t:s0 key=(null) Expected results: image create will success. Additional info: