Bug 1402561 - Unable to launch instance when /var/lib/nova/instances is mounted on an NFS share
Summary: Unable to launch instance when /var/lib/nova/instances is mounted on an NFS s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.3
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: pre-dev-freeze
: 7.3
Assignee: Lukas Vrabec
QA Contact: Milos Malik
URL:
Whiteboard:
: 1442070 (view as bug list)
Depends On:
Blocks: 1442070 1469428
TreeView+ depends on / blocked
 
Reported: 2016-12-07 20:21 UTC by Marius Cornea
Modified: 2020-12-21 19:39 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Previously, an SELinux rule for the libvirtd virtualization server was missing. Consequently, when running SELinux in enforcing mode, starting new OpenStack instances failed if the /var/lib/nova/instances/ directory was an NFS share. The policy rule has been added, and starting instances from an NFS share now works as expected.
Clone Of:
: 1442070 1469428 (view as bug list)
Environment:
Last Closed: 2017-08-01 15:17:42 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1861 0 normal SHIPPED_LIVE selinux-policy bug fix update 2017-08-01 17:50:24 UTC

Description Marius Cornea 2016-12-07 20:21:07 UTC
Description of problem:
SELinux is blocking launching instance if /var/lib/nova/instances is mounted on an NFS share:

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy 1 x controller, 1 x compute with OSPd

2. On compute node mount /var/lib/libvirt/images on an NFS share. Add the following to fstab and mount -a:
10.0.0.254:/srv/nfs/nova /var/lib/nova/instances nfs4 defaults 0 0

mount | grep nova
10.0.0.254:/srv/nfs/nova on /var/lib/nova/instances type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.148,local_lock=none,addr=10.0.0.254)

3. setsebool -P virt_use_nfs 1

4. Launch an instance

Actual results:
/var/log/nova/nova-compute.log:
2016-12-07 20:17:43.999 43920 ERROR nova.compute.manager [req-a4292bd5-b3f4-43ed-a1f9-0d3083bf00cd 0450136018584892ae4b948004bf4bb9 ea88f38a6ddb401f9d19881ee19f2b07 - - -] [instance: 4c4ced4b-ac42-40b4-87c9-e9d1beea6327] Instance failed to spawn
2016-12-07 20:17:43.999 43920 ERROR nova.compute.manager [instance: 4c4ced4b-ac42-40b4-87c9-e9d1beea6327] libvirtError: Unable to open file: /var/lib/nova/instances/4c4ced4b-ac42-40b4-87c9-e9d1beea6327/console.log: Permission denied

/var/log/audit/audit.log:

type=AVC msg=audit(1481141863.755:696): avc:  denied  { search } for  pid=50197 comm="virtlogd" name="nova" dev="0:45" ino=1342284999 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir
type=AVC msg=audit(1481141863.977:711): avc:  denied  { search } for  pid=50197 comm="virtlogd" name="nova" dev="0:45" ino=1342284999 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir
type=AVC msg=audit(1481141871.660:748): avc:  denied  { search } for  pid=50197 comm="virtlogd" name="nova" dev="0:45" ino=1342284999 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir
type=AVC msg=audit(1481141873.490:781): avc:  denied  { search } for  pid=50197 comm="virtlogd" name="nova" dev="0:45" ino=1342284999 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir
type=AVC msg=audit(1481141882.533:814): avc:  denied  { search } for  pid=50197 comm="virtlogd" name="nova" dev="0:45" ino=1342284999 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir

ls -lZd /var/lib/nova/instances
drwxrwxrwx. nova nova system_u:object_r:nfs_t:s0       /var/lib/nova/instances

Expected results:
Launching an instance is successful.

Additional info:

I'm following the instruction at https://access.redhat.com/solutions/881063

Comment 1 Martin Schuppert 2016-12-08 07:53:36 UTC
Seems you'd need to relabel the instances /var/lib/nova/instances to 'nova_var_lib_t'. Check BZ 1396518#c6

Comment 2 Marius Cornea 2016-12-08 09:00:31 UTC
(In reply to Martin Schuppert from comment #1)
> Seems you'd need to relabel the instances /var/lib/nova/instances to
> 'nova_var_lib_t'. Check BZ 1396518#c6

Thanks, Martin. I mounted the NFS share with this context but now a different type of AVCs show up. I wonder if there's something that I'm missing here:

fstab:
10.0.0.254:/srv/nfs/nova  /var/lib/nova/instances nfs4 defaults,context=system_u:object_r:nova_var_lib_t:s0 0 0

ls -lZd /var/lib/nova/instances/
drwxrwxrwx. nova nova system_u:object_r:nova_var_lib_t:s0 /var/lib/nova/instances/

/var/log/audit/audit.log:
type=AVC msg=audit(1481187048.386:2338): avc:  denied  { getattr } for  pid=145245 comm="qemu-kvm" path="/var/lib/nova/instances/51f55bd4-abf4-4526-be0a-a7f0f8f4f381/disk" dev="0:43" ino=671104350 scontext=system_u:system_r:svirt_t:s0:c319,c997 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187048.387:2339): avc:  denied  { read } for  pid=145245 comm="qemu-kvm" name="disk" dev="0:43" ino=671104350 scontext=system_u:system_r:svirt_t:s0:c319,c997 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187055.279:2387): avc:  denied  { read } for  pid=145337 comm="qemu-kvm" name="disk" dev="0:43" ino=771796179 scontext=system_u:system_r:svirt_t:s0:c461,c602 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187055.279:2388): avc:  denied  { getattr } for  pid=145337 comm="qemu-kvm" path="/var/lib/nova/instances/02e49d53-9b4c-45c8-ac22-851eeb4f29c7/disk" dev="0:43" ino=771796179 scontext=system_u:system_r:svirt_t:s0:c461,c602 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187055.279:2389): avc:  denied  { read } for  pid=145337 comm="qemu-kvm" name="disk" dev="0:43" ino=771796179 scontext=system_u:system_r:svirt_t:s0:c461,c602 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187089.052:2453): avc:  denied  { read } for  pid=145491 comm="qemu-kvm" name="disk" dev="0:43" ino=805334497 scontext=system_u:system_r:svirt_t:s0:c59,c678 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187089.052:2454): avc:  denied  { getattr } for  pid=145491 comm="qemu-kvm" path="/var/lib/nova/instances/ea6e538c-db05-4717-b954-04e9464af42f/disk" dev="0:43" ino=805334497 scontext=system_u:system_r:svirt_t:s0:c59,c678 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187089.052:2455): avc:  denied  { read } for  pid=145491 comm="qemu-kvm" name="disk" dev="0:43" ino=805334497 scontext=system_u:system_r:svirt_t:s0:c59,c678 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187097.292:2492): avc:  denied  { read } for  pid=145585 comm="qemu-kvm" name="disk" dev="0:43" ino=872449368 scontext=system_u:system_r:svirt_t:s0:c95,c536 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187097.292:2493): avc:  denied  { getattr } for  pid=145585 comm="qemu-kvm" path="/var/lib/nova/instances/d0d46e4e-1cb8-4e9c-a08f-5084d0ad5f8a/disk" dev="0:43" ino=872449368 scontext=system_u:system_r:svirt_t:s0:c95,c536 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
type=AVC msg=audit(1481187097.292:2494): avc:  denied  { read } for  pid=145585 comm="qemu-kvm" name="disk" dev="0:43" ino=872449368 scontext=system_u:system_r:svirt_t:s0:c95,c536 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file

Comment 5 Martin Schuppert 2016-12-29 15:53:58 UTC
Had the same issue, virtlogd has access to nfs with the changed context, but qemu-kvm has no access to nfs share if it is mounted using nova_var_lib_t .

What works in my lab env is:

* mount nfs share with default nfs_t context:
# mount |grep nova
192.168.122.1:/srv/nfs/nova on /var/lib/nova/instances type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.122.1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.122.1)

# ll -Z /var/lib/nova/ | grep instances
drwxrwxrwx. nova nova system_u:object_r:nfs_t:s0       instances

* make sure virt_use_nfs is set
# getsebool virt_use_nfs
virt_use_nfs --> on

* use file stdio_handler handler for libvirt:
# grep stdio_handler /etc/libvirt/qemu.conf
#stdio_handler = "logd"
stdio_handler = "file"

* restart libvirtd
# systemctl restart libvirtd

# nova list
+--------------------------------------+-------------+--------+------------+-------------+------------------+
| ID                                   | Name        | Status | Task State | Power State | Networks         |
+--------------------------------------+-------------+--------+------------+-------------+------------------+
| 4bbbeb58-d305-46eb-86a7-858505f07111 | cirros-test | ACTIVE | -          | Running     | private=10.0.0.8 |
+--------------------------------------+-------------+--------+------------+-------------+------------------+

Comment 6 Matthew Booth 2017-01-06 09:19:24 UTC
As described in comment 5, I'm pretty confident this works when correctly configured. There's a KB about it here:

  https://access.redhat.com/articles/1323213

I'm going to close this to keep BZ tidy, but please feel free to re-open if the above doesn't resolve the problem.

Comment 7 Chaitanya 2017-04-13 12:24:51 UTC
Hi,

   I am able to spawn and migrate the instances with NFS backend as per Comment#5, but I am still reopening this bug because of the security concerns involved in setting 'stdio_handler' value to 'file' in '/etc/libvirt/qemu.conf' as per the comments present in the configuration file:

~~~
# The backend to use for handling stdout/stderr output from
# QEMU processes.
#
#  'file': QEMU writes directly to a plain file. This is the
#          historical default, but allows QEMU to inflict a
#          denial of service attack on the host by exhausting
#          filesystem space
#
#  'logd': QEMU writes to a pipe provided by virtlogd daemon.
#          This is the current default, providing protection
#          against denial of service by performing log file
#          rollover when a size limit is hit.
#
#stdio_handler = "logd"
~~~

I am able to reproduce this bug in RHOS 7 and I am sure that its present though versions 7 to 10.

Comment 11 Artom Lifshitz 2017-04-21 14:35:11 UTC
The 'stdio_handler = file' workaround should not be necessary. The fix is a selinux policy that allows virtlogd to write to NFS. We need to figure out if:

1. There's such a policy available in OSP, and how to enable it if it is and/or
2. Create such a policy if one doesn't already exist

In order to do this I'm re-targeting this bug to selinux-policy.

Comment 12 Artom Lifshitz 2017-04-24 20:11:35 UTC
*** Bug 1442070 has been marked as a duplicate of this bug. ***

Comment 27 errata-xmlrpc 2017-08-01 15:17:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1861


Note You need to log in before you can comment on or make changes to this bug.