Bug 1658815 - Nova-api fails to start , sudo in nova-rootwrap blocked by SELinux
Summary: Nova-api fails to start , sudo in nova-rootwrap blocked by SELinux
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Julie Pichon
QA Contact: Jon Schlueter
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-12 22:03 UTC by Jeremy
Modified: 2019-04-18 15:00 UTC (History)
9 users (show)

Fixed In Version: openstack-selinux-0.8.18-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1694064 (view as bug list)
Environment:
Last Closed: 2019-04-18 15:00:45 UTC
Target Upstream Version:


Attachments (Terms of Use)
Diff of what solution (1) may look like (717 bytes, patch)
2019-03-07 10:11 UTC, Julie Pichon
no flags Details | Diff
Example of what solution (3) may look like (400 bytes, patch)
2019-03-08 08:37 UTC, Julie Pichon
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1676954 None ON_DEV After minor update (rhel 7.5 to 7.6) instance actions fail and neutron networking is broken 2019-03-25 13:03:16 UTC

Internal Links: 1676954

Description Jeremy 2018-12-12 22:03:22 UTC
Description of problem:

Similar to :
See same error as : https://bugzilla.redhat.com/show_bug.cgi?id=1652035

https://bugzilla.redhat.com/show_bug.cgi?id=1638547 fixed in: openstack-selinux-0.8.15-1.el7ost ###customer has : openstack-selinux-0.8.15-1.el7ost.noarch                    Thu Dec  6 08:15:00 2018

Customer has https://access.redhat.com/errata/RHBA-2018:3664 for https://bugzilla.redhat.com/show_bug.cgi?id=1645270 ,

 but https://bugzilla.redhat.com/show_bug.cgi?id=1640528 which has fixed in version: selinux- policy-3.13.1-229.el7_6.6 , is not released ?



grep denied /var/log/audit/audit.log |grep nova
type=AVC msg=audit(1544557830.595:1357): avc:  denied  { write } for  pid=10185 comm="sudo" name=".lsassd" dev="dm-3" ino=262401 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1544557830.595:1358): avc:  denied  { write } for  pid=10185 comm="sudo" name=".lsassd" dev="dm-3" ino=262401 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file permissive=0
...



tail /var/log/nova/nova-api.log 
2018-12-11 12:50:39.283 10419 ERROR nova Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
2018-12-11 12:50:39.283 10419 ERROR nova Exit code: 1
2018-12-11 12:50:39.283 10419 ERROR nova Stdout: u''
2018-12-11 12:50:39.283 10419 ERROR nova Stderr: u'sudo: PAM account management error: Error in service module\n'
2018-12-11 12:50:39.283 10419 ERROR nova 


Version-Release number of selected component (if applicable):
installed-rpms 
container-selinux-2.73-2.el7.noarch                         Thu Dec  6 08:14:28 2018
libselinux-2.5-14.1.el7.i686                                Thu Dec  6 08:16:13 2018
libselinux-2.5-14.1.el7.x86_64                              Thu Dec  6 08:11:59 2018
libselinux-devel-2.5-14.1.el7.x86_64                        Thu Dec  6 08:12:23 2018
libselinux-python-2.5-14.1.el7.x86_64                       Thu Dec  6 08:12:12 2018
libselinux-ruby-2.5-14.1.el7.x86_64                         Thu Dec  6 08:16:07 2018
libselinux-utils-2.5-14.1.el7.x86_64                        Thu Dec  6 08:12:09 2018
openstack-selinux-0.8.15-1.el7ost.noarch                    Thu Dec  6 08:15:00 2018
selinux-policy-3.13.1-229.el7_6.6.noarch                    Tue Dec 11 10:17:45 2018
selinux-policy-devel-3.13.1-229.el7_6.6.noarch              Tue Dec 11 10:17:57 2018
selinux-policy-targeted-3.13.1-229.el7_6.6.noarch           Tue Dec 11 10:18:12 2018

How reproducible:
100%

Steps to Reproduce:
1. enable selinux enforcing ; try to start nova-api
2.
3.

Actual results:
fails with ERROR nova Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
2018-12-11 12:50:39.283 10419 ERROR nova Exit code: 1

Expected results:
works

Additional info:

works with setenforce 0; the command sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c works via cli, but not from nova. 

Note: customer has fix for the bz's listed above where the same problem is identified. Need to see why the fix isn't working here..

Comment 3 Mike Burns 2019-01-14 21:48:28 UTC
Zoli any update here?

Comment 4 Zoli Caplovic 2019-01-15 16:47:11 UTC
Checking the status of the BZ #1640528 https://bugzilla.redhat.com/show_bug.cgi?id=1640528 which seems to address the same problem.

Comment 5 Zoli Caplovic 2019-01-17 16:01:44 UTC
Jeremy, 

there is related bug 1640528 (https://bugzilla.redhat.com/show_bug.cgi?id=1640528). The fix for the issue should be included in the next 7.6 update. Could you try to verify it (as soon as the update will be released) and let me know about the result, please? 


Thank you 

Zoli Caplovic

Comment 6 Jeremy 2019-01-17 16:28:35 UTC
Yes, As soon as 1640528 is released I will have the customer update and test.

Comment 7 Jeremy 2019-01-31 18:55:12 UTC
Zoil, We just realized that bug 1640528  is for rhel7.7, but the fix is already available , because erratum with BZ#1645270 is already out. The customer has even already tried the fix for 1645270 a while ago with selinux-policy-3.13.1-229.el7_6.6 and it did not help.  So it seems there is some other problem. Any thoughts?

Comment 11 Zoli Caplovic 2019-02-06 16:34:05 UTC
Jeremy, 

I have discussed the issue with colleagues. To analyse it deeper we need the (relevant part of) audit.log with SELinux set to permissive. 


Thank you in advance

Zoli Caplovic

Comment 25 Milos Malik 2019-03-01 19:02:13 UTC
Based on SELinux policy rules, there is a label that allows programs running under nova_t to write a to a lsassd socket:

# sesearch -s nova_t -c sock_file -p write -A -C | grep lsassd
   allow nsswitch_domain lsassd_var_socket_t : sock_file { write getattr append open } ; 
#

Based on SELinux fcontexts patterns, the expected location of .lsassd sockets are:

# semanage fcontext -l | grep '\.lsassd'
/var/lib/likewise/\.lsassd                         socket             system_u:object_r:lsassd_var_socket_t:s0 
/var/lib/likewise-open/\.lsassd                    socket             system_u:object_r:lsassd_var_socket_t:s0 
#

I don't think that current SELinux policy defines a more specific label for /var/lib/pbis directory, but for the .lsassd socket inside that directory we can use the lsassd_var_socket_t label. My suggestion is to run following commands:

# semanage fcontext -a -t lsassd_var_socket_t '/var/lib/pbis/.lsassd'
# restorecon -Rv /var/lib/pbis

SELinux denials mentioned in comment#0 will disappear as a result of above-mentioned commands.

Comment 28 Julie Pichon 2019-03-04 15:06:21 UTC
Many thanks for the helpful comment, Milos. It looks like the denials in audit.log are still present, and they still reference var_lib_t despite the relabelling. Would that kind of issue ring a bell?

Comment 29 Milos Malik 2019-03-04 16:54:39 UTC
It seems that /var/lib/pbis/.lsassd socket is labeled incorrectly, even if the customer applied the fcontext change via semanage and executed 'restorecon -Rv ...'

Possible cause is that /var/lib/pbis/.lsassd socket got removed. Because we don't know which process creates the .lsassd socket, we cannot run 'restorecon -Rv ...' everytime the socket gets created.

What can we do? We can use the restorecon daemon. The restorecond service makes sure that all locations written in the /etc/selinux/restorecond.conf file are labeled correctly.

# yum -y install /usr/sbin/restorecond
# echo '/var/lib/pbis/*' >> /etc/selinux/restorecond.conf
# systemctl enable restorecond
# systemctl start restorecond

Now, the /var/lib/pbis/.lsassd socket will get a correct label as soon as it gets created.

Comment 31 Julie Pichon 2019-03-05 14:39:47 UTC
Thank you for the link to the powerbroker SELinux policies, that was instructive.

https://github.com/BeyondTrust/pbis-open/blob/master/config/linux/redhat/rhel/7.0/pbis.te#L298-L476
https://github.com/BeyondTrust/pbis-open/blob/master/config/linux/redhat/rhel/7.0/pbis.if

It looks like the expectation for services communicating with powerbroker really is to have access to all var_lib_t and unconfined sockets. We'd rather not allow this by default in openstack-selinux, but we'll look at adding an optional policy that can be enabled when needed.

Comment 34 Julie Pichon 2019-03-05 16:45:27 UTC
Thank you for trying this out! This is similar to the solution we are working toward. What would be ideal is if the pbis-open selinux policy upstream could be updated to include these two requires + pbis_client calls you added. nova_t and neutron_t are defined in selinux-policy upstream so the pbis policy could reference the types directly, with no extra dependency.

Here are the possible solutions:

(1) Update pbis-open to include the pbis_client() calls for both nova_t and neutron_t

This is in my opinion the cleanest solution. It would just work.

(2) Add a boolean to openstack-selinux to optionally support pbis

Downside: pbis_client() does not exist in selinux-policy upstream, and we can't add a dependency to the pbis-open package. This means we will have to copy the current interface as is into openstack-selinux and may miss future changes. This boolean will be disabled by default so we'll need to document it as well, and the customer will need to enable it themselves. (Suggested in comment 31)

Upside: Easier from the customer perspective than (3), just a boolean to manage

(3) Customer to update pbis.te manually as described in comment 32

Downside: The change must be repeated and module recompiled with every pbis-open update.

Comment 35 Julie Pichon 2019-03-06 09:07:12 UTC
Actually about (3), it should be possible instead for the customer to create a separate .te file they can maintain separately (so no risk of overwriting) with these 4 lines (and then compile it manually + load it with semodule -i). They have the pbis-open rules on their system so they should be able to call the pbis_client() interface from any other file. The downside would be if there is a name change to the interface down the line.

Comment 36 Julie Pichon 2019-03-07 10:11:12 UTC
Created attachment 1541795 [details]
Diff of what solution (1) may look like

Comment 37 Julie Pichon 2019-03-08 08:34:22 UTC
Pull request for stopgap solution (2) until (1) is available - https://github.com/redhat-openstack/openstack-selinux/pull/29

Comment 38 Julie Pichon 2019-03-08 08:37:46 UTC
Created attachment 1542069 [details]
Example of what solution (3) may look like

Comment 39 Julie Pichon 2019-03-29 11:19:41 UTC
Build with "openstack_pbis_support" boolean (#2 above).

Comment 40 Mike Burns 2019-04-18 15:00:45 UTC
This bug is being closed due to the retirement of OSP 8.


Note You need to log in before you can comment on or make changes to this bug.