Bug 1377272 - packstack-AIO+COMPUTE- Rhel7.3 - VMs wil not boot unless selinux permissive
Summary: packstack-AIO+COMPUTE- Rhel7.3 - VMs wil not boot unless selinux permissive
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 10.0 (Newton)
Assignee: Ryan Hallisey
QA Contact: Alexander Stafeyev
URL:
Whiteboard:
: 1375766 (view as bug list)
Depends On:
Blocks: 1499800
TreeView+ depends on / blocked
 
Reported: 2016-09-19 11:12 UTC by Alexander Stafeyev
Modified: 2017-10-12 17:34 UTC (History)
12 users (show)

Fixed In Version: openstack-selinux-0.7.9-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1499800 (view as bug list)
Environment:
Last Closed: 2017-05-26 13:50:18 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Alexander Stafeyev 2016-09-19 11:12:39 UTC
Description of problem:
VMs are not booting due to permission deny message from virtlib.

2016-09-19 09:50:51.484 2489 ERROR nova.compute.manager [instance: ae2464b8-2492-4a5b-9573-56476e2a0168] libvirtError: Unable to open file: /var/lib/nova/instances/ae2464b8-2492-4a5b-9573-56476e2a0168/console.log: Permission denied


After setting "setenforce 0" the VMs are booting well. 

Version-Release number of selected component (if applicable):
[root@rose11 ~(keystone_alex2)]# rpm -qa | grep nova
openstack-nova-novncproxy-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-scheduler-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-cert-14.0.0-0.20160823074012.21312bf.el7ost.noarch
python-nova-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-compute-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-console-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-conductor-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-nova-common-14.0.0-0.20160823074012.21312bf.el7ost.noarch
puppet-nova-9.1.0-0.20160823051658.5075d8b.el7ost.noarch
openstack-nova-api-14.0.0-0.20160823074012.21312bf.el7ost.noarch
python-novaclient-5.0.0-0.20160802172215.5eb7b65.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy rhos10 with packstack 
2. Boot VM 
3.

Actual results:
VM is in error state 

Expected results:
VM should be running

Additional info:
[root@rose11 ~(keystone_alex2)]# sealert -a /var/log/audit/audit.log
100% done
found 2 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------

SELinux is preventing /usr/sbin/virtlogd from search access on the directory /var/lib/nova.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that virtlogd should be allowed search access on the nova directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd
# semodule -i my-virtlogd.pp


Additional Information:
Source Context                system_u:system_r:virtlogd_t:s0-s0:c0.c1023
Target Context                system_u:object_r:nova_var_lib_t:s0
Target Objects                /var/lib/nova [ dir ]
Source                        virtlogd
Source Path                   /usr/sbin/virtlogd
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           libvirt-daemon-2.0.0-6.el7.x86_64
Target RPM Packages           openstack-nova-common-14.0.0-0.20160823074012.2131
                              2bf.el7ost.noarch
Policy RPM                    selinux-policy-3.13.1-99.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     rose11.scl.lab.tlv.redhat.com
Platform                      Linux rose11.scl.lab.tlv.redhat.com
                              3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04
                              EDT 2016 x86_64 x86_64
Alert Count                   16
First Seen                    2016-09-19 09:50:57 IDT
Last Seen                     2016-09-19 11:26:09 IDT
Local ID                      124de55d-fbff-43b2-9322-54f2f2e5718e

Raw Audit Messages
type=AVC msg=audit(1474273569.161:2226): avc:  denied  { search } for  pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir


type=SYSCALL msg=audit(1474273569.161:2226): arch=x86_64 syscall=open success=no exit=EACCES a0=7f0f1c003620 a1=80441 a2=180 a3=7f0f1c000cc0 items=0 ppid=1 pid=7027 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null)

Hash: virtlogd,virtlogd_t,nova_var_lib_t,dir,search

--------------------------------------------------------------------------------

SELinux is preventing /usr/bin/bash from execute access on the file /usr/bin/plymouth.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that bash should be allowed execute access on the plymouth file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'iptables.init' --raw | audit2allow -M my-iptablesinit
# semodule -i my-iptablesinit.pp


Additional Information:
Source Context                system_u:system_r:iptables_t:s0
Target Context                system_u:object_r:plymouth_exec_t:s0
Target Objects                /usr/bin/plymouth [ file ]
Source                        iptables.init
Source Path                   /usr/bin/bash
Port                          <Unknown>
Host                          <Unknown>
Source RPM Packages           bash-4.2.46-20.el7_2.x86_64
Target RPM Packages           plymouth-0.8.9-0.26.20140113.el7.x86_64
Policy RPM                    selinux-policy-3.13.1-99.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     rose11.scl.lab.tlv.redhat.com
Platform                      Linux rose11.scl.lab.tlv.redhat.com
                              3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04
                              EDT 2016 x86_64 x86_64
Alert Count                   1
First Seen                    2016-09-19 09:52:46 IDT
Last Seen                     2016-09-19 09:52:46 IDT
Local ID                      6222b6bc-60b6-4879-8729-daa055478071

Raw Audit Messages
type=AVC msg=audit(1474267966.57:942): avc:  denied  { execute } for  pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file


type=SYSCALL msg=audit(1474267966.57:942): arch=x86_64 syscall=faccessat success=no exit=EACCES a0=ffffffffffffff9c a1=245a090 a2=1 a3=8 items=0 ppid=1 pid=19169 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=iptables.init exe=/usr/bin/bash subj=system_u:system_r:iptables_t:s0 key=(null)

Hash: iptables.init,iptables_t,plymouth_exec_t,file,execute


[root@rose11 ~(keystone_alex2)]# rpm -qa | grep seli
selinux-policy-devel-3.13.1-99.el7.noarch
libselinux-2.5-6.el7.x86_64
libselinux-ruby-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-3.13.1-99.el7.noarch
selinux-policy-targeted-3.13.1-99.el7.noarch
libselinux-python-2.5-6.el7.x86_64
openstack-selinux-0.7.7-1.el7ost.noarch

Comment 2 Alexander Stafeyev 2016-09-19 12:40:46 UTC
Additional info : 

[root@rose11 ~(keystone_alex2)]# cat /var/log/audit/audit.log | audit2allow -R

require {
	type iptables_t;
	type virtlogd_t;
}

#============= iptables_t ==============
plymouthd_exec_plymouth(iptables_t)

#============= virtlogd_t ==============
nova_manage_lib_files(virtlogd_t)

Comment 3 Ryan Hallisey 2016-09-19 13:35:48 UTC
This AVC is already fixed in the latest policy.

type=AVC msg=audit(1474273569.161:2226): avc:  denied  { search } for  pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir

Comment 4 Alexander Stafeyev 2016-09-19 13:39:05 UTC
I already tested with latest rpm.

The issue still there. 

[root@rose11 ~]# rpm -qa | grep openstack-selinux-0.
openstack-selinux-0.7.8-2.el7ost.noarch

Comment 5 Jon Schlueter 2016-09-19 14:36:40 UTC
Also seen impacting tripleo based installs

Comment 6 Matthew Booth 2016-09-19 14:44:43 UTC
The failure I'm seeing locally is this one:

type=AVC msg=audit(1474295568.657:7377): avc:  denied  { dac_override } for  pid=22071 comm="virtlogd" capability=1  scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability
type=SYSCALL msg=audit(1474295568.657:7377): arch=c000003e syscall=2 success=no exit=-13 a0=7f90c4000d30 a1=80441 a2=180 a3=7f90c4000d90 items=0 ppid=1 pid=22071 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="virtlogd" exe="/usr/sbin/virtlogd" subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null)

Note that Nova (intentionally, for compatibility with the weird config requirements of Quobyte) pre-creates console.log owned by the nova user. For everybody else, we then trust libvirt (now virtlogd) to fixup the ownership and permissions automatically for us. Seems this isn't happening due to the above AVC.

Comment 7 Matthew Booth 2016-09-19 14:46:51 UTC
I can confirm that piping the above into audit2allow and loading it fixes the problem on my rhos10 packstack.

Comment 8 Ryan Hallisey 2016-09-19 15:08:04 UTC
Are you booting with a graphical display?

type=AVC msg=audit(1474267966.57:942): avc:  denied  { execute } for  pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file

Comment 9 Matthew Booth 2016-09-19 15:59:45 UTC
Ryan,

No, that's not it. My testing was already against 0.7.8-2 as I only installed this morning. See comment 6 for the required fix. It specifically relates to console.log, which is not related to a graphical console.

Comment 10 Matthew Booth 2016-09-19 16:12:09 UTC
Confirmed openstack-selinux-0.7.9-1.el7ost wfm.

Comment 11 Ryan Hallisey 2016-09-20 17:32:31 UTC
*** Bug 1375766 has been marked as a duplicate of this bug. ***

Comment 13 Alexander Stafeyev 2016-09-25 06:21:40 UTC
[root@rose11 ~(keystone_alex2)]# nova list 
+--------------------------------------+------------------+---------+------------+-------------+--------------------------+
| ID                                   | Name             | Status  | Task State | Power State | Networks                 |
+--------------------------------------+------------------+---------+------------+-------------+--------------------------+

| 6e0346d1-508b-4323-a065-d43401776a3b | 2Alex_1_bug_ver  | ACTIVE  | -          | Running     | Alex2_net=192.168.100.9  |
| 95c22f09-15c0-42d0-9a5c-e68cef52dbd9 | 2Alex_1_bug_ver2 | ACTIVE  | -          | Running     | Alex2_net=192.168.100.10 |

+--------------------------------------+------------------+---------+------------+-------------+--------------------------+

openstack-selinux-0.7.9-1.el7ost.noarch


[root@rose11 ~(keystone_alex2)]# getenforce 
Enforcing

Comment 15 Siggy Sigwald 2016-11-10 15:59:12 UTC
I'm having the same issue with Red Hat Enterprise Linux Server release 7.3 RHOSP8 

Linux e1-compute-05.eng1.moc.edu 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

nova/nova-compute.log:2016-11-07 18:20:30.842 4453 ERROR nova.compute.manager [instance: 09f31518-1cd1-4904-b51d-4bdadf78a9a0] libvirtError: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied

messages:Nov  7 18:20:30 e1-compute-05 journal: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied

type=AVC msg=audit(1478619625.854:97549): avc:  denied  { open } for  pid=11208 comm="virtlogd" path="/var/lib/nova/instances/98c175ec-6be0-4de0-88c9-e54774fed778/console.log" dev="dm-0" ino=268919030 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file

installed packages in compute node

openstack-nova-api-12.0.4-8.el7ost.noarch                   Thu Sep  1 03:41:12 2016
openstack-nova-common-12.0.4-8.el7ost.noarch                Thu Sep  1 03:41:12 2016
openstack-nova-compute-12.0.4-8.el7ost.noarch               Thu Sep  1 03:41:12 2016
python-nova-12.0.4-8.el7ost.noarch                          Thu Sep  1 03:41:11 2016
python-novaclient-3.1.0-2.el7ost.noarch                     Thu Jul 28 01:21:44 2016

libselinux-2.5-6.el7.x86_64                                 Fri Nov  4 05:13:07 2016
libselinux-python-2.5-6.el7.x86_64                          Fri Nov  4 05:13:34 2016
libselinux-ruby-2.5-6.el7.x86_64                            Fri Nov  4 05:15:55 2016
libselinux-utils-2.5-6.el7.x86_64                           Fri Nov  4 05:13:38 2016
openstack-selinux-0.6.58-1.el7ost.noarch                    Thu Jul 28 01:22:27 2016
selinux-policy-3.13.1-102.el7_3.4.noarch                    Fri Nov  4 05:13:38 2016
selinux-policy-targeted-3.13.1-102.el7_3.4.noarch           Fri Nov  4 05:14:01 2016

Comment 16 Ryan Hallisey 2016-11-10 21:32:49 UTC
Lon, when will openstack-selinux be tagged around to all the releases if it hasn't already?

Comment 19 errata-xmlrpc 2016-12-14 16:02:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Comment 20 ben haubeck 2017-05-24 14:23:27 UTC
re-open it.
we had a PoC with RHOSP11 these days and hit this bug again. 

Joachim von Thadden has solved it by:

- login on a compute
- yum -y install setroubleshoot
- setenforce 0
- start the VMs
- ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd
- semodule -i my-virtlogd.pp
- setenfoce 1

distribute my-virtlogd.pp to all nodes

Comment 21 Mike Burns 2017-05-26 13:50:18 UTC
Please do not re-open bugs that are closed Errata.  If you are encountering this issue, please clone the bug or file a new bug against the appropriate release.  

This bug was for OSP 10.  You reopened but are testing OSP 11.  

Please provide details in the new bug:

* What version of of packages are being used, especially openstack-selinux and packstack if that is what you used.  Please also include the RHEL selinux package versions.
* full audit.log with the system in permissive.

Note:  audit2allow will generally provide *a* solution, but not necessarily the *right* solution.  We need the audit.log with details to determine that.


Note You need to log in before you can comment on or make changes to this bug.