Bug 1377272
| Summary: | packstack-AIO+COMPUTE- Rhel7.3 - VMs wil not boot unless selinux permissive | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Stafeyev <astafeye> | |
| Component: | openstack-selinux | Assignee: | Ryan Hallisey <rhallise> | |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Stafeyev <astafeye> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 10.0 (Newton) | CC: | bhaubeck, fwissing, jschluet, lhh, mbooth, mburns, mgrepl, oblaut, rcernin, slinaber, srevivo, ssigwald | |
| Target Milestone: | rc | Keywords: | Reopened, Triaged, ZStream | |
| Target Release: | 10.0 (Newton) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-selinux-0.7.9-1.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1499800 (view as bug list) | Environment: | ||
| Last Closed: | 2017-05-26 13:50:18 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1499800 | |||
Additional info :
[root@rose11 ~(keystone_alex2)]# cat /var/log/audit/audit.log | audit2allow -R
require {
type iptables_t;
type virtlogd_t;
}
#============= iptables_t ==============
plymouthd_exec_plymouth(iptables_t)
#============= virtlogd_t ==============
nova_manage_lib_files(virtlogd_t)
This AVC is already fixed in the latest policy.
type=AVC msg=audit(1474273569.161:2226): avc: denied { search } for pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir
I already tested with latest rpm. The issue still there. [root@rose11 ~]# rpm -qa | grep openstack-selinux-0. openstack-selinux-0.7.8-2.el7ost.noarch Also seen impacting tripleo based installs The failure I'm seeing locally is this one:
type=AVC msg=audit(1474295568.657:7377): avc: denied { dac_override } for pid=22071 comm="virtlogd" capability=1 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability
type=SYSCALL msg=audit(1474295568.657:7377): arch=c000003e syscall=2 success=no exit=-13 a0=7f90c4000d30 a1=80441 a2=180 a3=7f90c4000d90 items=0 ppid=1 pid=22071 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="virtlogd" exe="/usr/sbin/virtlogd" subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null)
Note that Nova (intentionally, for compatibility with the weird config requirements of Quobyte) pre-creates console.log owned by the nova user. For everybody else, we then trust libvirt (now virtlogd) to fixup the ownership and permissions automatically for us. Seems this isn't happening due to the above AVC.
I can confirm that piping the above into audit2allow and loading it fixes the problem on my rhos10 packstack. Are you booting with a graphical display?
type=AVC msg=audit(1474267966.57:942): avc: denied { execute } for pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file
Ryan, No, that's not it. My testing was already against 0.7.8-2 as I only installed this morning. See comment 6 for the required fix. It specifically relates to console.log, which is not related to a graphical console. Confirmed openstack-selinux-0.7.9-1.el7ost wfm. *** Bug 1375766 has been marked as a duplicate of this bug. *** [root@rose11 ~(keystone_alex2)]# nova list +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ | 6e0346d1-508b-4323-a065-d43401776a3b | 2Alex_1_bug_ver | ACTIVE | - | Running | Alex2_net=192.168.100.9 | | 95c22f09-15c0-42d0-9a5c-e68cef52dbd9 | 2Alex_1_bug_ver2 | ACTIVE | - | Running | Alex2_net=192.168.100.10 | +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ openstack-selinux-0.7.9-1.el7ost.noarch [root@rose11 ~(keystone_alex2)]# getenforce Enforcing I'm having the same issue with Red Hat Enterprise Linux Server release 7.3 RHOSP8
Linux e1-compute-05.eng1.moc.edu 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
nova/nova-compute.log:2016-11-07 18:20:30.842 4453 ERROR nova.compute.manager [instance: 09f31518-1cd1-4904-b51d-4bdadf78a9a0] libvirtError: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied
messages:Nov 7 18:20:30 e1-compute-05 journal: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied
type=AVC msg=audit(1478619625.854:97549): avc: denied { open } for pid=11208 comm="virtlogd" path="/var/lib/nova/instances/98c175ec-6be0-4de0-88c9-e54774fed778/console.log" dev="dm-0" ino=268919030 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file
installed packages in compute node
openstack-nova-api-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016
openstack-nova-common-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016
openstack-nova-compute-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016
python-nova-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:11 2016
python-novaclient-3.1.0-2.el7ost.noarch Thu Jul 28 01:21:44 2016
libselinux-2.5-6.el7.x86_64 Fri Nov 4 05:13:07 2016
libselinux-python-2.5-6.el7.x86_64 Fri Nov 4 05:13:34 2016
libselinux-ruby-2.5-6.el7.x86_64 Fri Nov 4 05:15:55 2016
libselinux-utils-2.5-6.el7.x86_64 Fri Nov 4 05:13:38 2016
openstack-selinux-0.6.58-1.el7ost.noarch Thu Jul 28 01:22:27 2016
selinux-policy-3.13.1-102.el7_3.4.noarch Fri Nov 4 05:13:38 2016
selinux-policy-targeted-3.13.1-102.el7_3.4.noarch Fri Nov 4 05:14:01 2016
Lon, when will openstack-selinux be tagged around to all the releases if it hasn't already? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html re-open it. we had a PoC with RHOSP11 these days and hit this bug again. Joachim von Thadden has solved it by: - login on a compute - yum -y install setroubleshoot - setenforce 0 - start the VMs - ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd - semodule -i my-virtlogd.pp - setenfoce 1 distribute my-virtlogd.pp to all nodes Please do not re-open bugs that are closed Errata. If you are encountering this issue, please clone the bug or file a new bug against the appropriate release. This bug was for OSP 10. You reopened but are testing OSP 11. Please provide details in the new bug: * What version of of packages are being used, especially openstack-selinux and packstack if that is what you used. Please also include the RHEL selinux package versions. * full audit.log with the system in permissive. Note: audit2allow will generally provide *a* solution, but not necessarily the *right* solution. We need the audit.log with details to determine that. |
Description of problem: VMs are not booting due to permission deny message from virtlib. 2016-09-19 09:50:51.484 2489 ERROR nova.compute.manager [instance: ae2464b8-2492-4a5b-9573-56476e2a0168] libvirtError: Unable to open file: /var/lib/nova/instances/ae2464b8-2492-4a5b-9573-56476e2a0168/console.log: Permission denied After setting "setenforce 0" the VMs are booting well. Version-Release number of selected component (if applicable): [root@rose11 ~(keystone_alex2)]# rpm -qa | grep nova openstack-nova-novncproxy-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-scheduler-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-cert-14.0.0-0.20160823074012.21312bf.el7ost.noarch python-nova-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-compute-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-console-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-conductor-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-common-14.0.0-0.20160823074012.21312bf.el7ost.noarch puppet-nova-9.1.0-0.20160823051658.5075d8b.el7ost.noarch openstack-nova-api-14.0.0-0.20160823074012.21312bf.el7ost.noarch python-novaclient-5.0.0-0.20160802172215.5eb7b65.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy rhos10 with packstack 2. Boot VM 3. Actual results: VM is in error state Expected results: VM should be running Additional info: [root@rose11 ~(keystone_alex2)]# sealert -a /var/log/audit/audit.log 100% done found 2 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/virtlogd from search access on the directory /var/lib/nova. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that virtlogd should be allowed search access on the nova directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd # semodule -i my-virtlogd.pp Additional Information: Source Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Context system_u:object_r:nova_var_lib_t:s0 Target Objects /var/lib/nova [ dir ] Source virtlogd Source Path /usr/sbin/virtlogd Port <Unknown> Host <Unknown> Source RPM Packages libvirt-daemon-2.0.0-6.el7.x86_64 Target RPM Packages openstack-nova-common-14.0.0-0.20160823074012.2131 2bf.el7ost.noarch Policy RPM selinux-policy-3.13.1-99.el7.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name rose11.scl.lab.tlv.redhat.com Platform Linux rose11.scl.lab.tlv.redhat.com 3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04 EDT 2016 x86_64 x86_64 Alert Count 16 First Seen 2016-09-19 09:50:57 IDT Last Seen 2016-09-19 11:26:09 IDT Local ID 124de55d-fbff-43b2-9322-54f2f2e5718e Raw Audit Messages type=AVC msg=audit(1474273569.161:2226): avc: denied { search } for pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir type=SYSCALL msg=audit(1474273569.161:2226): arch=x86_64 syscall=open success=no exit=EACCES a0=7f0f1c003620 a1=80441 a2=180 a3=7f0f1c000cc0 items=0 ppid=1 pid=7027 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Hash: virtlogd,virtlogd_t,nova_var_lib_t,dir,search -------------------------------------------------------------------------------- SELinux is preventing /usr/bin/bash from execute access on the file /usr/bin/plymouth. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that bash should be allowed execute access on the plymouth file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'iptables.init' --raw | audit2allow -M my-iptablesinit # semodule -i my-iptablesinit.pp Additional Information: Source Context system_u:system_r:iptables_t:s0 Target Context system_u:object_r:plymouth_exec_t:s0 Target Objects /usr/bin/plymouth [ file ] Source iptables.init Source Path /usr/bin/bash Port <Unknown> Host <Unknown> Source RPM Packages bash-4.2.46-20.el7_2.x86_64 Target RPM Packages plymouth-0.8.9-0.26.20140113.el7.x86_64 Policy RPM selinux-policy-3.13.1-99.el7.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name rose11.scl.lab.tlv.redhat.com Platform Linux rose11.scl.lab.tlv.redhat.com 3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04 EDT 2016 x86_64 x86_64 Alert Count 1 First Seen 2016-09-19 09:52:46 IDT Last Seen 2016-09-19 09:52:46 IDT Local ID 6222b6bc-60b6-4879-8729-daa055478071 Raw Audit Messages type=AVC msg=audit(1474267966.57:942): avc: denied { execute } for pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=SYSCALL msg=audit(1474267966.57:942): arch=x86_64 syscall=faccessat success=no exit=EACCES a0=ffffffffffffff9c a1=245a090 a2=1 a3=8 items=0 ppid=1 pid=19169 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=iptables.init exe=/usr/bin/bash subj=system_u:system_r:iptables_t:s0 key=(null) Hash: iptables.init,iptables_t,plymouth_exec_t,file,execute [root@rose11 ~(keystone_alex2)]# rpm -qa | grep seli selinux-policy-devel-3.13.1-99.el7.noarch libselinux-2.5-6.el7.x86_64 libselinux-ruby-2.5-6.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 selinux-policy-3.13.1-99.el7.noarch selinux-policy-targeted-3.13.1-99.el7.noarch libselinux-python-2.5-6.el7.x86_64 openstack-selinux-0.7.7-1.el7ost.noarch