We are hitting still same on OSP10 with RHEL7.4. Versions: [root@overcloud-compute-27 audit]# rpm -qa | grep openstack-selinux- openstack-selinux-0.8.9-0.1.el7ost.noarch [root@overcloud-compute-27 audit]# rpm -qa | grep nova openstack-nova-migration-14.0.8-2.el7ost.noarch openstack-nova-cert-14.0.8-2.el7ost.noarch python-nova-14.0.8-2.el7ost.noarch openstack-nova-conductor-14.0.8-2.el7ost.noarch openstack-nova-compute-14.0.8-2.el7ost.noarch python-novaclient-6.0.1-1.el7ost.noarch openstack-nova-common-14.0.8-2.el7ost.noarch openstack-nova-novncproxy-14.0.8-2.el7ost.noarch openstack-nova-scheduler-14.0.8-2.el7ost.noarch puppet-nova-9.6.0-1.el7ost.noarch openstack-nova-console-14.0.8-2.el7ost.noarch openstack-nova-api-14.0.8-2.el7ost.noarch [root@overcloud-compute-27 audit]# egrep "console.log" * | grep -v "success" audit.log:type=AVC msg=audit(1507377606.080:415539): avc: denied { rename } for pid=15357 comm="virtlogd" name="console.log" dev="dm-0" ino=3221227146 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c274,c402 tclass=file audit.log:type=AVC msg=audit(1507385492.308:269): avc: denied { unlink } for pid=8809 comm="virtlogd" name="console.log.0" dev="dm-0" ino=167 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c990,c1016 tclass=file audit.log:type=AVC msg=audit(1507385492.819:287): avc: denied { unlink } for pid=8809 comm="virtlogd" name="console.log.0" dev="dm-0" ino=3221227146 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c274,c402 tclass=file audit.log:type=AVC msg=audit(1507549569.916:20400): avc: denied { unlink } for pid=8809 comm="virtlogd" name="console.log.0" dev="dm-0" ino=3221227146 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c274,c402 tclass=file audit.log.1:type=AVC msg=audit(1507327380.037:407151): avc: denied { rename } for pid=15357 comm="virtlogd" name="console.log" dev="dm-0" ino=3758096531 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c674,c854 tclass=file in logs: nova-compute.log:8975:2017-10-09 11:46:10.219 55172 INFO nova.compute.manager [req-fbff579f-5dbd-49b2-bc93-7da7f0cb57e8 59d2cda2f4df4cc5853468c881cec091 4cbee9a6624941c4966017b73132228f - - -] [instance: 8c3111c0-3d0b-41f7-b344-90c4eb0af27b] Successfully reverted task state from powering-on on failure for instance. nova-compute.log:9046:2017-10-09 11:46:10.242 55172 ERROR oslo_messaging.rpc.server libvirtError: Unable to delete file /var/lib/nova/instances/8c3111c0-3d0b-41f7-b344-90c4eb0af27b/console.log.0: Permission denied Solved by: rm /var/lib/nova/instances/8c3111c0-3d0b-41f7-b344-90c4eb0af27b/console.log.0 Start the instance, to re-create the console.log +++ This bug was initially created as a clone of Bug #1377272 +++ Description of problem: VMs are not booting due to permission deny message from virtlib. 2016-09-19 09:50:51.484 2489 ERROR nova.compute.manager [instance: ae2464b8-2492-4a5b-9573-56476e2a0168] libvirtError: Unable to open file: /var/lib/nova/instances/ae2464b8-2492-4a5b-9573-56476e2a0168/console.log: Permission denied After setting "setenforce 0" the VMs are booting well. Version-Release number of selected component (if applicable): [root@rose11 ~(keystone_alex2)]# rpm -qa | grep nova openstack-nova-novncproxy-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-scheduler-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-cert-14.0.0-0.20160823074012.21312bf.el7ost.noarch python-nova-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-compute-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-console-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-conductor-14.0.0-0.20160823074012.21312bf.el7ost.noarch openstack-nova-common-14.0.0-0.20160823074012.21312bf.el7ost.noarch puppet-nova-9.1.0-0.20160823051658.5075d8b.el7ost.noarch openstack-nova-api-14.0.0-0.20160823074012.21312bf.el7ost.noarch python-novaclient-5.0.0-0.20160802172215.5eb7b65.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy rhos10 with packstack 2. Boot VM 3. Actual results: VM is in error state Expected results: VM should be running Additional info: [root@rose11 ~(keystone_alex2)]# sealert -a /var/log/audit/audit.log 100% done found 2 alerts in /var/log/audit/audit.log -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/virtlogd from search access on the directory /var/lib/nova. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that virtlogd should be allowed search access on the nova directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd # semodule -i my-virtlogd.pp Additional Information: Source Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Context system_u:object_r:nova_var_lib_t:s0 Target Objects /var/lib/nova [ dir ] Source virtlogd Source Path /usr/sbin/virtlogd Port <Unknown> Host <Unknown> Source RPM Packages libvirt-daemon-2.0.0-6.el7.x86_64 Target RPM Packages openstack-nova-common-14.0.0-0.20160823074012.2131 2bf.el7ost.noarch Policy RPM selinux-policy-3.13.1-99.el7.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name rose11.scl.lab.tlv.redhat.com Platform Linux rose11.scl.lab.tlv.redhat.com 3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04 EDT 2016 x86_64 x86_64 Alert Count 16 First Seen 2016-09-19 09:50:57 IDT Last Seen 2016-09-19 11:26:09 IDT Local ID 124de55d-fbff-43b2-9322-54f2f2e5718e Raw Audit Messages type=AVC msg=audit(1474273569.161:2226): avc: denied { search } for pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir type=SYSCALL msg=audit(1474273569.161:2226): arch=x86_64 syscall=open success=no exit=EACCES a0=7f0f1c003620 a1=80441 a2=180 a3=7f0f1c000cc0 items=0 ppid=1 pid=7027 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Hash: virtlogd,virtlogd_t,nova_var_lib_t,dir,search -------------------------------------------------------------------------------- SELinux is preventing /usr/bin/bash from execute access on the file /usr/bin/plymouth. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that bash should be allowed execute access on the plymouth file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'iptables.init' --raw | audit2allow -M my-iptablesinit # semodule -i my-iptablesinit.pp Additional Information: Source Context system_u:system_r:iptables_t:s0 Target Context system_u:object_r:plymouth_exec_t:s0 Target Objects /usr/bin/plymouth [ file ] Source iptables.init Source Path /usr/bin/bash Port <Unknown> Host <Unknown> Source RPM Packages bash-4.2.46-20.el7_2.x86_64 Target RPM Packages plymouth-0.8.9-0.26.20140113.el7.x86_64 Policy RPM selinux-policy-3.13.1-99.el7.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name rose11.scl.lab.tlv.redhat.com Platform Linux rose11.scl.lab.tlv.redhat.com 3.10.0-500.el7.x86_64 #1 SMP Tue Aug 30 18:58:04 EDT 2016 x86_64 x86_64 Alert Count 1 First Seen 2016-09-19 09:52:46 IDT Last Seen 2016-09-19 09:52:46 IDT Local ID 6222b6bc-60b6-4879-8729-daa055478071 Raw Audit Messages type=AVC msg=audit(1474267966.57:942): avc: denied { execute } for pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file type=SYSCALL msg=audit(1474267966.57:942): arch=x86_64 syscall=faccessat success=no exit=EACCES a0=ffffffffffffff9c a1=245a090 a2=1 a3=8 items=0 ppid=1 pid=19169 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=iptables.init exe=/usr/bin/bash subj=system_u:system_r:iptables_t:s0 key=(null) Hash: iptables.init,iptables_t,plymouth_exec_t,file,execute [root@rose11 ~(keystone_alex2)]# rpm -qa | grep seli selinux-policy-devel-3.13.1-99.el7.noarch libselinux-2.5-6.el7.x86_64 libselinux-ruby-2.5-6.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 selinux-policy-3.13.1-99.el7.noarch selinux-policy-targeted-3.13.1-99.el7.noarch libselinux-python-2.5-6.el7.x86_64 openstack-selinux-0.7.7-1.el7ost.noarch --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-09-19 07:12:45 EDT --- Since this issue was entered in bugzilla without a release flag set, rhos-10.0? has been automatically added to ensure that it is properly evaluated for this release. --- Additional comment from Alexander Stafeyev on 2016-09-19 08:40:46 EDT --- Additional info : [root@rose11 ~(keystone_alex2)]# cat /var/log/audit/audit.log | audit2allow -R require { type iptables_t; type virtlogd_t; } #============= iptables_t ============== plymouthd_exec_plymouth(iptables_t) #============= virtlogd_t ============== nova_manage_lib_files(virtlogd_t) --- Additional comment from Ryan Hallisey on 2016-09-19 09:35:48 EDT --- This AVC is already fixed in the latest policy. type=AVC msg=audit(1474273569.161:2226): avc: denied { search } for pid=7027 comm="virtlogd" name="nova" dev="sda5" ino=17564758 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir --- Additional comment from Alexander Stafeyev on 2016-09-19 09:39:05 EDT --- I already tested with latest rpm. The issue still there. [root@rose11 ~]# rpm -qa | grep openstack-selinux-0. openstack-selinux-0.7.8-2.el7ost.noarch --- Additional comment from Jon Schlueter on 2016-09-19 10:36:40 EDT --- Also seen impacting tripleo based installs --- Additional comment from Matthew Booth on 2016-09-19 10:44:43 EDT --- The failure I'm seeing locally is this one: type=AVC msg=audit(1474295568.657:7377): avc: denied { dac_override } for pid=22071 comm="virtlogd" capability=1 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability type=SYSCALL msg=audit(1474295568.657:7377): arch=c000003e syscall=2 success=no exit=-13 a0=7f90c4000d30 a1=80441 a2=180 a3=7f90c4000d90 items=0 ppid=1 pid=22071 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="virtlogd" exe="/usr/sbin/virtlogd" subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Note that Nova (intentionally, for compatibility with the weird config requirements of Quobyte) pre-creates console.log owned by the nova user. For everybody else, we then trust libvirt (now virtlogd) to fixup the ownership and permissions automatically for us. Seems this isn't happening due to the above AVC. --- Additional comment from Matthew Booth on 2016-09-19 10:46:51 EDT --- I can confirm that piping the above into audit2allow and loading it fixes the problem on my rhos10 packstack. --- Additional comment from Ryan Hallisey on 2016-09-19 11:08:04 EDT --- Are you booting with a graphical display? type=AVC msg=audit(1474267966.57:942): avc: denied { execute } for pid=19169 comm="iptables.init" name="plymouth" dev="sda5" ino=3411763 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file --- Additional comment from Matthew Booth on 2016-09-19 11:59:45 EDT --- Ryan, No, that's not it. My testing was already against 0.7.8-2 as I only installed this morning. See comment 6 for the required fix. It specifically relates to console.log, which is not related to a graphical console. --- Additional comment from Matthew Booth on 2016-09-19 12:12:09 EDT --- Confirmed openstack-selinux-0.7.9-1.el7ost wfm. --- Additional comment from Ryan Hallisey on 2016-09-20 13:32:31 EDT --- --- Additional comment from errata-xmlrpc on 2016-09-20 15:47:26 EDT --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHEA-2016:23819-02 https://errata.devel.redhat.com/advisory/23819 --- Additional comment from Alexander Stafeyev on 2016-09-25 02:21:40 EDT --- [root@rose11 ~(keystone_alex2)]# nova list +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ | 6e0346d1-508b-4323-a065-d43401776a3b | 2Alex_1_bug_ver | ACTIVE | - | Running | Alex2_net=192.168.100.9 | | 95c22f09-15c0-42d0-9a5c-e68cef52dbd9 | 2Alex_1_bug_ver2 | ACTIVE | - | Running | Alex2_net=192.168.100.10 | +--------------------------------------+------------------+---------+------------+-------------+--------------------------+ openstack-selinux-0.7.9-1.el7ost.noarch [root@rose11 ~(keystone_alex2)]# getenforce Enforcing --- Additional comment from Mike Burns on 2016-10-12 09:45:38 EDT --- Adding Triaged for resolved bugs --- Additional comment from Siggy Sigwald on 2016-11-10 10:59:12 EST --- I'm having the same issue with Red Hat Enterprise Linux Server release 7.3 RHOSP8 Linux e1-compute-05.eng1.moc.edu 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux nova/nova-compute.log:2016-11-07 18:20:30.842 4453 ERROR nova.compute.manager [instance: 09f31518-1cd1-4904-b51d-4bdadf78a9a0] libvirtError: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied messages:Nov 7 18:20:30 e1-compute-05 journal: Unable to open file: /var/lib/nova/instances/09f31518-1cd1-4904-b51d-4bdadf78a9a0/console.log: Permission denied type=AVC msg=audit(1478619625.854:97549): avc: denied { open } for pid=11208 comm="virtlogd" path="/var/lib/nova/instances/98c175ec-6be0-4de0-88c9-e54774fed778/console.log" dev="dm-0" ino=268919030 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file installed packages in compute node openstack-nova-api-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016 openstack-nova-common-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016 openstack-nova-compute-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:12 2016 python-nova-12.0.4-8.el7ost.noarch Thu Sep 1 03:41:11 2016 python-novaclient-3.1.0-2.el7ost.noarch Thu Jul 28 01:21:44 2016 libselinux-2.5-6.el7.x86_64 Fri Nov 4 05:13:07 2016 libselinux-python-2.5-6.el7.x86_64 Fri Nov 4 05:13:34 2016 libselinux-ruby-2.5-6.el7.x86_64 Fri Nov 4 05:15:55 2016 libselinux-utils-2.5-6.el7.x86_64 Fri Nov 4 05:13:38 2016 openstack-selinux-0.6.58-1.el7ost.noarch Thu Jul 28 01:22:27 2016 selinux-policy-3.13.1-102.el7_3.4.noarch Fri Nov 4 05:13:38 2016 selinux-policy-targeted-3.13.1-102.el7_3.4.noarch Fri Nov 4 05:14:01 2016 --- Additional comment from Ryan Hallisey on 2016-11-10 16:32:49 EST --- Lon, when will openstack-selinux be tagged around to all the releases if it hasn't already? --- Additional comment from Freddy Wissing on 2016-11-15 11:44:20 EST --- MOC is also requesting an ETA for this in reference to 01736330. --- Additional comment from errata-xmlrpc on 2016-12-14 08:41:42 EST --- Bug report changed to RELEASE_PENDING status by Errata System. Advisory RHEA-2016:23819-05 has been changed to PUSH_READY status. https://errata.devel.redhat.com/advisory/23819 --- Additional comment from errata-xmlrpc on 2016-12-14 11:02:28 EST --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html --- Additional comment from ben haubeck on 2017-05-24 10:23:27 EDT --- re-open it. we had a PoC with RHOSP11 these days and hit this bug again. Joachim von Thadden has solved it by: - login on a compute - yum -y install setroubleshoot - setenforce 0 - start the VMs - ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd - semodule -i my-virtlogd.pp - setenfoce 1 distribute my-virtlogd.pp to all nodes --- Additional comment from Mike Burns on 2017-05-26 09:50:18 EDT --- Please do not re-open bugs that are closed Errata. If you are encountering this issue, please clone the bug or file a new bug against the appropriate release. This bug was for OSP 10. You reopened but are testing OSP 11. Please provide details in the new bug: * What version of of packages are being used, especially openstack-selinux and packstack if that is what you used. Please also include the RHEL selinux package versions. * full audit.log with the system in permissive. Note: audit2allow will generally provide *a* solution, but not necessarily the *right* solution. We need the audit.log with details to determine that.
FYI this is not the same issue. Previously, the problem was virtlogd_t could not manage nova_var_lib_t files. This issue seems peculiar to log rotation for console.log on instances. When this fails, what does 'ls -lZ /var/lib/nova/instances/98c175ec-6be0-4de0-88c9-e54774fed778' say?
Basically, it sounds like virtlogd is set up to do log rotation for instances, but doesn't set the correct context (it remains in the svirt context), so when we go to boot things, the label is incorrect on disk, meaning that when Nova launches things, SELinux denies the rename/unlink.
Need: * ls -lZ /var/lib/nova/instances/98c175ec-6be0-4de0-88c9-e54774fed778 * /etc/libvirt/virtlogd.conf from the affected machine(s) I'm not sure off-hand how to fix this.
Eduardo pointed out that this bug was also posed to the ML in 2014: https://www.redhat.com/archives/libvir-list/2014-November/msg00500.html In my digging, I also found that Atlassian Eucalyptus (another cloud system) hits this, too: https://eucalyptus.atlassian.net/browse/EUCA-13447 However, I believe this bug is in libvirtd/virtlogd, and not in SELinux policies.
You may want to have a look at: https://bugzilla.redhat.com/show_bug.cgi?id=1501957 I hit this issue with OSP 10 and OSP 8, both after an upgrade to RHEL 7.4 I'm not setting them as a duplicate of this bug yet. Feel free to do so, though.
I think 1501957 is a duplicate of this. If this is new in RHEL 7.4, then we may want to note that it is a regression.
Reported here, too: https://bugs.launchpad.net/kolla-ansible/+bug/1668654
Please make sure that the directory with console.log files are correctly labelled with virt_log_d selinux label. This requirement is described in the following chapter of virtualization deployment guide. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-manipulating_the_domain_xml-devices#sect-Devices-Host_physical_machine_interface *** This bug has been marked as a duplicate of bug 1371125 ***
I am moving this issue back to Open Stack product as there is nothing libvirt can if nova puts log file in a custom directory. It is mandatory that the directory is labeled with virt_log_d.
From the Eucalyptus bug, seems that this bugfix is what changed the behavior in 7.4 vs 7.3: https://bugzilla.redhat.com/show_bug.cgi?id=1420205 It also appears one can work around the behavior change by adding log rotation capability to selinux policies: allow virtlogd_t self:capability dac_override; create_files_pattern(virtlogd_t, virt_image_t, virt_log_t) delete_files_pattern(virtlogd_t, virt_image_t, virt_log_t) rename_files_pattern(virtlogd_t, virt_image_t, virt_log_t) delete_files_pattern(virtlogd_t, virt_image_t, svirt_image_t) rename_files_pattern(virtlogd_t, virt_image_t, svirt_image_t) filetrans_pattern(virtlogd_t, virt_image_t, virt_log_t, file, "console.log") This makes sense: https://danwalsh.livejournal.com/73796.html
https://github.com/redhat-openstack/openstack-selinux/commit/ce3cff747f48594b21ebced8e81842db30f87aeb https://github.com/redhat-openstack/openstack-selinux/commit/974060c318399a877ba1dc1de309f3964b7c4dc2 https://github.com/redhat-openstack/openstack-selinux/commit/904af4727741e38887d57072f42c60383da16f13
*** Bug 1501957 has been marked as a duplicate of this bug. ***
http://people.redhat.com/lhh/openstack-selinux-0.8.11-1.el7ost.src.rpm http://people.redhat.com/lhh/openstack-selinux-0.8.11-1.el7ost.noarch.rpm http://people.redhat.com/lhh/openstack-selinux-test-0.8.11-1.el7ost.noarch.rpm
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0365