Description of problem: RHOS 11 OVS DPDK setup unable to boot dpdk instance. Version-Release number of selected component (if applicable): RHOS 11 2017-03-08.3 puddle [root@compute-0 ~]# rpm -qa |grep openvswitch openvswitch-ovn-central-2.6.1-8.git20161206.el7fdb.x86_64 openstack-neutron-openvswitch-10.0.0-4.el7ost.noarch openvswitch-2.6.1-8.git20161206.el7fdb.x86_64 openvswitch-ovn-common-2.6.1-8.git20161206.el7fdb.x86_64 python-openvswitch-2.6.1-8.git20161206.el7fdb.noarch openvswitch-ovn-host-2.6.1-8.git20161206.el7fdb.x86_64 How reproducible: Deploy RHOS11 with OVS DPDK Try to boot a dpdk instance. Actual results: The instance gets ERROR state on boot. Expected results: The instance should boot successfully. Additional info: Instance error: {u'message': u'internal error: process exited while connecting to monitor: t=1 -vnc 10.10.111.107:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on\n2017-03-13T08:40:47.885506Z qemu-kvm:', u'code': 500, u'details': u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1780, in _do_build_and_run_instance\n filter_properties)\n File "/usr/lib/python2.7/site- packages/nova/compute/manager.py", line 2016, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'created': u'2017-03-13T08:40:49Z'} Libvirt instance log error: 2017-03-13T08:40:47.885506Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu003c709e-28,server: Failed to bind socket to /var/run/openvswitch/vhu003c709e-28: Permission denied 2017-03-13 08:40:47.922+0000: shutting down In RHOS10 the error described above has a workaround that applied by providing the first-boot and post-install files and modifies the openvswitch socket permissions - "/var/run/openvswitch" to root:qemu ownership. The RHOS11 deployed with the same workaround files, but was unable to boot an instance. If setting the ownership of /var/run/openvswitch folder to qemu user, the instance is able to boot.
2 issues 1. The first-boot and post-install scripts for permission changes are not required, since OSP11 has support for DPDKVhostuserClient mode 2. Log snippet from nova-compute.log 2017-03-13 12:12:11.222 20138 ERROR nova.compute.manager [instance: ad1a0432-93ff-4a03-af93-71dfab4a7639] libvirtError: internal error: process exited while connecting to monitor: 2017-03-13T12:12:11.065953Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhubc36c373-e1,server: Failed to bind socket to /var/run/openvswitch/vhubc36c373-e1: Permission denied We changed the vhostuser_socket_dir in /etc/neutron/plugins/ml2/openvswitch_agent.ini as #vhostuser_socket_dir = /var/run/openvswitch vhostuser_socket_dir = /tmp After this change, we are able to boot the instance. For DPDKvhostuserClient mode, is TripleO expected to create a directory and change the owner as qemu ? Need suggestions on addressing this issue properly.
For client mode, the permissions are set by the server side. In this case, the server side would be qemu. Thus, when provisioning qemu/libvirt you'll need to make sure that the socket it creates has the correct permissions. Conversely, server mode means that Open vSwitch would be in charge of setting the permissions, but it does not have that capability. I'm not sure what the vhostuser_socket_dir variable you are setting will influence. I hope this answers your question.
Thanks Aaron. Terry, IIUC, the issue is that the permissions are not set correctly for vhostuser_socket_dir in client mode. I am not clear which component is expected to set the permissions. Can you please provide your inputs
During the RHOS10 DPDK deployment, we have modified the ownership of the openvswitch socket to root:qemu by using the post-install scripts. What was changed in 2.6 that brakes the workaround? If I set the user ownership to qemu as well, I'm able to boot the instance. I don't think changing the vhostuser_socket_dir location is a right thing to do.
(In reply to Maxim Babushkin from comment #6) > What was changed in 2.6 that brakes the workaround? It's not clear to me if you are talking about the server to client mode change or if you are saying that the workaround doesn't work anymore with server mode used by OSP10.
This is the first I've really even looked at dpdkvhostuserclient mode, so you might want to keep that in mind when reading my answer. From the reading I just did and reading the comments above, it seems reasonable to me for OOO to create the vhostuser_socket_dir w/ appropriate permissions and stop doing the various permissions hacks if using dpdkvhostuserclient. I'm not sure why any of that matters if the permissions truly are set root:qemu g+w on /var/run/openvswitch, though.
(In reply to Terry Wilson from comment #8) In client mode qemu creates the socket and maybe it has a different permission? I can't see that from sosreport because there is was no socket/vhu port when it was captured. I don't see any SELinux issue too.
Created attachment 1263219 [details] First-boot
Created attachment 1263220 [details] post-install Maxim, while we find the proper solution to address the permission issue, inorder to unblock the testing i think we shall modify the first-boot scripts and proceed. In network-environment.yaml, please change NeutronVhostuserSocketDir: "/usr/local/openvswitch" Also please use the attached first-boot and post-install scripts. With these changes please check if it helps to proceed with testing.
Created attachment 1263319 [details] First-boot As suggested by Maxim, modified the permissions for vhostuser_socket_dir as qemu:qemu g+w. Terry, With this permissions, it works. Is this the right permission for vhostuser_socket_dir in client mode ?
Terry please see comment 12.
Amnon, If DpdkVhostuserClient mode is supported by openvswitch, we are getting a permission issue while creating the guests (sosreport attached). We would like to understand 1. which component is expected to set the permissions for vhostuser_socket_dir 2. If the installer needs to do this permission setting, what should be the permissions vhostuser_socket_dir in client mode (OVS) ?
Karthik, whatever directory is used would need write access by qemu. It seems like using a directory other than /var/run/openvswitch would be ideal in client mode since that directory is created by openvswitch. I think OOO should either create a directory with the appropriate permissions or find an existing one that works. Maybe someone from libvirt/qemu could suggest a place. It seems possible that using /var/lib/libvirt/qemu would work, but you'd need to double-check with someone on the qemu team. Messing with the permissions of /var/run/openvswitch was always a hack.
Dan, Michal, can you comment on this bz? I thought it was already resolved, but maybe it was only resolved for server mode, and with the client mode it requires some other way?
I would like to see whether just switching vhostuser_socket_dir to /var/lib/libvirt/qemu would work since it is owned qemu:qemu already. It would be a simple solution.
Hi Terry, Thanks for the location suggestion. I have verified the deployment and it works lie a charm. No need any permission workarounds. Just changing the NeutronVhostuserSocketDir to "/var/lib/libvirt/qemu" within network-environment file did the trick. I sure we could use it in RHOS 10 OVS DPDK deployments as well.
(In reply to Terry Wilson from comment #18) > I would like to see whether just switching vhostuser_socket_dir to > /var/lib/libvirt/qemu would work since it is owned qemu:qemu already. It > would be a simple solution. This is a private libvirt directory. This is just a temporary workaround (that I'm not a big fan of either). What the problem here is, is that qemu creates a socket. Thus libvirt has no option, no action to take. It can't chown the parent directory - other domains (with different uid:gid) might use the directory as well. So in here, we just surrender and leave it up to system admin to create a directory with proper security labels and we make our promise to no touch them (the labels I mean). Ideal solution would be that it is libvirt who creates the socket (among with proper security labels), and then just pass it to qemu to use. But we are long way from there. IOW, I'd suggest to create your own directory and set proper owner on it instead of misusing a libvirt's internal directory.
(In reply to Michal Privoznik from comment #20) Is it ok to use /usr/local/openvswitch with permissions qemu:qemu g+w ?
Since qemu is in charge of creating the sockets, it seems weird to me to put the sockets in a directory named "openvswitch" that is owned by qemu. It also seems a little weird for /usr/local to be used for storing data for programs that are installed under /usr. What are the permissions on the sockets created by qemu? If they are restrictive enough, could /tmp be used much like pgsql sockets, etc.? In any case, we know what works so someone needs to decide.
Please decide on agreed solution and document in the BZ Thanks
Terry, agreed with /usr/local. I think we shall go ahead with /var/lib/vhost_sockets. /var/lib is meant for dynamic data libraries and files. Does it make sense to use this.
As mentioned by Anjali, the testing requirements have been unblocked (the path mentioned by Karthik has been included as a work-around in the first boot. The upstream patch for the right fix is been tracked at https://review.openstack.org/#/c/449530/
I redeployed the environment with suggested workaround by relocating the ovs socket directory to /var/lib/vhost_sockets + setting the qemu ownership on the directory. I'm unable to boot an instance because of selinux permissions. type=AVC msg=audit(1490614953.792:294): avc: denied { write } for pid=21232 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=29360576 scontext=system_u:system_r:svirt_t:s0:c603,c627 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=dir If we agree regarding the socket directory location, we should get a support from selinux team for the proper policy.
The label of /var/lib/vhost_sockets[] needs to be set to something that svirt can write to.
Hi Lukas Can you reply to https://bugzilla.redhat.com/show_bug.cgi?id=1431556#c28 Thanks
Hi Guys, Sorry for late reply, Could you try it with following workaround: # semanage fcontext -a -t virt_cache_t /var/lib/vhosts_sockets # restorecon -Rv /var/lib/vhosts_sockets Thanks.
If that works, we can certainly add that to local_settings in openstack-selinux until the fix is more permanent in RHEL 7.4
(In reply to Lukas Vrabec from comment #32) > Hi Guys, > Sorry for late reply, > > Could you try it with following workaround: > # semanage fcontext -a -t virt_cache_t /var/lib/vhosts_sockets > # restorecon -Rv /var/lib/vhosts_sockets > > Thanks. Correcting typo. # semanage fcontext -a -t virt_cache_t /var/lib/vhost_sockets # restorecon -Rv /var/lib/vhost_sockets
Hi, I've verified it [root@compute-0 ~]# ll -Z /var/lib/vhost_sockets/ srwxrwxr-x. qemu qemu system_u:object_r:virt_cache_t:s0 vhu20044fc8-82 srwxrwxr-x. qemu qemu system_u:object_r:virt_cache_t:s0 vhu67f99767-e0 [root@compute-0 ~]# ll -dZ /var/lib/vhost_sockets/ drwxr-xr-x. qemu qemu system_u:object_r:virt_cache_t:s0 /var/lib/vhost_sockets/ [root@compute-0 ~]# getenforce Permissive Using the following packages: [root@compute-0 ~]# rpm -qa | grep selinux openstack-selinux-0.8.5-3.el7ost.noarch selinux-policy-3.13.1-102.el7_3.16.noarch libselinux-ruby-2.5-6.el7.x86_64 ceph-selinux-10.2.5-37.el7cp.x86_64 selinux-policy-targeted-3.13.1-102.el7_3.16.noarch libselinux-2.5-6.el7.x86_64 libselinux-utils-2.5-6.el7.x86_64 container-selinux-2.9-4.el7.noarch libselinux-python-2.5-6.el7.x86_64 [root@compute-0 ~]# rpm -qa | grep openvswitch openvswitch-ovn-common-2.6.1-10.git20161206.el7fdp.x86_64 python-openvswitch-2.6.1-10.git20161206.el7fdp.noarch openvswitch-ovn-central-2.6.1-10.git20161206.el7fdp.x86_64 openstack-neutron-openvswitch-10.0.0-11.el7ost.noarch openvswitch-ovn-host-2.6.1-10.git20161206.el7fdp.x86_64 openvswitch-2.6.1-10.git20161206.el7fdp.x86_64 Let me know if any further information is needed. Thanks, Eyal
Eyal, Could you also attach output of: #ausearch -m AVC -ts today Thanks.
Sure, [root@compute-0 ~]# ausearch -m AVC -ts today ---- time->Mon Apr 3 11:18:48 2017 type=SYSCALL msg=audit(1491218328.362:92): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=24daed0 a2=90800 a3=0 items=0 ppid=1 pid=14511 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables.init" exe="/usr/bin/bash" subj=system_u:system_r:iptables_t:s0 key=(null) type=AVC msg=audit(1491218328.362:92): avc: denied { read } for pid=14511 comm="ip6tables.init" name="modprobe.d" dev="sda2" ino=8409822 scontext=system_u:system_r:iptables_t:s0 tcontext=unconfined_u:object_r:modules_conf_t:s0 tclass=dir ---- time->Mon Apr 3 11:31:10 2017 type=SYSCALL msg=audit(1491219070.160:177): arch=c000003e syscall=2 success=no exit=-13 a0=7fa39b23f0b0 a1=0 a2=1b6 a3=24 items=0 ppid=19818 pid=19824 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vsctl" exe="/usr/bin/ovs-vsctl" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491219070.160:177): avc: denied { search } for pid=19824 comm="ovs-vsctl" name="19818" dev="proc" ino=93443 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir ---- time->Mon Apr 3 11:31:11 2017 type=SYSCALL msg=audit(1491219071.918:180): arch=c000003e syscall=2 success=no exit=-13 a0=7f24ab4d9270 a1=0 a2=1b6 a3=24 items=0 ppid=19818 pid=20000 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vsctl" exe="/usr/bin/ovs-vsctl" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491219071.918:180): avc: denied { search } for pid=20000 comm="ovs-vsctl" name="19818" dev="proc" ino=93443 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir ---- time->Mon Apr 3 11:49:58 2017 type=SYSCALL msg=audit(1491220198.013:273): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffc63575130 a2=6e a3=25 items=0 ppid=1 pid=20761 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c19,c410 key=(null) type=AVC msg=audit(1491220198.013:273): avc: denied { create } for pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file type=AVC msg=audit(1491220198.013:273): avc: denied { add_name } for pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1491220198.013:273): avc: denied { write } for pid=20761 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=50331712 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir ---- time->Mon Apr 3 11:49:58 2017 type=SYSCALL msg=audit(1491220198.134:274): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff26168e9a0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491220198.134:274): avc: denied { connectto } for pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu80c4beaa-ce" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c19,c410 tclass=unix_stream_socket type=AVC msg=audit(1491220198.134:274): avc: denied { write } for pid=20752 comm="ovs-vswitchd" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file type=AVC msg=audit(1491220198.134:274): avc: denied { dac_override } for pid=20752 comm="ovs-vswitchd" capability=1 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability ---- time->Mon Apr 3 11:50:21 2017 type=SYSCALL msg=audit(1491220221.758:287): arch=c000003e syscall=87 success=yes exit=0 a0=7f00e1cb45b0 a1=6 a2=0 a3=0 items=0 ppid=1 pid=20761 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c19,c410 key=(null) type=AVC msg=audit(1491220221.758:287): avc: denied { unlink } for pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file type=AVC msg=audit(1491220221.758:287): avc: denied { remove_name } for pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir ---- time->Mon Apr 3 11:56:08 2017 type=SYSCALL msg=audit(1491220568.958:313): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffccccf5820 a2=6e a3=25 items=0 ppid=1 pid=20969 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c731,c817 key=(null) type=AVC msg=audit(1491220568.958:313): avc: denied { create } for pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file type=AVC msg=audit(1491220568.958:313): avc: denied { add_name } for pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1491220568.958:313): avc: denied { write } for pid=20969 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=50331712 scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir ---- time->Mon Apr 3 11:56:09 2017 type=SYSCALL msg=audit(1491220569.163:314): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff26168dec0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491220569.163:314): avc: denied { connectto } for pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu045c2ee5-01" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c731,c817 tclass=unix_stream_socket type=AVC msg=audit(1491220569.163:314): avc: denied { write } for pid=20752 comm="ovs-vswitchd" name="vhu045c2ee5-01" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file type=AVC msg=audit(1491220569.163:314): avc: denied { dac_override } for pid=20752 comm="ovs-vswitchd" capability=1 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability ---- time->Mon Apr 3 12:29:11 2017 type=SYSCALL msg=audit(1491222551.581:340): arch=c000003e syscall=87 success=yes exit=0 a0=7fcbc95e85b0 a1=6 a2=0 a3=0 items=0 ppid=1 pid=20969 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c731,c817 key=(null) type=AVC msg=audit(1491222551.581:340): avc: denied { unlink } for pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file ---- time->Mon Apr 3 12:29:56 2017 type=SYSCALL msg=audit(1491222596.903:361): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffff1877980 a2=6e a3=25 items=0 ppid=1 pid=21911 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c345,c392 key=(null) type=AVC msg=audit(1491222596.903:361): avc: denied { create } for pid=21911 comm="qemu-kvm" name="vhu67f99767-e0" scontext=system_u:system_r:svirt_t:s0:c345,c392 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file ---- time->Mon Apr 3 12:29:57 2017 type=SYSCALL msg=audit(1491222597.314:362): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff2616a2ad0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491222597.314:362): avc: denied { connectto } for pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu67f99767-e0" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c345,c392 tclass=unix_stream_socket type=AVC msg=audit(1491222597.314:362): avc: denied { write } for pid=20752 comm="ovs-vswitchd" name="vhu67f99767-e0" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file type=AVC msg=audit(1491222597.314:362): avc: denied { dac_override } for pid=20752 comm="ovs-vswitchd" capability=1 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability ---- time->Mon Apr 3 12:30:28 2017 type=SYSCALL msg=audit(1491222628.438:385): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7fff2bb54560 a2=6e a3=25 items=0 ppid=1 pid=38750 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c308,c1010 key=(null) type=AVC msg=audit(1491222628.438:385): avc: denied { create } for pid=38750 comm="qemu-kvm" name="vhu20044fc8-82" scontext=system_u:system_r:svirt_t:s0:c308,c1010 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file ---- time->Mon Apr 3 12:30:29 2017 type=SYSCALL msg=audit(1491222629.316:386): arch=c000003e syscall=42 success=yes exit=0 a0=83 a1=7ff2616a3f60 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null) type=AVC msg=audit(1491222629.316:386): avc: denied { connectto } for pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu20044fc8-82" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c308,c1010 tclass=unix_stream_socket type=AVC msg=audit(1491222629.316:386): avc: denied { write } for pid=20752 comm="ovs-vswitchd" name="vhu20044fc8-82" dev="sda2" ino=50331733 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file type=AVC msg=audit(1491222629.316:386): avc: denied { dac_override } for pid=20752 comm="ovs-vswitchd" capability=1 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability
https://github.com/redhat-openstack/openstack-selinux/pull/8 Merged changeset #8 from Lukas Vrabec
HI It seems that we have the latest paddle openstack-selinux-0.8.5-4.el7ost.noarch It seems that AVC does not contain the right rule We could not deploy vm Only when se-linux permissive
To fix this, can you attach the audit.log from permissive mode?
Created attachment 1269422 [details] Audit after fix from compute 06-04-17
#============= iptables_t ============== allow iptables_t modules_conf_t:dir read; #============= openvswitch_t ============== allow openvswitch_t neutron_t:dir search; allow openvswitch_t self:capability net_raw; allow openvswitch_t self:packet_socket { bind create setopt }; I don't think iptables_t is relevant here. The rest looks like new functionality in openvswitch, except the first part, which is openstack-specific. All of the rest are okay to add to openstack-selinux.
https://github.com/redhat-openstack/openstack-selinux/commit/c943f3825dc95fdc3d501ec41a76142c72ce22b6
PR Created: https://github.com/redhat-openstack/openstack-selinux/pull/10
Hi Lon, Installed the openstack-selinux-0.8.5-8 rpm. Got errors. Audit log in permissive mode attached with full cycle of instance boot.
Hi Lon, Lukas, Thank you for the help. The rpm is working. Lukas, thank you for the DAC_OVERRIDE article. It helped me to understand what should be added to the deployment. OVS permission workaround used in RHOS 10 should be implemented in RHOS 11 as well. Updated first-boot attached. Franck, is the ovs vhu permission workaround acceptable for us in RHOS 11? I thought we will not be using it in ovs 2.6.
Created attachment 1272572 [details] first-boot.yaml updated version
Created attachment 1272577 [details] post-install.yaml updated version
I've tried to verify with 2017-04-24.2 which includes : openstack-selinux-0.8.6-2.el7ost.noarch package using suggested first+post yamls(comment 60-61) Still could not boot an instance while selinux enabled type=AVC msg=audit(1493546340.597:99): avc: denied { read } for pid=7919 comm="ip6tables.init" name="modprobe.d" dev="sda2" ino=271 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=dir type=AVC msg=audit(1493547076.863:184): avc: denied { read } for pid=13862 comm="ovs-vsctl" name="cmdline" dev="proc" ino=107935 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file type=AVC msg=audit(1493547078.844:189): avc: denied { read } for pid=13898 comm="ovs-vsctl" name="cmdline" dev="proc" ino=107935 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file type=AVC msg=audit(1493547723.571:251): avc: denied { write } for pid=14518 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c863,c890 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1493547727.435:269): avc: denied { write } for pid=14558 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c346,c994 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1493547730.027:287): avc: denied { write } for pid=14599 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c228,c522 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1493548051.232:314): avc: denied { write } for pid=14695 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c292,c688 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1493548054.321:332): avc: denied { write } for pid=14736 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c65,c1007 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir type=AVC msg=audit(1493548056.926:350): avc: denied { write } for pid=14777 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c460,c1006 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
The following error seems to occur even with SELinux disabled; is this potentially a red herring? 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] Traceback (most recent call last): 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] yield resources 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] block_device_info=block_device_info) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2698, in spawn 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] destroy_disks_on_failure=True) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5112, in _create_domain_and_network 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] destroy_disks_on_failure) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] self.force_reraise() 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] six.reraise(self.type_, self.value, self.tb) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5084, in _create_domain_and_network 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] post_xml_callback=post_xml_callback) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5002, in _create_domain 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] guest.launch(pause=pause) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] self._encoded_xml, errors='ignore') 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] self.force_reraise() 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] six.reraise(self.type_, self.value, self.tb) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] return self._domain.createWithFlags(flags) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] rv = execute(f, *args, **kwargs) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] six.reraise(c, e, tb) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] rv = meth(*args, **kwargs) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] libvirtError: internal error: process exited while connecting to monitor: 2017-04-30T19:30:55.102672Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu43f166c3-e9,server: Failed to bind socket to /var/run/openvswitch/vhu43f166c3-e9: Permission denied 2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] Here's my attempt to create servers on the same host: 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [req-4c5588f2-fc1e-4e27-baec-4f2fa67a72bf 4423f30fd4764b49a5df5b97de52a139 cd143ac333e94ad4b956b7d78ff1652b - - -] [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] Instance failed to spawn 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] Traceback (most recent call last): 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] yield resources 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] block_device_info=block_device_info) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2698, in spawn 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] destroy_disks_on_failure=True) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5112, in _create_domain_and_network 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] destroy_disks_on_failure) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] self.force_reraise() 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] six.reraise(self.type_, self.value, self.tb) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5084, in _create_domain_and_network 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] post_xml_callback=post_xml_callback) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5002, in _create_domain 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] guest.launch(pause=pause) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] self._encoded_xml, errors='ignore') 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] self.force_reraise() 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] six.reraise(self.type_, self.value, self.tb) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] return self._domain.createWithFlags(flags) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] rv = execute(f, *args, **kwargs) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] six.reraise(c, e, tb) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] rv = meth(*args, **kwargs) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] libvirtError: internal error: qemu unexpectedly closed the monitor: 2017-05-01T16:48:28.307048Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhuce684eb9-15,server: Failed to bind socket to /var/run/openvswitch/vhuce684eb9-15: Permission denied 2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] This happens even with the entire overcloud with SELinux in permissive.
/var/run/openvswitch is trying to be written to by a non-permitted user (mode 755); this is one problem. There are also some AVCs that can be observed in permissive mode:
Created attachment 1275438 [details] permissive creation and teardown (with directories manually chmodded on controller)
With SELinux in permissive mode and having changed /var/run/openvswitch to mode 0777 (a+rw), I was able to launch instances. It's almost as though something changed locations recently; I'd like to understand that prior to using /var/run/openvswitch.
The deployment appears to have changed from using "/var/lib/vhost_sockets" to "/var/run/openvswitch", which is the likely cause for these AVCs.
Maxim, do you have any idea why using /var/run/openvswitch for vhost_sockets would be used instead of /var/lib/vhost_sockets as we built code for (and/or do you know what sets NeutronVhostuserSocketDir )? This should be working if /var/lib/vhost_sockets is the value used for NeutronVhostuserSocketDir, I think...
See comment #24
It looks like https://review.openstack.org/#/c/449530/3 was never merged.
Hi Lon, Thanks a lot for the help. The line that was missing after the vhost socket dir creation: # restorecon /var/lib/vhost_sockets Updating the first-boot.yaml file.
The solution has been successfully verified with a fresh deployment.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245