Bug 1431556 - RHOS 11 OVS DPDK unable to boot an instance
Summary: RHOS 11 OVS DPDK unable to boot an instance
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Lon Hohberger
QA Contact: Maxim Babushkin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-13 09:36 UTC by Maxim Babushkin
Modified: 2017-07-24 17:10 UTC (History)
31 users (show)

Fixed In Version: openstack-selinux-0.8.6-1.el7ost
Doc Type: Release Note
Doc Text:
Because SELinux policies concerning launching instances with DPDK enabled are incomplete, launching instances using DPDK with SELinux in enforcing mode will cause the launch to fail and AVC denials will appear in /var/log/audit/audit.log* concerning openvswitch and svirt. As a workaround, set SELinux to permissive on each compute node where DPDK is utilized as documented in section 4.4.1.2 here: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Security-Enhanced_Linux-Working_with_SELinux-Changing_SELinux_Modes.html#sect-Security-Enhanced_Linux-Enabling_and_Disabling_SELinux-Disabling_SELinux This will allow DPDK-enabled virtual machines to launch. This is a workaround and is expected to be temporary while the issue is investigated further.
Clone Of:
: 1447112 (view as bug list)
Environment:
Last Closed: 2017-05-17 20:07:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
First-boot (8.15 KB, text/plain)
2017-03-15 08:37 UTC, Karthik Sundaravel
no flags Details
post-install (1.80 KB, text/plain)
2017-03-15 08:39 UTC, Karthik Sundaravel
no flags Details
First-boot (8.15 KB, text/plain)
2017-03-15 12:21 UTC, Karthik Sundaravel
no flags Details
Audit after fix from compute 06-04-17 (226.59 KB, text/plain)
2017-04-06 15:55 UTC, Yariv
no flags Details
first-boot.yaml updated version (10.47 KB, text/plain)
2017-04-19 10:40 UTC, Maxim Babushkin
no flags Details
post-install.yaml updated version (1.88 KB, text/plain)
2017-04-19 11:01 UTC, Maxim Babushkin
no flags Details
permissive creation and teardown (with directories manually chmodded on controller) (401.49 KB, text/plain)
2017-05-01 17:20 UTC, Lon Hohberger
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1675690 0 None None None 2017-03-24 09:50:51 UTC
OpenStack gerrit 449530 0 'None' MERGED vhostuser socket dir shall be created for vhostuserclient mode 2020-08-08 18:08:34 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Maxim Babushkin 2017-03-13 09:36:27 UTC
Description of problem:
RHOS 11 OVS DPDK setup unable to boot dpdk instance.

Version-Release number of selected component (if applicable):
RHOS 11
2017-03-08.3 puddle

[root@compute-0 ~]# rpm -qa |grep openvswitch
openvswitch-ovn-central-2.6.1-8.git20161206.el7fdb.x86_64
openstack-neutron-openvswitch-10.0.0-4.el7ost.noarch
openvswitch-2.6.1-8.git20161206.el7fdb.x86_64
openvswitch-ovn-common-2.6.1-8.git20161206.el7fdb.x86_64
python-openvswitch-2.6.1-8.git20161206.el7fdb.noarch
openvswitch-ovn-host-2.6.1-8.git20161206.el7fdb.x86_64

How reproducible:
Deploy RHOS11 with OVS DPDK
Try to boot a dpdk instance.

Actual results:
The instance gets ERROR state on boot.

Expected results:
The instance should boot successfully.

Additional info:
Instance error:
{u'message': u'internal error: process exited while connecting to monitor: t=1 -vnc 10.10.111.107:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on\n2017-03-13T08:40:47.885506Z qemu-kvm:', u'code': 500, u'details': u'  File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1780, in _do_build_and_run_instance\n    filter_properties)\n  File "/usr/lib/python2.7/site-
packages/nova/compute/manager.py", line 2016, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'created':
u'2017-03-13T08:40:49Z'}


Libvirt instance log error:
2017-03-13T08:40:47.885506Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu003c709e-28,server: Failed to bind socket to /var/run/openvswitch/vhu003c709e-28: Permission denied
2017-03-13 08:40:47.922+0000: shutting down


In RHOS10 the error described above has a workaround that applied by providing the first-boot and post-install files
and modifies the openvswitch socket permissions - "/var/run/openvswitch" to root:qemu ownership.

The RHOS11 deployed with the same workaround files, but was unable to boot an instance.
If setting the ownership of /var/run/openvswitch folder to qemu user, the instance is able to boot.

Comment 3 Karthik Sundaravel 2017-03-13 13:42:45 UTC
2 issues
1. The first-boot and post-install scripts for permission changes are not required, since OSP11 has support for DPDKVhostuserClient mode
2. Log snippet from nova-compute.log
2017-03-13 12:12:11.222 20138 ERROR nova.compute.manager [instance: ad1a0432-93ff-4a03-af93-71dfab4a7639] libvirtError: internal error: process exited while connecting to monitor: 2017-03-13T12:12:11.065953Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhubc36c373-e1,server: Failed to bind socket to /var/run/openvswitch/vhubc36c373-e1: Permission denied

We changed the vhostuser_socket_dir in /etc/neutron/plugins/ml2/openvswitch_agent.ini as 
#vhostuser_socket_dir = /var/run/openvswitch
vhostuser_socket_dir = /tmp

After this change, we are able to boot the instance. 

For DPDKvhostuserClient mode, is TripleO expected to create a directory and change the owner as qemu ? Need suggestions on addressing this issue properly.

Comment 4 Aaron Conole 2017-03-14 13:40:23 UTC
For client mode, the permissions are set by the server side.  In this case, the server side would be qemu.  Thus, when provisioning qemu/libvirt you'll need to make sure that the socket it creates has the correct permissions.

Conversely, server mode means that Open vSwitch would be in charge of setting the permissions, but it does not have that capability.

I'm not sure what the vhostuser_socket_dir variable you are setting will influence.

I hope this answers your question.

Comment 5 Karthik Sundaravel 2017-03-14 14:23:36 UTC
Thanks Aaron.  

Terry,
IIUC, the issue is that the permissions are not set correctly for vhostuser_socket_dir in client mode. 

I am not clear which component is expected to set the permissions. Can you please provide your inputs

Comment 6 Maxim Babushkin 2017-03-14 15:45:49 UTC
During the RHOS10 DPDK deployment, we have modified the ownership of the openvswitch socket to root:qemu by using the post-install scripts.

What was changed in 2.6 that brakes the workaround?

If I set the user ownership to qemu as well, I'm able to boot the instance.

I don't think changing the vhostuser_socket_dir location is a right thing to do.

Comment 7 Flavio Leitner 2017-03-14 16:20:20 UTC
(In reply to Maxim Babushkin from comment #6)
> What was changed in 2.6 that brakes the workaround?

It's not clear to me if you are talking about the server to client mode change or if you are saying that the workaround doesn't work anymore with server mode used by OSP10.

Comment 8 Terry Wilson 2017-03-14 16:24:04 UTC
This is the first I've really even looked at dpdkvhostuserclient mode, so you might want to keep that in mind when reading my answer.

From the reading I just did and reading the comments above, it seems reasonable to me for OOO to create the vhostuser_socket_dir w/ appropriate permissions and stop doing the various permissions hacks if using dpdkvhostuserclient. I'm not sure why any of that matters if the permissions truly are set root:qemu g+w on /var/run/openvswitch, though.

Comment 9 Flavio Leitner 2017-03-14 17:37:11 UTC
(In reply to Terry Wilson from comment #8)
In client mode qemu creates the socket and maybe it has a different permission?

I can't see that from sosreport because there is was no socket/vhu port when it was captured.  I don't see any SELinux issue too.

Comment 10 Karthik Sundaravel 2017-03-15 08:37:15 UTC
Created attachment 1263219 [details]
First-boot

Comment 11 Karthik Sundaravel 2017-03-15 08:39:11 UTC
Created attachment 1263220 [details]
post-install

Maxim,
 while we find the proper solution to address the permission issue, inorder to unblock the testing i think we shall modify the first-boot scripts and proceed.

In network-environment.yaml, please change 
NeutronVhostuserSocketDir: "/usr/local/openvswitch" 

Also please use the attached first-boot and post-install scripts.
With these changes please check if it helps to proceed with testing.

Comment 12 Karthik Sundaravel 2017-03-15 12:21:59 UTC
Created attachment 1263319 [details]
First-boot

As suggested by Maxim, modified the permissions for vhostuser_socket_dir as qemu:qemu g+w. 

Terry,
With this permissions, it works.  Is this the right permission for vhostuser_socket_dir in client mode ?

Comment 13 Assaf Muller 2017-03-15 12:50:06 UTC
Terry please see comment 12.

Comment 14 Karthik Sundaravel 2017-03-15 12:54:52 UTC
Amnon,

If DpdkVhostuserClient mode is supported by openvswitch,  we are getting a permission issue while creating the guests (sosreport attached). We would like to understand 
1. which component is expected to set the permissions for vhostuser_socket_dir
2. If the installer needs to do this permission setting, what should be the permissions vhostuser_socket_dir in client mode (OVS)  ?

Comment 15 Terry Wilson 2017-03-15 15:51:21 UTC
Karthik, whatever directory is used would need write access by qemu. It seems like using a directory other than /var/run/openvswitch would be ideal in client mode since that directory is created by openvswitch. I think OOO should either create a directory with the appropriate permissions or find an existing one that works. Maybe someone from libvirt/qemu could suggest a place. It seems possible that using /var/lib/libvirt/qemu would work, but you'd need to double-check with someone on the qemu team. Messing with the permissions of /var/run/openvswitch was always a hack.

Comment 16 Amnon Ilan 2017-03-15 18:00:40 UTC
Dan, Michal, can you comment on this bz?
I thought it was already resolved, but maybe it was only resolved 
for server mode, and with the client mode it requires some other way?

Comment 18 Terry Wilson 2017-03-17 05:47:14 UTC
I would like to see whether just switching vhostuser_socket_dir to /var/lib/libvirt/qemu would work since it is owned qemu:qemu already. It would be a simple solution.

Comment 19 Maxim Babushkin 2017-03-17 22:00:59 UTC
Hi Terry,

Thanks for the location suggestion.

I have verified the deployment and it works lie a charm.
No need any permission workarounds.
Just changing the NeutronVhostuserSocketDir to "/var/lib/libvirt/qemu" within network-environment file did the trick.

I sure we could use it in RHOS 10 OVS DPDK deployments as well.

Comment 20 Michal Privoznik 2017-03-18 06:19:35 UTC
(In reply to Terry Wilson from comment #18)
> I would like to see whether just switching vhostuser_socket_dir to
> /var/lib/libvirt/qemu would work since it is owned qemu:qemu already. It
> would be a simple solution.

This is a private libvirt directory. This is just a temporary workaround (that I'm not a big fan of either). What the problem here is, is that qemu creates a socket. Thus libvirt has no option, no action to take. It can't chown the parent directory - other domains (with different uid:gid) might use the directory as well. So in here, we just surrender and leave it up to system admin to create a directory with proper security labels and we make our promise to no touch them (the labels I mean). Ideal solution would be that it is libvirt who creates the socket (among with proper security labels), and then just pass it to qemu to use. But we are long way from there.

IOW, I'd suggest to create your own directory and set proper owner on it instead of misusing a libvirt's internal directory.

Comment 21 Karthik Sundaravel 2017-03-20 11:37:08 UTC
(In reply to Michal Privoznik from comment #20)
Is it ok to use /usr/local/openvswitch with permissions qemu:qemu g+w ?

Comment 22 Terry Wilson 2017-03-20 23:48:18 UTC
Since qemu is in charge of creating the sockets, it seems weird to me to put the sockets in a directory named "openvswitch" that is owned by qemu. It also seems a little weird for /usr/local to be used for storing data for programs that are installed under /usr.

What are the permissions on the sockets created by qemu? If they are restrictive enough, could /tmp be used much like pgsql sockets, etc.? In any case, we know what works so someone needs to decide.

Comment 23 Yariv 2017-03-21 16:09:27 UTC
Please decide on agreed solution and document in the BZ
Thanks

Comment 24 Karthik Sundaravel 2017-03-22 07:52:17 UTC
Terry, agreed with /usr/local.
I think we shall go ahead with /var/lib/vhost_sockets. 
/var/lib is meant for dynamic data libraries and files. Does it make sense to use this.

Comment 27 Vijay Chundury 2017-03-27 11:28:53 UTC
As mentioned by Anjali, the testing requirements have been unblocked (the path mentioned by Karthik has been included as a work-around in the first boot.

The upstream patch for the right fix is been tracked at
https://review.openstack.org/#/c/449530/

Comment 28 Maxim Babushkin 2017-03-27 11:54:08 UTC
I redeployed the environment with suggested workaround by relocating the ovs socket directory to /var/lib/vhost_sockets + setting the qemu ownership on the directory.

I'm unable to boot an instance because of selinux permissions.

type=AVC msg=audit(1490614953.792:294): avc:  denied  { write } for  pid=21232 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=29360576 scontext=system_u:system_r:svirt_t:s0:c603,c627 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=dir

If we agree regarding the socket directory location, we should get a support from selinux team for the proper policy.

Comment 29 Lon Hohberger 2017-03-29 12:56:40 UTC
The label of /var/lib/vhost_sockets[] needs to be set to something that svirt can write to.

Comment 30 Yariv 2017-03-29 13:37:03 UTC
Hi Lukas

Can you reply to https://bugzilla.redhat.com/show_bug.cgi?id=1431556#c28

Thanks

Comment 32 Lukas Vrabec 2017-03-30 15:41:57 UTC
Hi Guys, 
Sorry for late reply, 

Could you try it with following workaround:
# semanage fcontext -a -t virt_cache_t /var/lib/vhosts_sockets
# restorecon -Rv /var/lib/vhosts_sockets

Thanks.

Comment 33 Lon Hohberger 2017-03-30 20:17:29 UTC
If that works, we can certainly add that to local_settings in openstack-selinux until the fix is more permanent in RHEL 7.4

Comment 34 Karthik Sundaravel 2017-03-31 08:31:37 UTC
(In reply to Lukas Vrabec from comment #32)
> Hi Guys, 
> Sorry for late reply, 
> 
> Could you try it with following workaround:
> # semanage fcontext -a -t virt_cache_t /var/lib/vhosts_sockets
> # restorecon -Rv /var/lib/vhosts_sockets
> 
> Thanks.

Correcting typo.

# semanage fcontext -a -t virt_cache_t /var/lib/vhost_sockets
# restorecon -Rv /var/lib/vhost_sockets

Comment 36 Eyal Dannon 2017-04-03 12:34:25 UTC
Hi,

I've verified it
[root@compute-0 ~]# ll -Z /var/lib/vhost_sockets/
srwxrwxr-x. qemu qemu system_u:object_r:virt_cache_t:s0 vhu20044fc8-82
srwxrwxr-x. qemu qemu system_u:object_r:virt_cache_t:s0 vhu67f99767-e0
[root@compute-0 ~]# ll -dZ /var/lib/vhost_sockets/
drwxr-xr-x. qemu qemu system_u:object_r:virt_cache_t:s0 /var/lib/vhost_sockets/

[root@compute-0 ~]# getenforce 
Permissive

Using the following packages:

[root@compute-0 ~]# rpm -qa | grep selinux
openstack-selinux-0.8.5-3.el7ost.noarch
selinux-policy-3.13.1-102.el7_3.16.noarch
libselinux-ruby-2.5-6.el7.x86_64
ceph-selinux-10.2.5-37.el7cp.x86_64
selinux-policy-targeted-3.13.1-102.el7_3.16.noarch
libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
container-selinux-2.9-4.el7.noarch
libselinux-python-2.5-6.el7.x86_64
[root@compute-0 ~]# rpm -qa | grep openvswitch
openvswitch-ovn-common-2.6.1-10.git20161206.el7fdp.x86_64
python-openvswitch-2.6.1-10.git20161206.el7fdp.noarch
openvswitch-ovn-central-2.6.1-10.git20161206.el7fdp.x86_64
openstack-neutron-openvswitch-10.0.0-11.el7ost.noarch
openvswitch-ovn-host-2.6.1-10.git20161206.el7fdp.x86_64
openvswitch-2.6.1-10.git20161206.el7fdp.x86_64

Let me know if any further information is needed.
Thanks,
Eyal

Comment 37 Lukas Vrabec 2017-04-03 12:38:34 UTC
Eyal, 
Could you also attach output of:

#ausearch -m AVC -ts today 

Thanks.

Comment 38 Eyal Dannon 2017-04-03 12:45:25 UTC
Sure,
[root@compute-0 ~]# ausearch -m AVC -ts today 
----
time->Mon Apr  3 11:18:48 2017
type=SYSCALL msg=audit(1491218328.362:92): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=24daed0 a2=90800 a3=0 items=0 ppid=1 pid=14511 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables.init" exe="/usr/bin/bash" subj=system_u:system_r:iptables_t:s0 key=(null)
type=AVC msg=audit(1491218328.362:92): avc:  denied  { read } for  pid=14511 comm="ip6tables.init" name="modprobe.d" dev="sda2" ino=8409822 scontext=system_u:system_r:iptables_t:s0 tcontext=unconfined_u:object_r:modules_conf_t:s0 tclass=dir
----
time->Mon Apr  3 11:31:10 2017
type=SYSCALL msg=audit(1491219070.160:177): arch=c000003e syscall=2 success=no exit=-13 a0=7fa39b23f0b0 a1=0 a2=1b6 a3=24 items=0 ppid=19818 pid=19824 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vsctl" exe="/usr/bin/ovs-vsctl" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491219070.160:177): avc:  denied  { search } for  pid=19824 comm="ovs-vsctl" name="19818" dev="proc" ino=93443 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir
----
time->Mon Apr  3 11:31:11 2017
type=SYSCALL msg=audit(1491219071.918:180): arch=c000003e syscall=2 success=no exit=-13 a0=7f24ab4d9270 a1=0 a2=1b6 a3=24 items=0 ppid=19818 pid=20000 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vsctl" exe="/usr/bin/ovs-vsctl" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491219071.918:180): avc:  denied  { search } for  pid=20000 comm="ovs-vsctl" name="19818" dev="proc" ino=93443 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir
----
time->Mon Apr  3 11:49:58 2017
type=SYSCALL msg=audit(1491220198.013:273): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffc63575130 a2=6e a3=25 items=0 ppid=1 pid=20761 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c19,c410 key=(null)
type=AVC msg=audit(1491220198.013:273): avc:  denied  { create } for  pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1491220198.013:273): avc:  denied  { add_name } for  pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1491220198.013:273): avc:  denied  { write } for  pid=20761 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=50331712 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
----
time->Mon Apr  3 11:49:58 2017
type=SYSCALL msg=audit(1491220198.134:274): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff26168e9a0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491220198.134:274): avc:  denied  { connectto } for  pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu80c4beaa-ce" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c19,c410 tclass=unix_stream_socket
type=AVC msg=audit(1491220198.134:274): avc:  denied  { write } for  pid=20752 comm="ovs-vswitchd" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1491220198.134:274): avc:  denied  { dac_override } for  pid=20752 comm="ovs-vswitchd" capability=1  scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability
----
time->Mon Apr  3 11:50:21 2017
type=SYSCALL msg=audit(1491220221.758:287): arch=c000003e syscall=87 success=yes exit=0 a0=7f00e1cb45b0 a1=6 a2=0 a3=0 items=0 ppid=1 pid=20761 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c19,c410 key=(null)
type=AVC msg=audit(1491220221.758:287): avc:  denied  { unlink } for  pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1491220221.758:287): avc:  denied  { remove_name } for  pid=20761 comm="qemu-kvm" name="vhu80c4beaa-ce" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c19,c410 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
----
time->Mon Apr  3 11:56:08 2017
type=SYSCALL msg=audit(1491220568.958:313): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffccccf5820 a2=6e a3=25 items=0 ppid=1 pid=20969 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c731,c817 key=(null)
type=AVC msg=audit(1491220568.958:313): avc:  denied  { create } for  pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1491220568.958:313): avc:  denied  { add_name } for  pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1491220568.958:313): avc:  denied  { write } for  pid=20969 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=50331712 scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
----
time->Mon Apr  3 11:56:09 2017
type=SYSCALL msg=audit(1491220569.163:314): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff26168dec0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491220569.163:314): avc:  denied  { connectto } for  pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu045c2ee5-01" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c731,c817 tclass=unix_stream_socket
type=AVC msg=audit(1491220569.163:314): avc:  denied  { write } for  pid=20752 comm="ovs-vswitchd" name="vhu045c2ee5-01" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1491220569.163:314): avc:  denied  { dac_override } for  pid=20752 comm="ovs-vswitchd" capability=1  scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability
----
time->Mon Apr  3 12:29:11 2017
type=SYSCALL msg=audit(1491222551.581:340): arch=c000003e syscall=87 success=yes exit=0 a0=7fcbc95e85b0 a1=6 a2=0 a3=0 items=0 ppid=1 pid=20969 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c731,c817 key=(null)
type=AVC msg=audit(1491222551.581:340): avc:  denied  { unlink } for  pid=20969 comm="qemu-kvm" name="vhu045c2ee5-01" dev="sda2" ino=50331720 scontext=system_u:system_r:svirt_t:s0:c731,c817 tcontext=system_u:object_r:var_lib_t:s0 tclass=sock_file
----
time->Mon Apr  3 12:29:56 2017
type=SYSCALL msg=audit(1491222596.903:361): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7ffff1877980 a2=6e a3=25 items=0 ppid=1 pid=21911 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c345,c392 key=(null)
type=AVC msg=audit(1491222596.903:361): avc:  denied  { create } for  pid=21911 comm="qemu-kvm" name="vhu67f99767-e0" scontext=system_u:system_r:svirt_t:s0:c345,c392 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file
----
time->Mon Apr  3 12:29:57 2017
type=SYSCALL msg=audit(1491222597.314:362): arch=c000003e syscall=42 success=yes exit=0 a0=80 a1=7ff2616a2ad0 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491222597.314:362): avc:  denied  { connectto } for  pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu67f99767-e0" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c345,c392 tclass=unix_stream_socket
type=AVC msg=audit(1491222597.314:362): avc:  denied  { write } for  pid=20752 comm="ovs-vswitchd" name="vhu67f99767-e0" dev="sda2" ino=50331720 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file
type=AVC msg=audit(1491222597.314:362): avc:  denied  { dac_override } for  pid=20752 comm="ovs-vswitchd" capability=1  scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability
----
time->Mon Apr  3 12:30:28 2017
type=SYSCALL msg=audit(1491222628.438:385): arch=c000003e syscall=49 success=yes exit=0 a0=c a1=7fff2bb54560 a2=6e a3=25 items=0 ppid=1 pid=38750 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c308,c1010 key=(null)
type=AVC msg=audit(1491222628.438:385): avc:  denied  { create } for  pid=38750 comm="qemu-kvm" name="vhu20044fc8-82" scontext=system_u:system_r:svirt_t:s0:c308,c1010 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file
----
time->Mon Apr  3 12:30:29 2017
type=SYSCALL msg=audit(1491222629.316:386): arch=c000003e syscall=42 success=yes exit=0 a0=83 a1=7ff2616a3f60 a2=6e a3=8 items=0 ppid=1 pid=20752 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1491222629.316:386): avc:  denied  { connectto } for  pid=20752 comm="ovs-vswitchd" path="/var/lib/vhost_sockets/vhu20044fc8-82" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:svirt_t:s0:c308,c1010 tclass=unix_stream_socket
type=AVC msg=audit(1491222629.316:386): avc:  denied  { write } for  pid=20752 comm="ovs-vswitchd" name="vhu20044fc8-82" dev="sda2" ino=50331733 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=sock_file
type=AVC msg=audit(1491222629.316:386): avc:  denied  { dac_override } for  pid=20752 comm="ovs-vswitchd" capability=1  scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability

Comment 39 Lon Hohberger 2017-04-04 13:14:16 UTC
https://github.com/redhat-openstack/openstack-selinux/pull/8

Merged changeset #8 from Lukas Vrabec

Comment 41 Yariv 2017-04-06 15:21:18 UTC
HI 

It seems that we have the latest paddle

openstack-selinux-0.8.5-4.el7ost.noarch

It seems that AVC does not contain the right rule
We could not deploy vm

Only when se-linux permissive

Comment 42 Lon Hohberger 2017-04-06 15:37:20 UTC
To fix this, can you attach the audit.log from permissive mode?

Comment 43 Yariv 2017-04-06 15:55:58 UTC
Created attachment 1269422 [details]
Audit after fix from compute 06-04-17

Comment 44 Lon Hohberger 2017-04-07 13:54:03 UTC
#============= iptables_t ==============
allow iptables_t modules_conf_t:dir read;

#============= openvswitch_t ==============
allow openvswitch_t neutron_t:dir search;
allow openvswitch_t self:capability net_raw;
allow openvswitch_t self:packet_socket { bind create setopt };

I don't think iptables_t is relevant here. The rest looks like new functionality in openvswitch, except the first part, which is openstack-specific.

All of the rest are okay to add to openstack-selinux.

Comment 48 Lukas Vrabec 2017-04-10 07:23:36 UTC
PR Created: 
https://github.com/redhat-openstack/openstack-selinux/pull/10

Comment 55 Maxim Babushkin 2017-04-13 09:32:51 UTC
Hi Lon,

Installed the openstack-selinux-0.8.5-8 rpm.

Got errors.
Audit log in permissive mode attached with full cycle of instance boot.

Comment 59 Maxim Babushkin 2017-04-19 10:38:38 UTC
Hi Lon, Lukas,

Thank you for the help.
The rpm is working.

Lukas, thank you for the DAC_OVERRIDE article. It helped me to understand what should be added to the deployment.

OVS permission workaround used in RHOS 10 should be implemented in RHOS 11 as well. Updated first-boot attached.


Franck, is the ovs vhu permission workaround acceptable for us in RHOS 11? I thought we will not be using it in ovs 2.6.

Comment 60 Maxim Babushkin 2017-04-19 10:40:21 UTC
Created attachment 1272572 [details]
first-boot.yaml updated version

Comment 61 Maxim Babushkin 2017-04-19 11:01:33 UTC
Created attachment 1272577 [details]
post-install.yaml updated version

Comment 62 Eyal Dannon 2017-04-30 10:42:23 UTC
I've tried to verify with 2017-04-24.2 
which includes : openstack-selinux-0.8.6-2.el7ost.noarch package
using suggested first+post yamls(comment 60-61)

Still could not boot an instance while selinux enabled

type=AVC msg=audit(1493546340.597:99): avc:  denied  { read } for  pid=7919 comm="ip6tables.init" name="modprobe.d" dev="sda2" ino=271 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=dir
type=AVC msg=audit(1493547076.863:184): avc:  denied  { read } for  pid=13862 comm="ovs-vsctl" name="cmdline" dev="proc" ino=107935 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file
type=AVC msg=audit(1493547078.844:189): avc:  denied  { read } for  pid=13898 comm="ovs-vsctl" name="cmdline" dev="proc" ino=107935 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file
type=AVC msg=audit(1493547723.571:251): avc:  denied  { write } for  pid=14518 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c863,c890 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1493547727.435:269): avc:  denied  { write } for  pid=14558 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c346,c994 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1493547730.027:287): avc:  denied  { write } for  pid=14599 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c228,c522 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1493548051.232:314): avc:  denied  { write } for  pid=14695 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c292,c688 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1493548054.321:332): avc:  denied  { write } for  pid=14736 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c65,c1007 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir
type=AVC msg=audit(1493548056.926:350): avc:  denied  { write } for  pid=14777 comm="qemu-kvm" name="vhost_sockets" dev="sda2" ino=14004356 scontext=system_u:system_r:svirt_t:s0:c460,c1006 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir

Comment 64 Lon Hohberger 2017-05-01 16:53:27 UTC
The following error seems to occur even with SELinux disabled; is this potentially a red herring?


2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] Traceback (most recent call last):
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     yield resources
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     block_device_info=block_device_info)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2698, in spawn
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     destroy_disks_on_failure=True)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5112, in _create_domain_and_network
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     destroy_disks_on_failure)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     self.force_reraise()
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     six.reraise(self.type_, self.value, self.tb)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5084, in _create_domain_and_network
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     post_xml_callback=post_xml_callback)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5002, in _create_domain
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     guest.launch(pause=pause)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     self._encoded_xml, errors='ignore')
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     self.force_reraise()
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     six.reraise(self.type_, self.value, self.tb)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     return self._domain.createWithFlags(flags)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     rv = execute(f, *args, **kwargs)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     six.reraise(c, e, tb)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     rv = meth(*args, **kwargs)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320] libvirtError: internal error: process exited while connecting to monitor: 2017-04-30T19:30:55.102672Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu43f166c3-e9,server: Failed to bind socket to /var/run/openvswitch/vhu43f166c3-e9: Permission denied
2017-04-30 19:30:55.605 30855 ERROR nova.compute.manager [instance: 69492f63-a1f2-4dcf-bc7a-09c68ae76320]

Here's my attempt to create servers on the same host:

2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [req-4c5588f2-fc1e-4e27-baec-4f2fa67a72bf 4423f30fd4764b49a5df5b97de52a139 cd143ac333e94ad4b956b7d78ff1652b - - -] [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] Instance failed to spawn
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] Traceback (most recent call last):
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     yield resources
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     block_device_info=block_device_info)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2698, in spawn
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     destroy_disks_on_failure=True)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5112, in _create_domain_and_network
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     destroy_disks_on_failure)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     self.force_reraise()
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     six.reraise(self.type_, self.value, self.tb)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5084, in _create_domain_and_network
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     post_xml_callback=post_xml_callback)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5002, in _create_domain
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     guest.launch(pause=pause)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     self._encoded_xml, errors='ignore')
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     self.force_reraise()
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     six.reraise(self.type_, self.value, self.tb)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     return self._domain.createWithFlags(flags)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     rv = execute(f, *args, **kwargs)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     six.reraise(c, e, tb)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     rv = meth(*args, **kwargs)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51] libvirtError: internal error: qemu unexpectedly closed the monitor: 2017-05-01T16:48:28.307048Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhuce684eb9-15,server: Failed to bind socket to /var/run/openvswitch/vhuce684eb9-15: Permission denied
2017-05-01 16:48:28.745 30855 ERROR nova.compute.manager [instance: 22fac6c6-a8d6-4585-a398-d05a1d0dde51]


This happens even with the entire overcloud with SELinux in permissive.

Comment 65 Lon Hohberger 2017-05-01 17:16:15 UTC
/var/run/openvswitch is trying to be written to by a non-permitted user (mode 755); this is one problem.

There are also some AVCs that can be observed in permissive mode:

Comment 66 Lon Hohberger 2017-05-01 17:20:20 UTC
Created attachment 1275438 [details]
permissive creation and teardown (with directories manually chmodded on controller)

Comment 68 Lon Hohberger 2017-05-01 17:35:12 UTC
With SELinux in permissive mode and having changed /var/run/openvswitch to mode 0777 (a+rw), I was able to launch instances.

It's almost as though something changed locations recently; I'd like to understand that prior to using /var/run/openvswitch.

Comment 69 Lon Hohberger 2017-05-01 17:38:58 UTC
The deployment appears to have changed from using "/var/lib/vhost_sockets" to "/var/run/openvswitch", which is the likely cause for these AVCs.

Comment 70 Lon Hohberger 2017-05-01 17:45:32 UTC
Maxim, do you have any idea why using /var/run/openvswitch for vhost_sockets would be used instead of /var/lib/vhost_sockets as we built code for (and/or do you know what sets NeutronVhostuserSocketDir )?

This should be working if /var/lib/vhost_sockets is the value used for NeutronVhostuserSocketDir, I think...

Comment 71 Lon Hohberger 2017-05-01 17:47:04 UTC
See comment #24

Comment 72 Lon Hohberger 2017-05-01 17:49:51 UTC
It looks like https://review.openstack.org/#/c/449530/3 was never merged.

Comment 79 Maxim Babushkin 2017-05-03 12:39:17 UTC
Hi Lon,

Thanks a lot for the help.

The line that was missing after the vhost socket dir creation:
# restorecon /var/lib/vhost_sockets

Updating the first-boot.yaml file.

Comment 81 Maxim Babushkin 2017-05-03 14:36:00 UTC
The solution has been successfully verified with a fresh deployment.

Comment 85 errata-xmlrpc 2017-05-17 20:07:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.