Bug 1012551 - Neutron fails to function with SELinux enabled
Neutron fails to function with SELinux enabled
Status: CLOSED DUPLICATE of bug 1020052
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux (Show other bugs)
4.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 4.0
Assigned To: Lon Hohberger
Ami Jeain
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 11:52 EDT by Ben Nemec
Modified: 2016-04-26 20:14 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-03 16:59:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ben Nemec 2013-09-26 11:52:34 EDT
Description of problem: Neutron doesn't function correctly when run as a systemd service.


Version-Release number of selected component (if applicable):


How reproducible: I've only tried it once, but it happened consistently to me.


Steps to Reproduce:
1. Install Fedora 19
2. Use Packstack to install Havana with Neutron
3. Try to configure Neutron networks

Actual results:
Log messages such as the following:

1098 ERROR neutron.agent.l3_agent [-] Failed synchronizing routers
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Traceback (most recent call last):
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 726, in _sync_routers_task
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     self._process_routers(routers, all_routers=True)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 674, in _process_routers
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     self._router_added(r['id'], r)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 283, in _router_added
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     self._create_router_namespace(ri)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 259, in _create_router_namespace
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     ip_wrapper = ip_wrapper_root.ensure_namespace(ri.ns_name())
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 127, in ensure_namespace
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     ip = self.netns.add(name)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 415, in add
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     self._as_root('add', name, use_root_namespace=True)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 185, in _as_root
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     kwargs.get('use_root_namespace', False))
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 65, in _as_root
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     namespace)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 76, in _execute
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     root_helper=root_helper)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 61, in execute
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent     raise RuntimeError(m)
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent RuntimeError: 
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-c326abf4-a914-49eb-a1ab-b292dab7970d']
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Exit code: 255
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Stdout: ''
2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Stderr: 'mount --make-shared /var/run/netns failed: Permission denied\n'


Expected results: Successful configuration of the network.


Additional info: Stopping the Neutron services through systemd and restarting them directly as root provided a workaround, but it's obviously not ideal.

Also, I should note that I believe this is a systemd problem and not a rootwrap problem because I enabled the neutron user and ran the command manually under that user and it worked fine.  So this error only seems to happen under systemd, and I assume it's some systemd security setting that is blocking the mount command.
Comment 2 Alan Pevec 2013-09-26 18:51:32 EDT
Any SELinux denials in the audit.log, does it work with 
Difference could be that when you run from shell, process runs unconfined.
Comment 3 Ben Nemec 2013-09-27 11:48:08 EDT
Shoot, you're right.  I thought SELinux was turned off on this system, but I checked again and it wasn't.  With it disabled Neutron works correctly under systemd.  Still a problem, just not the one I initially reported. :-)  I'll update the title.

Will you need entries from the audit log to fix this?  For some reason I only seem to have audit logs from today, and since I tried shutting off SELinux last night there's nothing related to this that I can see.  If needed, I can try recreating the problem though.
Comment 4 Martin Magr 2013-10-01 10:11:11 EDT
Yes please. Output of "ausearch -m avc" should be enough.
Comment 5 Ben Nemec 2013-10-02 12:50:06 EDT
Okay, here are the last few lines from that (there were 694 hits when I ran the command, so I assume you don't want them all :-):

time->Wed Oct  2 11:44:37 2013
type=SYSCALL msg=audit(1380732277.809:13751): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13679 pid=13682 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732277.809:13751): avc:  denied  { mounton } for  pid=13682 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
----
time->Wed Oct  2 11:44:37 2013
type=SYSCALL msg=audit(1380732277.828:13754): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13681 pid=13683 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732277.828:13754): avc:  denied  { mounton } for  pid=13683 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
----
time->Wed Oct  2 11:45:07 2013
type=SYSCALL msg=audit(1380732307.897:13963): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13852 pid=13856 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732307.897:13963): avc:  denied  { mounton } for  pid=13856 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
----
time->Wed Oct  2 11:45:07 2013
type=SYSCALL msg=audit(1380732307.910:13966): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13854 pid=13857 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732307.910:13966): avc:  denied  { mounton } for  pid=13857 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
----
time->Wed Oct  2 11:45:37 2013
type=SYSCALL msg=audit(1380732337.989:14175): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=14074 pid=14076 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732337.989:14175): avc:  denied  { mounton } for  pid=14076 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
----
time->Wed Oct  2 11:45:37 2013
type=SYSCALL msg=audit(1380732337.999:14178): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=14075 pid=14077 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null)
type=AVC msg=audit(1380732337.999:14178): avc:  denied  { mounton } for  pid=14077 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
Comment 6 Lon Hohberger 2013-12-03 16:59:05 EST
This is a duplicate of another bug.  As it turns out, most of the AVCs are related to wrong file labels on /usr/bin/neutron-*.

semanage fcontext -a -t neutron_exec_t /usr/bin/neutron-lbaas-agent
semanage fcontext -a -t neutron_exec_t /usr/bin/neutron-rootwrap
restorecon /usr/bin/neutron*

In the updated openstack-selinux which is in the beta channel for RHOS 4.0, the above is done for you during RPM installation.

*** This bug has been marked as a duplicate of bug 1020052 ***

Note You need to log in before you can comment on or make changes to this bug.