Hide Forgot
Description of problem: Neutron doesn't function correctly when run as a systemd service. Version-Release number of selected component (if applicable): How reproducible: I've only tried it once, but it happened consistently to me. Steps to Reproduce: 1. Install Fedora 19 2. Use Packstack to install Havana with Neutron 3. Try to configure Neutron networks Actual results: Log messages such as the following: 1098 ERROR neutron.agent.l3_agent [-] Failed synchronizing routers 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Traceback (most recent call last): 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 726, in _sync_routers_task 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent self._process_routers(routers, all_routers=True) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 674, in _process_routers 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent self._router_added(r['id'], r) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 283, in _router_added 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent self._create_router_namespace(ri) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 259, in _create_router_namespace 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent ip_wrapper = ip_wrapper_root.ensure_namespace(ri.ns_name()) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 127, in ensure_namespace 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent ip = self.netns.add(name) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 415, in add 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent self._as_root('add', name, use_root_namespace=True) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 185, in _as_root 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent kwargs.get('use_root_namespace', False)) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 65, in _as_root 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent namespace) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 76, in _execute 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent root_helper=root_helper) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 61, in execute 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent raise RuntimeError(m) 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent RuntimeError: 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qrouter-c326abf4-a914-49eb-a1ab-b292dab7970d'] 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Exit code: 255 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Stdout: '' 2013-09-25 12:34:41.300 1098 TRACE neutron.agent.l3_agent Stderr: 'mount --make-shared /var/run/netns failed: Permission denied\n' Expected results: Successful configuration of the network. Additional info: Stopping the Neutron services through systemd and restarting them directly as root provided a workaround, but it's obviously not ideal. Also, I should note that I believe this is a systemd problem and not a rootwrap problem because I enabled the neutron user and ran the command manually under that user and it worked fine. So this error only seems to happen under systemd, and I assume it's some systemd security setting that is blocking the mount command.
Any SELinux denials in the audit.log, does it work with Difference could be that when you run from shell, process runs unconfined.
Shoot, you're right. I thought SELinux was turned off on this system, but I checked again and it wasn't. With it disabled Neutron works correctly under systemd. Still a problem, just not the one I initially reported. :-) I'll update the title. Will you need entries from the audit log to fix this? For some reason I only seem to have audit logs from today, and since I tried shutting off SELinux last night there's nothing related to this that I can see. If needed, I can try recreating the problem though.
Yes please. Output of "ausearch -m avc" should be enough.
Okay, here are the last few lines from that (there were 694 hits when I ran the command, so I assume you don't want them all :-): time->Wed Oct 2 11:44:37 2013 type=SYSCALL msg=audit(1380732277.809:13751): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13679 pid=13682 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732277.809:13751): avc: denied { mounton } for pid=13682 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir ---- time->Wed Oct 2 11:44:37 2013 type=SYSCALL msg=audit(1380732277.828:13754): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13681 pid=13683 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732277.828:13754): avc: denied { mounton } for pid=13683 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir ---- time->Wed Oct 2 11:45:07 2013 type=SYSCALL msg=audit(1380732307.897:13963): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13852 pid=13856 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732307.897:13963): avc: denied { mounton } for pid=13856 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir ---- time->Wed Oct 2 11:45:07 2013 type=SYSCALL msg=audit(1380732307.910:13966): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=13854 pid=13857 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732307.910:13966): avc: denied { mounton } for pid=13857 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir ---- time->Wed Oct 2 11:45:37 2013 type=SYSCALL msg=audit(1380732337.989:14175): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=14074 pid=14076 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732337.989:14175): avc: denied { mounton } for pid=14076 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir ---- time->Wed Oct 2 11:45:37 2013 type=SYSCALL msg=audit(1380732337.999:14178): arch=c000003e syscall=165 success=no exit=-13 a0=43a1b4 a1=43523a a2=436315 a3=104000 items=0 ppid=14075 pid=14077 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ip" exe="/usr/sbin/ip" subj=system_u:system_r:ifconfig_t:s0 key=(null) type=AVC msg=audit(1380732337.999:14178): avc: denied { mounton } for pid=14077 comm="ip" path="/run/netns" dev="tmpfs" ino=45625 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:ifconfig_var_run_t:s0 tclass=dir
This is a duplicate of another bug. As it turns out, most of the AVCs are related to wrong file labels on /usr/bin/neutron-*. semanage fcontext -a -t neutron_exec_t /usr/bin/neutron-lbaas-agent semanage fcontext -a -t neutron_exec_t /usr/bin/neutron-rootwrap restorecon /usr/bin/neutron* In the updated openstack-selinux which is in the beta channel for RHOS 4.0, the above is done for you during RPM installation. *** This bug has been marked as a duplicate of bug 1020052 ***