Bug 1169859
| Summary: | Neutron l3-agent has SELinux denial when starting or stopping keepalived | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | yfried | ||||
| Component: | openstack-selinux | Assignee: | Ryan Hallisey <rhallise> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Ami Jeain <ajeain> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.0 (Juno) | CC: | amuller, bperkins, chrisw, dnavale, emilien.macchi, jguiditt, jlibosva, lhh, mgrepl, mori, nyechiel, oblaut, yeylon | ||||
| Target Milestone: | ga | ||||||
| Target Release: | 6.0 (Juno) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-selinux-0.6.5-1.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
SELinux did not allow OpenStack Networking to fully interact with 'keepalived', leading to OpenStack Networking failing to run properly.
With this update, SELinux is given the proper 'allow' rules so that OpenStack Networking can successfully run without any issues.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-02-09 14:22:05 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Note that the machine's /usr/share/neutron/rootwrap/l3.filters has a line for keepalived, which works in Devstack on F20. Not sure why we're getting a rootwrap issue here. From my experience with rootwrap, I'd explore /proc/<keepalived_pid>/cmd and /proc/<keepalived_pid>/exe to see why filter failed. After some debugging we consider this selinux policy issue. After switching to permissive mode, denials started to appear:
type=AVC msg=audit(1417622542.944:32983): avc: denied { execute } for pid=27516 comm="neutron-rootwra" name="keepalived" dev="vda3" ino=138190 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file
type=AVC msg=audit(1417622542.958:32984): avc: denied { read open } for pid=27517 comm="ip" path="/usr/sbin/keepalived" dev="vda3" ino=138190 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file
type=AVC msg=audit(1417622542.958:32984): avc: denied { execute_no_trans } for pid=27517 comm="ip" path="/usr/sbin/keepalived" dev="vda3" ino=138190 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file
type=AVC msg=audit(1417622542.995:32987): avc: denied { create } for pid=27519 comm="keepalived" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=netlink_socket
type=AVC msg=audit(1417622542.995:32988): avc: denied { setopt } for pid=27519 comm="keepalived" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=netlink_socket
type=AVC msg=audit(1417622542.995:32989): avc: denied { bind } for pid=27519 comm="keepalived" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=netlink_socket
type=AVC msg=audit(1417622542.995:32990): avc: denied { getattr } for pid=27519 comm="keepalived" scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=netlink_socket
type=AVC msg=audit(1417622542.997:32991): avc: denied { execute } for pid=27521 comm="sh" name="notify_backup.sh" dev="vda3" ino=65030 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_var_lib_t:s0 tclass=file
type=AVC msg=audit(1417622542.997:32991): avc: denied { execute_no_trans } for pid=27521 comm="sh" path="/var/lib/neutron/ha_confs/90ecb37a-7050-4ca6-b4c8-29bf5950c42e/notify_backup.sh" dev="vda3" ino=65030 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:neutron_var_lib_t:s0 tclass=file
I think the reason why denial was not visible in enforcing mode was due to os.access() in rootwrap when checking whether binary can be executed. We observed when process running in neutron_t, os.access() returns False for /sbin/keepalived file. Thus no filter was found.
I'm switching component to selinux.
Hi, I encountered similar phenomenon with RDO. I tried to create a patch to fix TE rules. Please find attached file in this bz: https://bugzilla.redhat.com/show_bug.cgi?id=1170238 allow neutron_t keepalived_exec_t:file { read execute open execute_no_trans };
allow neutron_t neutron_var_lib_t:file { execute execute_no_trans };
allow neutron_t self:netlink_socket { bind create setopt getattr };
This seem ok. Miroslav do you have any comment about it?
So we have a file which wants to have write/execute perms which is not good idea. Are scripts located in /var/lib/neutron/ha_confs/90ecb37a-7050-4ca6-b4c8-29bf5950c42e/ created on the fly? First we need the L3 agent to be able to start and stop keepalived. Secondly, keepalived.conf and the notifier scripts are created and written by the agent whenever a router is created / updated / deleted. The 'state' file is written by the notifier scripts. can_exec(neutron_t,neutron_var_lib_t) keepalived_domtrans(neutron_t) Verified on: openstack-selinux-0.6.6-1.el7ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2015-0144.html |
Created attachment 963787 [details] files from packstack, controller and nodes Description of problem: Neutron l3-agents unable to bring ha-routers online due to keepalived not having root permissions Version-Release number of selected component (if applicable): [root@RHEL7Server ~(keystone_admin)]# rpm -qa | grep "neutron\|packstack" python-neutronclient-2.3.9-1.el7ost.noarch python-neutron-2014.2-11.el7ost.noarch openstack-neutron-ml2-2014.2-11.el7ost.noarch openstack-packstack-puppet-2014.2-0.5.dev1316.g733aa73.el7ost.noarch openstack-packstack-2014.2-0.5.dev1316.g733aa73.el7ost.noarch openstack-neutron-2014.2-11.el7ost.noarch How reproducible: create ha-router look for "state" file in keepalived dir on l3-agent $ ls /var/lib/neutron/ha_confs/<router-id>/ "state" file doesn't exist [root@RHEL7Server rootwrap]# grep "TRACE\|ERROR" /var/log/neutron/l3-agent.log 2014-12-02 16:25:31.622 1858 ERROR neutron.agent.linux.utils [-] 2014-12-02 16:25:31.622 1858 ERROR neutron.agent.l3_agent [-] 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent Traceback (most recent call last): 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 341, in call 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent return func(*args, **kwargs) 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 1015, in process_router 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent ri.spawn_keepalived() 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_ha_agent.py", line 87, in spawn_keepalived 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent self.keepalived_manager.spawn_or_restart() 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py", line 354, in spawn_or_restart 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent self.spawn() 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py", line 345, in spawn 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent self.process.enable(callback) 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 80, in enable 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent ip_wrapper.netns.execute(cmd, addl_env=self.cmd_addl_env) 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 550, in execute 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent check_exit_code=check_exit_code, extra_ok_codes=extra_ok_codes) 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 84, in execute 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent raise RuntimeError(m) 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent RuntimeError: 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-16e18d0b-04ed-4aa9-bafb-404361076537', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537.pid', '-r', '/var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537.pid-vrrp'] 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent Exit code: 99 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent Stdout: '' 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-16e18d0b-04ed-4aa9-bafb-404361076537 keepalived -P -f /var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537/keepalived.conf -p /var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537.pid -r /var/lib/neutron/ha_confs/16e18d0b-04ed-4aa9-bafb-404361076537.pid-vrrp (no filter matched)\n' 2014-12-02 16:25:31.622 1858 TRACE neutron.agent.l3_agent