Bug 915906
| Summary: | Quantum DHCP is being denied by Selinux | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ofer Blaut <oblaut> | ||||||||||||||||
| Component: | openstack-selinux | Assignee: | Lon Hohberger <lhh> | ||||||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Yaniv Kaul <ykaul> | ||||||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||||||
| Priority: | high | ||||||||||||||||||
| Version: | 2.0 (Folsom) | CC: | chrisw, dwalsh, jkt, mgrepl, mmalik, rkukura, ykaul | ||||||||||||||||
| Target Milestone: | snapshot4 | Keywords: | Triaged | ||||||||||||||||
| Target Release: | 2.1 | ||||||||||||||||||
| Hardware: | Unspecified | ||||||||||||||||||
| OS: | Unspecified | ||||||||||||||||||
| Whiteboard: | |||||||||||||||||||
| Fixed In Version: | openstack-selinux-0.1.2-6.el6 | Doc Type: | Bug Fix | ||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||
| Last Closed: | 2013-03-21 19:06:27 UTC | Type: | Bug | ||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||
| Embargoed: | |||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
The 'ip' issue should be resolved, the other AVCs are new.
Feb 26 19:00:26 puma34 kernel: type=1400 audit(1361898026.669:213205): avc: denied { search } for pid=20470 comm="python" name="dhcp" dev=dm-0 ino=5243516 scontext=unconfined_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:quantum_var_lib_t:s0 tclass=dir
Feb 26 19:27:42 puma34 kernel: type=1400 audit(1361899662.236:213207): avc: denied { getattr } for pid=31922 comm="python" path="/var/lib/quantum/dhcp/lease_relay" dev=dm-0 ino=5243528 scontext=unconfined_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:quantum_var_lib_t:s0 tclass=sock_file
Feb 26 19:27:42 puma34 kernel: type=1400 audit(1361899662.236:213208): avc: denied { write } for pid=31922 comm="python" name="lease_relay" dev=dm-0 ino=5243528 scontext=unconfined_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:object_r:quantum_var_lib_t:s0 tclass=sock_file
Feb 26 19:27:42 puma34 kernel: type=1400 audit(1361899662.237:213209): avc: denied { connectto } for pid=31922 comm="python" path="/var/lib/quantum/dhcp/lease_relay" scontext=unconfined_u:system_r:dnsmasq_t:s0 tcontext=unconfined_u:system_r:initrc_t:s0 tclass=unix_stream_socket
Hi, Can you flip to permissive mode and do a full cycle test on quantum, then post the /var/log/audit/audit.log? These AVCs appear to be generated with SELinux in 'enforcing' mode - which will prevent dnsmasq from running correctly when controlled by Quantum. This means that, once running correctly, there may be more AVCs to chase down. The ones here are a good start, however. Created attachment 703886 [details]
audit log - permissive mode
Audit log in permissive mode
What does # ps -eZ |grep initrc [root@puma34 ~(keystone_admin)]$ ps -eZ |grep initrc system_u:system_r:initrc_t:s0 2774 ? 00:00:00 cinder-api system_u:system_r:initrc_t:s0 2782 ? 00:01:10 cinder-schedule system_u:system_r:initrc_t:s0 2862 ? 00:00:05 python system_u:system_r:initrc_t:s0 17475 ? 00:00:05 python system_u:system_r:initrc_t:s0 17990 ? 00:00:05 python unconfined_u:system_r:initrc_t:s0 28607 ? 00:02:23 python unconfined_u:system_r:initrc_t:s0 28639 ? 00:00:31 python Looks like we may need to label quantum as a dhcpcd server. Is this all it does? You mean just a quantum binary which relates with dhcpcd, right? Ofer, probably better would be # ps -efZ |grep initrc The /usr/bin/quantum-dhcp-agent executable forks/execs dnsmasq, as well as various other networking commands. As Lon said, the anon_inode AVC (rhbz 878846) should be fixed in RHOS snapshot 4. Mgrepl yup or at least write new policy for all of those initrc_t. Awesome, thanks. [root@puma34 ~]# ps -efZ | grep initrc system_u:system_r:initrc_t:s0 cinder 2774 1 0 Feb27 ? 00:00:00 /usr/bin/python /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log system_u:system_r:initrc_t:s0 cinder 2782 1 0 Feb27 ? 00:01:25 /usr/bin/python /usr/bin/cinder-scheduler --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/scheduler.log system_u:system_r:initrc_t:s0 nova 2862 1 0 Feb27 ? 00:00:06 python /usr/bin/nova-novncproxy --web /usr/share/novnc/ unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 14200 13848 0 21:03 pts/0 00:00:00 grep initrc system_u:system_r:initrc_t:s0 nova 17475 2862 0 08:43 ? 00:00:05 [python] <defunct> system_u:system_r:initrc_t:s0 nova 17990 2862 0 08:44 ? 00:00:06 [python] <defunct> unconfined_u:system_r:initrc_t:s0 quantum 28607 1 0 Feb27 ? 00:02:56 python /usr/bin/quantum-l3-agent --log-file /var/log/quantum/l3-agent.log --config-file /usr/share/quantum/quantum-dist.conf --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/l3_agent.ini unconfined_u:system_r:initrc_t:s0 quantum 28639 1 0 Feb27 ? 00:00:38 python /usr/bin/quantum-dhcp-agent --log-file /var/log/quantum/dhcp-agent.log --config-file /usr/share/quantum/quantum-dist.conf --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/dhcp_agent.ini [root@puma34 dhcp]# ls -lZ lease_relay srwxr-xr-x. quantum quantum unconfined_u:object_r:quantum_var_lib_t:s0 lease_relay [root@puma34 dhcp]# fuser lease_relay lease_relay: 28639 type=AVC msg=audit(1362429993.870:886): avc: denied { search } for pid=4116 comm="python" name="quantum" dev=sda2 ino=15204688 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1362429995.250:916): avc: denied { write } for pid=4137 comm="dnsmasq" name="pid" dev=sda2 ino=15204863 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=file
type=AVC msg=audit(1362429995.250:916): avc: denied { open } for pid=4137 comm="dnsmasq" name="pid" dev=sda2 ino=15204863 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=file
type=AVC msg=audit(1362429995.250:917): avc: denied { getattr } for pid=4137 comm="dnsmasq" path="/var/lib/quantum/dhcp/9374a7c8-4337-4594-b79d-3c66ab2c1911/pid" dev=sda2 ino=15204863 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=file
type=AVC msg=audit(1362429995.252:918): avc: denied { read } for pid=4137 comm="dnsmasq" name="host" dev=sda2 ino=15204861 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=file
Notably, in my test environment, I don't have the lease_relay initrc AVC denial - only the others: [root@ayanami ~]# ls -lZ /var/lib/quantum/dhcp/lease_relay srwxr-xr-x. quantum quantum system_u:object_r:quantum_var_lib_t:s0 /var/lib/quantum/dhcp/lease_relay So, that's good. This is with the standard SELinux-policy as shipped with RHEL 6.4 and the current openstack-selinux package. We can either label /var/lib/quantum/dhcp/ as dnsmasq_lease_t. # semanage fcontext -a -t dnsmasq_lease_t '/var/lib/quantum/dhcp(/.*)?' # restorecon -R -v /var/lib/quantom Or we can just give dnsmasq access to all of /var/lib/quantum. Lon, could you test the commands above? I will, sure. This is all that popped up:
type=USER_CMD msg=audit(03/05/2013 10:15:50.496:1054) : user pid=4310 uid=quantum auid=unset ses=unset subj=system_u:system_r:quantum_t:s0 msg='cwd=/ cmd=quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl --timeout=2 get Interface tapc0837947-00 external_ids terminal=? res=success'
type=SYSCALL msg=audit(03/05/2013 10:15:50.663:1057) : arch=x86_64 syscall=stat success=no exit=-2(No such file or directory) a0=1ae09f0 a1=7fff2fb94c00 a2=7fff2fb94c00 a3=2e326e6f68747970 items=0 ppid=4312 pid=4313 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=python exe=/usr/bin/python subj=system_u:system_r:dnsmasq_t:s0 key=(null)
type=AVC msg=audit(03/05/2013 10:15:50.663:1057) : avc: denied { search } for pid=4313 comm=python name=quantum dev=sda2 ino=15204688 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=dir
[root@ayanami lhh]# debugfs /dev/sda2
debugfs 1.41.12 (17-May-2010)
debugfs: ncheck 15204688
Inode Pathname
15204688 /var/lib/quantum
So, we probably need it for the whole directory?
(I wonder what it was calling stat() on ...) Maybe: allow dnsmasq_t quantum_var_lib_t:dir search; allow dnsmasq_t quantum_var_lib_t:file rw_file_perms; Created attachment 705520 [details]
Updated policy
Created attachment 705528 [details]
Patch
Created attachment 705531 [details]
Patch
Created attachment 705546 [details]
Updated patch with Miroslav's recommend change
Created attachment 705547 [details]
Updated patch with Miroslav's recommend change
Previous patch was wrong.
I retested and most of the AVCs are gone, however, whatever I had previously was masking the lease_relay issue:
type=AVC msg=audit(1362521347.557:2451): avc: denied { getattr } for pid=6118 comm="python" path="/var/lib/quantum/dhcp/lease_relay" dev=sda2 ino=15204808 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1362521347.558:2452): avc: denied { write } for pid=6118 comm="python" name="lease_relay" dev=sda2 ino=15204808 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:quantum_var_lib_t:s0 tclass=sock_file
type=AVC msg=audit(1362521347.558:2452): avc: denied { connectto } for pid=6118 comm="python" path="/var/lib/quantum/dhcp/lease_relay" scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=unix_stream_socket
type=AVC msg=audit(1362521364.995:2713): avc: denied { connectto } for pid=25593 comm="python" path="/var/lib/quantum/dhcp/lease_relay" scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=unix_stream_socket
[root@ayanami lhh]# fuser /var/lib/quantum/dhcp/lease_relay
/var/lib/quantum/dhcp/lease_relay: 2937
[root@ayanami lhh]# ps auwwxZ | grep 2937
system_u:system_r:initrc_t:s0 quantum 2937 0.4 0.3 260660 18932 ? S 17:06 0:01 python /usr/bin/quantum-dhcp-agent --log-file /var/log/quantum/dhcp-agent.log --config-file /usr/share/quantum/quantum-dist.conf --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/dhcp_agent.ini
Setting /usr/bin/quantum-dhcp-agent to quantum_exec_t fixes this and then SELinux transitions properly, making the AVCs go away. [root@ayanami ~]# semodule -l | grep quant openstack-selinux-quantum 0.2.1 quantum 1.0.0 [root@ayanami ~]# rpm -q selinux-policy-targeted selinux-policy-targeted-3.7.19-195.el6.noarch The incorrect label on /usr/bin/quantum-dhcp-agent is resolved in selinux-policy-3.7.19-195.el6_4 or later from the 6.4.z channel. Side note, if the 6.4.z RPM is not available, the workaround is: semanage fcontext -a -t quantum_exec_t /usr/bin/quantum-dhcp-agent Tested works with selinux-policy-targeted-3.7.19-195.el6_4.noarch selinux-policy-3.7.19-195.el6_4.noarch openstack-quantum-2012.2.3-5.el6ost.noarch *** Bug 919193 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0672.html |
Created attachment 703059 [details] dhcp and avs logs Description of problem: Quantum DHCP agnet is being denied by Selinux looking at /var/log/quantum/dhcp.log file 2013-02-26 15:55:21 ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp. Stderr: '\ndnsmasq: failed to open pidfile /var/lib/quantum/dhcp/4a9d347d-4498-4ac9-8118-77f719f417ce/pid: Permission denied\n' Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. install quantum on machine with selinux enforcing 2. check DHCP and DHCP log file 3. Actual results: Expected results: Additional info: