Bug 917534
| Summary: | Nova: SELinux AVC Errors for "iptables-save" / "iptables-restor". | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Omri Hochman <ohochman> | ||||||||
| Component: | openstack-nova | Assignee: | Lon Hohberger <lhh> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Yaniv Kaul <ykaul> | ||||||||
| Severity: | urgent | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 2.0 (Folsom) | CC: | aortega, derekh, dwalsh, eparis, jhenner, jkt, lhh, mgrepl, ndipanov, ohochman, ykaul | ||||||||
| Target Milestone: | snapshot5 | Keywords: | Triaged | ||||||||
| Target Release: | 2.1 | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | openstack-nova-2012.2.3-6.el6ost | Doc Type: | Bug Fix | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 920662 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2013-04-04 20:21:51 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 917370, 920662, 1102484 | ||||||||||
| Attachments: |
|
||||||||||
|
Description
Omri Hochman
2013-03-04 09:59:20 UTC
Created attachment 704850 [details]
/var/log/messages
Created attachment 704851 [details]
/var/log/secure
Need /var/log/audit/audit.log when looking at AVCs #============= iptables_t ==============
allow iptables_t self:capability { sys_admin sys_resource };
Now, the question is why this is needed under Nova. Looking in to this.
Created attachment 706782 [details]
Audit log
sys_admin and sys_resource indicate the system was running out of resources. syscall=vfork sounds to me like ulimit -u is too low.... I did some more in-depth checking here based on Eric and Dan's comments. I'm pretty sure what's happening is that all of the processes created by dnsmasq have real UID of 'nobody' (and effective UID of root), so they're hitting the ulimit for the 'nobody' UID. We probably need to increase the ulimit for number of processes; this can be done by editing /etc/security/limits.d/90-nproc and adding the following line: nobody soft nproc 4096 Sadly, from the syscall record: auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 Sure, but I still think it's an inheritance problem: someone running iptables logged in as root from a root login session has different ulimits.
dnsmasq simply has CAP_NET_ADMIN and CAP_NET_RAW, and has a 1024 process limit as UID nobody. It uses nova-rootwrap, which calls sudo to call the iptables commands.
Generally, a task executed using sudo by non-root has the following (on 6.4; I checked with crash):
rlim_cur = 1024,
rlim_max = 30492
Vs. a login shell as root:
rlim_cur = 30492,
rlim_max = 30492
So, what I'm trying to figure out is why the capabilities are wrong.
They're not wrong, rather mismatched:
allow iptables_t self:capability { dac_read_search dac_override net_admin net_raw };
When iptables is run, it drops the capabilities needed to punch past the ulimit. So, when it calls vfork, it fails.
In the case of nova-rootwrap, what appears to happen is this:
- dnsmasq is running as 'nobody' with a 1024 process limit
- dnsmasq executes some code from nova to execute nova-rootwrap, which
wraps around sudo
- iptables* is spawned, and has the rlimits for 'nobody'
- it transitions to iptables_t, which drops CAP_SYS_ADMIN and CAP_SYS_RESOURCE
Now... in the code Eric pointed me at last week, I noticed that not only are we at this 1024 user limit for the iptables-* process.
The ugly part is that the iptables-* process's rlimit (in this case, 1024 max tasks) count is compared against *root's process count*, not *nobody's process count*.
This means that when running 'sudo', all other root processes on the system count against a user's ulimit.
I don't know if it's intentional or not, but I suspect this is why it's so easy to hit.
Clarification: This means that when running 'sudo', all other root processes on the system count against the rlimit for max processes as inherited from the original user. My error, the caller was not correct - we need to increase the rlimit for the 'nova' user, not the 'nobody' user. type=USER_CMD msg=audit(03/06/2013 10:38:25.101:65532) : user pid=16814 uid=nova auid=root ses=721 subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 msg='cwd=/ cmd=nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c terminal=? res=success' ---- type=CRED_ACQ msg=audit(03/06/2013 10:38:25.101:65533) : user pid=16814 uid=root auid=root ses=721 subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct=root exe=/usr/bin/sudo hostname=? addr=? terminal=? res=success' I would say this is a kernel bug, either the rlimit should change to root User or the process count should be done against the UID. I think we should start with a lower number and increase if we hit the limit during testing. nova soft nproc 2048 On second thought, openstack-nova could easily drop in 90-nova.conf in to /etc/security/limits.d - this would save the administrator steps and allow us to easily change it later without the administrator having to rerun packstack. 91-nova is better; we want to ensure that the nova override is pulled in after the 90-nproc defaults. (In reply to comment #21) > On second thought, openstack-nova could easily drop in 90-nova.conf in to > /etc/security/limits.d - this would save the administrator steps and allow > us to easily change it later without the administrator having to rerun > packstack. As suggested manage to artificially reproduce the issue by setting a low rlimit for nova user. /etc/security/limits.d/90-nproc.conf --> nova soft nproc 600. In previous build I was able to reproduce by setting an artificially low rlimit for the nova user to 600. the installation of the newer build openstack-selinux-0.1.2-7.el6ost.noarch, created the file /etc/security/limits.d/91-nova.conf created, which extend the rlimit for the nova user to 2048. # Due to the way process inheritance is set up on Linux, # calling sudo inherits the caller's ulimit. Processes # owned by the new UID are counted against inherited # ulimit. # # Transitioning to the iptables domain drops the ability # to break the soft ulimit for number of processes, which # causes iptables commands to fail in certain cases. # nova soft nproc 2048 Verified with openstack-selinux-0.1.2-7.el6ost.noarch. Unable to reproduce: I've tried to stress the controller by booting multiple instances 50(+), No AVCs errors found. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0709.html |