Red Hat Bugzilla – Bug 917534
Nova: SELinux AVC Errors for "iptables-save" / "iptables-restor".
Last modified: 2014-05-29 00:27:40 EDT
Nova: SELinux AVC Errors for "iptables-save" / "iptables-restor". Description: ------------ This issue is not always reproducible, but I've encountered it before and this time during investigation of Bz#917370. after booting VM, I found AVC Errors of "iptables-save" / "iptables-restor" in /var/log/messages. It seems that the Errors pops with - "ip addr add" , "ip route add" that occurred at the same time (from : /var/log/secure). Environment: ------------ selinux-policy-3.7.19-195.el6.noarch selinux-policy-targeted-3.7.19-195.el6.noarch openstack-selinux-0.1.2-5.el6ost.noarch openstack-nova-compute-2012.2.3-4.el6ost.noarch openstack-nova-cert-2012.2.3-4.el6ost.noarch openstack-nova-console-2012.2.3-4.el6ost.noarch python-django-openstack-auth-1.0.6-2.el6ost.noarch /var/log/messages: -------------------- Mar 3 16:16:08 puma01 kernel: type=1400 audit(1362320168.564:50311): avc: denied { sys_admin } for pid=16497 comm="iptables-save" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:08 puma01 kernel: type=1400 audit(1362320168.564:50312): avc: denied { sys_resource } for pid=16497 comm="iptables-save" capability=24 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:08 puma01 kernel: type=1400 audit(1362320168.733:50313): avc: denied { sys_admin } for pid=16501 comm="iptables-restor" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:08 puma01 kernel: type=1400 audit(1362320168.733:50314): avc: denied { sys_resource } for pid=16501 comm="iptables-restor" capability=24 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:09 puma01 kernel: type=1400 audit(1362320169.081:50315): avc: denied { sys_admin } for pid=16511 comm="iptables-save" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:09 puma01 kernel: type=1400 audit(1362320169.081:50316): avc: denied { sys_resource } for pid=16511 comm="iptables-save" capability=24 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:09 puma01 kernel: type=1400 audit(1362320169.198:50317): avc: denied { sys_admin } for pid=16515 comm="iptables-restor" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:09 puma01 kernel: type=1400 audit(1362320169.199:50318): avc: denied { sys_resource } for pid=16515 comm="iptables-restor" capability=24 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:11 puma01 dnsmasq[16552]: started, version 2.48 cachesize 150 Mar 3 16:16:11 puma01 dnsmasq[16552]: compile time options: IPv6 GNU-getopt DBus no-I18N DHCP TFTP "--bind-interfaces with SO_BINDTODEVICE" Mar 3 16:16:11 puma01 dnsmasq-dhcp[16552]: DHCP, static leases only on 192.168.32.2, lease time 2m Mar 3 16:16:11 puma01 dnsmasq[16552]: reading /etc/resolv.conf Mar 3 16:16:11 puma01 dnsmasq[16552]: using nameserver 10.34.32.3#53 Mar 3 16:16:11 puma01 dnsmasq[16552]: using nameserver 10.35.28.1#53 Mar 3 16:16:11 puma01 dnsmasq[16552]: using nameserver 10.35.28.28#53 Mar 3 16:16:11 puma01 dnsmasq[16552]: read /etc/hosts - 2 addresses Mar 3 16:16:11 puma01 dnsmasq[16552]: read /var/lib/nova/networks/nova-br100.conf Mar 3 16:16:11 puma01 kernel: type=1400 audit(1362320171.793:50319): avc: denied { sys_admin } for pid=16557 comm="iptables-save" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:16:11 puma01 kernel: type=1400 audit(1362320171.793:50320): avc: denied { sys_resource } for pid=16557 comm="iptables-save" capability=24 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability Mar 3 16:17:25 puma01 dnsmasq[16552]: read /etc/hosts - 2 addresses Mar 3 16:17:25 puma01 dnsmasq[16552]: read /var/lib/nova/networks/nova-br100.conf Mar 3 16:17:25 puma01 kernel: __ratelimit: 12 callbacks suppressed Mar 3 16:17:25 puma01 kernel: type=1400 audit(1362320245.672:50327): avc: denied { sys_admin } for pid=16594 comm="iptables-save" capability=21 scontext=unconfined_u:system_r:iptables_t:s0 tcontext=unconfined_u:system_r:iptables_t:s0 tclass=capability /var/log/secure : ----------------- Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr add 10.35.160.11/24 brd 10.35.160 .255 scope global dev br100 Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route add default via 10.35.160.254 Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:16:08 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf sysctl -w net.ipv4.ip_forward=1 Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr show dev br100 scope global Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route show dev br100 Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route del default dev br100 Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr del 10.35.160.11/24 brd 10.35.160 .255 scope global dev br100 Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr add 192.168.32.1/24 brd 192.168.3 2.255 dev br100 Mar 3 16:16:09 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip addr add 10.35.160.11/24 brd 10.35.160 .255 scope global dev br100 Mar 3 16:16:10 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip route add default via 10.35.160.254 Mar 3 16:16:10 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.32.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.32.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro Mar 3 16:16:11 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter Mar 3 16:16:11 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:16:11 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle Mar 3 16:16:12 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:16:12 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t nat Mar 3 16:16:12 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:17:25 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf kill -HUP 16552 Mar 3 16:17:25 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t filter Mar 3 16:17:25 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c Mar 3 16:17:25 puma01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c -t mangle
Created attachment 704850 [details] /var/log/messages
Created attachment 704851 [details] /var/log/secure
Need /var/log/audit/audit.log when looking at AVCs
#============= iptables_t ============== allow iptables_t self:capability { sys_admin sys_resource }; Now, the question is why this is needed under Nova. Looking in to this.
Created attachment 706782 [details] Audit log
sys_admin and sys_resource indicate the system was running out of resources.
syscall=vfork sounds to me like ulimit -u is too low....
I did some more in-depth checking here based on Eric and Dan's comments. I'm pretty sure what's happening is that all of the processes created by dnsmasq have real UID of 'nobody' (and effective UID of root), so they're hitting the ulimit for the 'nobody' UID. We probably need to increase the ulimit for number of processes; this can be done by editing /etc/security/limits.d/90-nproc and adding the following line: nobody soft nproc 4096
Sadly, from the syscall record: auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
Sure, but I still think it's an inheritance problem: someone running iptables logged in as root from a root login session has different ulimits. dnsmasq simply has CAP_NET_ADMIN and CAP_NET_RAW, and has a 1024 process limit as UID nobody. It uses nova-rootwrap, which calls sudo to call the iptables commands. Generally, a task executed using sudo by non-root has the following (on 6.4; I checked with crash): rlim_cur = 1024, rlim_max = 30492 Vs. a login shell as root: rlim_cur = 30492, rlim_max = 30492 So, what I'm trying to figure out is why the capabilities are wrong.
They're not wrong, rather mismatched: allow iptables_t self:capability { dac_read_search dac_override net_admin net_raw }; When iptables is run, it drops the capabilities needed to punch past the ulimit. So, when it calls vfork, it fails.
In the case of nova-rootwrap, what appears to happen is this: - dnsmasq is running as 'nobody' with a 1024 process limit - dnsmasq executes some code from nova to execute nova-rootwrap, which wraps around sudo - iptables* is spawned, and has the rlimits for 'nobody' - it transitions to iptables_t, which drops CAP_SYS_ADMIN and CAP_SYS_RESOURCE Now... in the code Eric pointed me at last week, I noticed that not only are we at this 1024 user limit for the iptables-* process. The ugly part is that the iptables-* process's rlimit (in this case, 1024 max tasks) count is compared against *root's process count*, not *nobody's process count*. This means that when running 'sudo', all other root processes on the system count against a user's ulimit. I don't know if it's intentional or not, but I suspect this is why it's so easy to hit.
Clarification: This means that when running 'sudo', all other root processes on the system count against the rlimit for max processes as inherited from the original user.
My error, the caller was not correct - we need to increase the rlimit for the 'nova' user, not the 'nobody' user. type=USER_CMD msg=audit(03/06/2013 10:38:25.101:65532) : user pid=16814 uid=nova auid=root ses=721 subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 msg='cwd=/ cmd=nova-rootwrap /etc/nova/rootwrap.conf iptables-restore -c terminal=? res=success' ---- type=CRED_ACQ msg=audit(03/06/2013 10:38:25.101:65533) : user pid=16814 uid=root auid=root ses=721 subj=unconfined_u:system_r:virtd_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct=root exe=/usr/bin/sudo hostname=? addr=? terminal=? res=success'
I would say this is a kernel bug, either the rlimit should change to root User or the process count should be done against the UID.
I think we should start with a lower number and increase if we hit the limit during testing. nova soft nproc 2048
On second thought, openstack-nova could easily drop in 90-nova.conf in to /etc/security/limits.d - this would save the administrator steps and allow us to easily change it later without the administrator having to rerun packstack.
91-nova is better; we want to ensure that the nova override is pulled in after the 90-nproc defaults.
(In reply to comment #21) > On second thought, openstack-nova could easily drop in 90-nova.conf in to > /etc/security/limits.d - this would save the administrator steps and allow > us to easily change it later without the administrator having to rerun > packstack. As suggested manage to artificially reproduce the issue by setting a low rlimit for nova user. /etc/security/limits.d/90-nproc.conf --> nova soft nproc 600.
In previous build I was able to reproduce by setting an artificially low rlimit for the nova user to 600. the installation of the newer build openstack-selinux-0.1.2-7.el6ost.noarch, created the file /etc/security/limits.d/91-nova.conf created, which extend the rlimit for the nova user to 2048. # Due to the way process inheritance is set up on Linux, # calling sudo inherits the caller's ulimit. Processes # owned by the new UID are counted against inherited # ulimit. # # Transitioning to the iptables domain drops the ability # to break the soft ulimit for number of processes, which # causes iptables commands to fail in certain cases. # nova soft nproc 2048
Verified with openstack-selinux-0.1.2-7.el6ost.noarch. Unable to reproduce: I've tried to stress the controller by booting multiple instances 50(+), No AVCs errors found.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0709.html