Bug 1486803

Summary: iptables-restore --wait have race on module loading
Product: Red Hat Enterprise Linux 7 Reporter: Tomas Dolezal <todoleza>
Component: iptablesAssignee: Phil Sutter <psutter>
Status: CLOSED ERRATA QA Contact: Tomas Dolezal <todoleza>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: ajb, atragler, baumanmo, dmsimard, dwd, egarver, herrold, igkioka, iptables-maint-list, kajtzu, myllynen, pasik, redhat-bugzilla, riehecky, tis, todoleza, toracat
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: iptables-1.4.21-23.el7 Doc Type: Bug Fix
Doc Text:
Previously, when stopping iptables or ip6tables services, a script tried to unload the netfilter modules related to the given address family. Since there were modules that both address families used, a potential race condition created when both services restarted simultaneously. This update adds the AFTER and BEFORE keywords in the service files. As a result, both services run in sequence, and "systemctl restart iptables ip6tables" works as expected.
Story Points: ---
Clone Of:
: 1486871 1491963 1504647 1544921 (view as bug list) Environment:
Last Closed: 2018-04-10 11:28:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1477413, 1481207, 1486871, 1491963, 1504647, 1544921, 1544922, 1544923, 1560012    
Attachments:
Description Flags
Proposed fix for issues none

Description Tomas Dolezal 2017-08-30 14:55:43 UTC
Description of problem:
when loading rulesets for ipv4 and ipv6 at the same time (boot, service restart), one of the services fails due to inapplicable rules requiring modules which weren't loaded.
Loading the rules for ipv4 and ipv6 in sequence does not produce this error.

Version-Release number of selected component (if applicable):
iptables-1.4.21-18.1.el7_4
iptables-1.6.999-1.el7.x86_64

How reproducible:
always

Steps to Reproduce:
save rules from firewalld where multiple modules are used
systemctl restart iptables ip6tables

Actual results:
:: [   FAIL   ] :: Command 'systemctl restart ip6tables iptables' (Expected 0, got 1)
:: [   FAIL   ] :: services didn't fail (Expected 1, got 0)
:: [   FAIL   ] :: nf_nat_ipv4 loaded (Expected 0, got 1)
:: [   PASS   ] :: nf_nat_ipv6 loaded (Expected 0, got 0)
:: [   FAIL   ] :: nf_conntrack_ipv4 loaded (Expected 0, got 1)
:: [   PASS   ] :: nf_conntrack_ipv6 loaded (Expected 0, got 0)

Expected results:
all modules are (un)loaded on respective service actions

Additional info:

Comment 3 Phil Sutter 2017-08-31 13:02:29 UTC
It seems that the recursive module unloading in iptables.service at some point hits modules which are used by ip6tables as well. My guess here is a race condition between that unloading and the already starting ip6tables.service which makes the kernel load required modules but that fails because depending modules are being unloaded at the same time. That would explain the "unknown symbol" messages logged when restarting fails at least.

I discussed possible fixes with Eric, and we concluded that either recursive module unloading should be limited to modules not in use by the other IP family or the two services should be excluded from being restarted in parallel by systemd.

Limiting recursive module unloading turned out to be not feasible since it's hard to detect whether a module is used by both families, hence I went for the other alternative - it is possible to establish serialization by adding After/Before variables to service definitions.

Note that although this will prevent the race condition for a user calling 'systemctl restart iptables ip6tables', we still need to add --wait option to iptables calls in both services' init files since it might still happen that another process/user calls plain iptables and therefore causes the lock issue.

Comment 4 Phil Sutter 2017-08-31 16:41:22 UTC
*** Bug 1486871 has been marked as a duplicate of this bug. ***

Comment 10 Tuomo Soini 2017-10-20 09:45:41 UTC
Created attachment 1341179 [details]
Proposed fix for issues

After 7.4.2 update there is still an error in log:

[/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument

Attached patch fixes it and also fixes wrong patching of iptables.service which happens to actual source file.

Comment 13 R P Herrold 2018-02-13 18:36:27 UTC
I applied the manual fix of changing teh comma to a space, and then did a relabel of the parent file

this removed the error

BUT

when using the second test, 

I now get SVG errors:

[root@router ~]# restorecon -Rv /usr/lib/systemd/system/ip6tables.service
[root@router ~]# systemctl daemon-reload
[root@router ~]# systemctl restart iptables ip6tables
[root@router ~]#

the last yields (newly)
]
Feb 13 13:25:49 router dbus[794]: [system] Activating service name='org.fedoraproject.Setroubleshootd' (using servicehelper)
Feb 13 13:25:49 router systemd: Started IPv6 firewall with ip6tables.
Feb 13 13:25:50 router dbus-daemon: dbus[794]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Feb 13 13:25:50 router dbus[794]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Feb 13 13:25:50 router kernel: Ebtables v2.0 unregistered
Feb 13 13:25:50 router kernel: [ 8910.020673] Ebtables v2.0 unregistered
Feb 13 13:25:50 router libvirtd: 2018-02-13 18:25:50.699+0000: 1428: info : libvirt version: 3.2.0, package: 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, 2018-01-04-19:31:34, c1bm.rdu2.centos.org)
Feb 13 13:25:50 router libvirtd: 2018-02-13 18:25:50.699+0000: 1428: info : hostname: router.owlriver.net
Feb 13 13:25:50 router libvirtd: 2018-02-13 18:25:50.699+0000: 1428: error : virFirewallApplyRuleFirewallD:790 : The name org.fedoraproject.FirewallD1 was not provided by any .service files

**** from something down in libvirt it seems -- noted here, but proceeding

Feb 13 13:25:50 router systemd: Stopped firewalld - dynamic firewall daemon.
Feb 13 13:25:53 router setroubleshoot: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/lockd.conf. For complete SELinux messages run: sealert -l 8f8426fb-329d-4786-872c-69ef36e9020d

Feb 13 13:25:53 router python: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/lockd.conf.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that grep should be allowed read access on the lockd.conf file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'grep' --raw | audit2allow -M my-grep#012# semodule -i my-grep.pp#012

Feb 13 13:25:53 router setroubleshoot: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/mlx4.conf. For complete SELinux messages run: sealert -l 8f8426fb-329d-4786-872c-69ef36e9020d

Feb 13 13:25:53 router python: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/mlx4.conf.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that grep should be allowed read access on the mlx4.conf file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'grep' --raw | audit2allow -M my-grep#012# semodule -i my-grep.pp#012

Feb 13 13:25:53 router setroubleshoot: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/truescale.conf. For complete SELinux messages run: sealert -l 8f8426fb-329d-4786-872c-69ef36e9020d

Feb 13 13:25:53 router python: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/truescale.conf.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that grep should be allowed read access on the truescale.conf file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'grep' --raw | audit2allow -M my-grep#012# semodule -i my-grep.pp#012

Feb 13 13:25:53 router setroubleshoot: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/tuned.conf. For complete SELinux messages run: sealert -l 8f8426fb-329d-4786-872c-69ef36e9020d
Feb 13 13:25:53 router python: SELinux is preventing /usr/bin/grep from read access on the file /etc/modprobe.d/tuned.conf.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that grep should be allowed read access on the tuned.conf file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'grep' --raw | audit2allow -M my-grep#012# semodule -i my-grep.pp#012



The 'owning packages' for those files are:

[root@router ~]# rpm -qf /etc/modprobe.d/lockd.conf /etc/modprobe.d/mlx4.conf /etc/modprobe.d/truescale.conf /etc/modprobe.d/tuned.conf 
nfs-utils-1.3.0-0.48.el7_4.1.x86_64
rdma-core-13-7.el7.x86_64
rdma-core-13-7.el7.x86_64
tuned-2.8.0-5.el7_4.2.noarch
[root@router ~]# 

each parent is unaltered:

[root@router ~]# rpm -V nfs-utils rdma-core tuned
[root@router ~]# 

I suggest that your testing be expanded to:
   . install the three packages
   . run: systemctl daemon-reload
(expected to run without execception)
   . systemctl restart iptables ip6tables
( demonstrate SELinux AVC's )


I will clone this comment on this bug into needed SELinux rules requests on the parent packages

Comment 14 R P Herrold 2018-02-13 19:17:50 UTC
The needed additional permissions for SELinux turn out to have had some hidden additional parts needed, and I believe a better rule is:


[root@router ~]# cat my-grep.te

module my-grep 1.0;

require {
        type iptables_t;
        type modules_conf_t;
        class file { getattr ioctl open read };
}

#============= iptables_t ==============

#!!!! This avc is allowed in the current policy
allow iptables_t modules_conf_t:file { getattr ioctl open read };
[root@router ~]#

Comment 16 errata-xmlrpc 2018-04-10 11:28:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0715