Bug 1774742

Summary: Rich rules with ipsets containing subnets cause nft segfault when starting firewalld
Product: Red Hat Enterprise Linux 8 Reporter: dbakken
Component: nftablesAssignee: Phil Sutter <psutter>
Status: CLOSED ERRATA QA Contact: Jiri Peska <jpeska>
Severity: high Docs Contact:
Priority: high    
Version: 8.1CC: atragler, dbakken, dbayly, egarver, jmaxwell, jpeska, lee.jnk, mabrown, marchenko, mihai, psutter, stephan.duehr, surkumar, todoleza, tsales, vhernand, vjadhav
Target Milestone: rcKeywords: ZStream
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nftables-0.9.3-7.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1802056 (view as bug list) Environment:
Last Closed: 2020-04-28 16:42:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1778558    
Bug Blocks: 1802056    

Description dbakken 2019-11-20 20:35:47 UTC
Description of problem:
Rich rules with ipsets containing subnets cause nft to segfault when starting firewalld.

Version-Release number of selected component (if applicable):
firewalld: 0.7.0-5.el8
nftables: 1:0.9.0-14.el8
kernel: 4.18.0-80.11.2.el8_0.x86_6

How reproducible:
Is triggered by creating a rich rule referencing an ipset with a subnet address in CIDR notation. A segfault occurs when firewalld uses the nftables backend, but does not occur with the iptables backend.

Steps to Reproduce:
1. Create /etc/firewalld/ipsets/example.xml:
<?xml version="1.0" encoding="utf-8"?>
<ipset type="hash:net">
  <entry>10.10.123.0/24</entry>
  <entry>10.9.9.1</entry>
</ipset>

2. Add a rich rule using the above ipset to /etc/firewalld/zones/public.xml:
  <rule family="ipv4">
    <source ipset="example"/>
    <port port="22" protocol="tcp"/>
    <accept/>
  </rule>

3. systemctl restart firewalld

Actual results:

systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 14:02:03 CST; 6min ago
...
Starting firewalld - dynamic firewall daemon...
Started firewalld - dynamic firewall daemon.
systemd-coredump[17331]: Process 17329 (nft) of user 0 dumped core.

/var/log/messages:

kernel: [4065676.510735] nft[17329]: segfault at 18 ip 00007fd322923189 sp 00007ffeb7bfdb98 error 4 in libgmp.so.10.3.2[7fd3228e7000+96000]
kernel: [4065676.510741] Code: 8b 4e 18 48 8d 76 10 4c 1b 42 10 4c 1b 4a 18 48 8d 52 10 48 8d 3f eb 77 4c 89 17 4c 89 5f 08 0f 92 c0 c3 a8 02 75 30 49 f7 d8 <4c> 8b 1e 4c 1b 1a e3 1e 4c 8b 46 08 4c 8b 4e 10 48 8d 76 08 48 8d
firewalld[16875]: ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public_allow meta nfproto ipv4 ip saddr @example tcp dport 22 ct state new,untracked accept' failed:
firewalld[16875]: ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public_allow meta nfproto ipv4 ip saddr @example tcp dport 22 ct state new,untracked accept' failed:
firewalld[16875]: ERROR: COMMAND_FAILED: '/usr/sbin/nft add rule inet firewalld filter_IN_public_allow meta nfproto ipv4 ip saddr @example tcp dport 22 ct state new,untracked accept' failed:
firewalld[16875]: ERROR: '/usr/sbin/nft insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public' failed: Error: Could not process rule: No such file or directory#012insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public#012^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
firewalld[16875]: ERROR: '/usr/sbin/nft insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public' failed: Error: Could not process rule: No such file or directory#012insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public#012^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
firewalld[16875]: ERROR: COMMAND_FAILED: '/usr/sbin/nft insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public' failed: Error: Could not process rule: No such file or directory#012insert rule inet firewalld raw_PREROUTING_ZONES iifname "ens192" goto raw_PRE_public#012^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Expected results:
Firewalld should not cause nft to segfault when creating rich rules.
Removing the first entry (10.10.123.0/24) from the example ipset or changing it to a host address (i.e., 10.10.123.1) resolves the error, and nft does not segfault.
Changing FirewallBackend to iptables in firewalld.conf also resolves the error.

Additional info:
May be related to CentOS bug https://bugs.centos.org/view.php?id=16518.

Comment 1 Eric Garver 2019-11-21 13:18:35 UTC
The logs clearly show nft segfault. Reassigning to nftables.

Comment 4 dbakken 2019-12-13 16:45:59 UTC
Any updates on when we can expect this segfault to be fixed in RHEL8?

Comment 9 Phil Sutter 2020-01-10 11:40:20 UTC
Upstream commits to backport:

commit 5d57fa3e99bb9f2044e236d4ddb7d874cfefe1dd
Author: Phil Sutter <phil>
Date:   Thu Jan 9 13:34:20 2020 +0100

    monitor: Do not decompose non-anonymous sets
    
    They have been decomposed already, trying to do that again causes a
    segfault. This is a similar fix as in commit 8ecb885589591 ("src:
    restore --echo with anonymous sets").
    
    Signed-off-by: Phil Sutter <phil>
    Acked-by: Pablo Neira Ayuso <pablo>

commit 02174ffad484d9711678e5d415c32307efc39857
Author: Phil Sutter <phil>
Date:   Thu Jan 9 17:43:11 2020 +0100

    monitor: Fix for use after free when printing map elements
    
    When populating the dummy set, 'data' field must be cloned just like
    'key' field.
    
    Fixes: 343a51702656a ("src: store expr, not dtype to track data in sets")
    Signed-off-by: Phil Sutter <phil>
    Acked-by: Pablo Neira Ayuso <pablo>

Comment 10 Phil Sutter 2020-01-10 16:44:07 UTC
Here's how to reproduce the issue firewalld exposed:

With a given ruleset of:

| table t {
| 	set s {
| 		type inet_service
| 		flags interval
| 		elements = { 20, 30-40 }
| 	}
| 	chain c {
| 	}
| }

Running the following command segfaults:

| nft --echo add rule t c tcp dport @s

This is caused by echo option handling code trying to decompose the referenced
interval set in cache although that had already happened before. The fix
contained in the first patch above avoids a call to interval_map_decompose() if
given set is not an anonymous one.

Echo code is shared with monitor code, so while the above command doesn't
segfault if --echo option is not used, it will crash a possibly running
instance of 'nft monitor'.

Requirements for reproducing are non-trivial, but not uncommon:

- set has interval flag
- set contains elements (not 100% sure, but at least a range is needed)
- nft is called with --echo option (or 'nft monitor' being used)

Comment 12 Phil Sutter 2020-01-10 18:57:53 UTC
(In reply to Phil Sutter from comment #9)
> Upstream commits to backport:
> 
[...]
> 
> commit 02174ffad484d9711678e5d415c32307efc39857
> Author: Phil Sutter <phil>
> Date:   Thu Jan 9 17:43:11 2020 +0100
> 
>     monitor: Fix for use after free when printing map elements
>     
>     When populating the dummy set, 'data' field must be cloned just like
>     'key' field.
>     
>     Fixes: 343a51702656a ("src: store expr, not dtype to track data in sets")
>     Signed-off-by: Phil Sutter <phil>
>     Acked-by: Pablo Neira Ayuso <pablo>

This one is in fact not needed, the commit it fixes is not present in RHEL8.

Comment 14 Phil Sutter 2020-01-13 14:09:38 UTC
Upstream fix is faulty, luckily covscan noticed it. I've sent a follow-up: http://patchwork.ozlabs.org/patch/1222144/

Comment 15 Phil Sutter 2020-01-13 15:57:49 UTC
Follow-up to backport:

commit ddbacd70d061eb1b6808f501969809bfb5d03001
Author: Phil Sutter <phil>
Date:   Mon Jan 13 14:53:24 2020 +0100

    monitor: Fix output for ranges in anonymous sets
    
    Previous fix for named interval sets was simply wrong: Instead of
    limiting decomposing to anonymous interval sets, it effectively disabled
    it entirely.
    
    Since code needs to check for both interval and anonymous bits
    separately, introduce set_is_interval() helper to keep the code
    readable.
    
    Also extend test case to assert ranges in anonymous sets are correctly
    printed by echo or monitor modes. Without this fix, range boundaries are
    printed as individual set elements.
    
    Fixes: 5d57fa3e99bb9 ("monitor: Do not decompose non-anonymous sets")
    Signed-off-by: Phil Sutter <phil>
    Reviewed-by: Pablo Neira Ayuso <pablo>

Comment 16 Phil Sutter 2020-01-17 16:10:09 UTC
*** Bug 1778558 has been marked as a duplicate of this bug. ***

Comment 17 Phil Sutter 2020-01-17 16:12:35 UTC
As per the process, requesting z-stream backport for this ticket to cover customer-case referenced in bug 1778558.

Comment 18 Thomas Stephen Lee 2020-01-30 04:34:16 UTC
Fixes work in RHEL 8.2 Beta
and
CentOS 8.1(packages rebuilt from Beta srpms)

It would be nice if we got the fix in RHEL 8.1 itself 😊.

thanks

Comment 25 Eric Garver 2020-02-11 22:25:49 UTC
*** Bug 1801861 has been marked as a duplicate of this bug. ***

Comment 29 Vlad 2020-03-03 15:59:28 UTC
any plans to backport the fix to 8.1?

Comment 30 Eric Garver 2020-03-03 17:51:57 UTC
(In reply to Vlad from comment #29)
> any plans to backport the fix to 8.1?

Yes. See bug 1802056.

Comment 33 errata-xmlrpc 2020-04-28 16:42:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1774