RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2005195 - running sosreport creates iptables built-in tables in nftables
Summary: running sosreport creates iptables built-in tables in nftables
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sos
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: Miroslav Hradílek
URL:
Whiteboard:
Depends On:
Blocks: 2011536
TreeView+ depends on / blocked
 
Reported: 2021-09-17 05:11 UTC by Christian Horn
Modified: 2024-12-20 21:06 UTC (History)
10 users (show)

Fixed In Version: sos-4.2-2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2011536 (view as bug list)
Environment:
Last Closed: 2022-05-10 15:16:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:
chorn: needinfo+
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github sosreport sos pull 2724 0 None open [firewall_tables] call iptables -t <table> based on nft list 2021-10-14 21:13:55 UTC
Red Hat Issue Tracker RHELPLAN-97411 0 None None None 2021-09-17 05:13:40 UTC
Red Hat Knowledge Base (Solution) 6337221 0 None None None 2021-09-17 05:23:37 UTC
Red Hat Product Errata RHEA-2022:1998 0 None None None 2022-05-10 15:16:58 UTC

Description Christian Horn 2021-09-17 05:11:14 UTC 
Description of problem:
running sosreport creates iptables built-in tables in nftables

When generating sosreport, these commands are executed in networking plugin:
...
2021-09-06 11:43:15,893 INFO: [plugin:networking] collecting output of 'ip6tables -t nat -nvL'
2021-09-06 11:43:15,908 INFO: [plugin:networking] collecting output of 'ip6tables -t mangle -nvL'
2021-09-06 11:43:15,923 INFO: [plugin:networking] collecting output of 'ip6tables -t filter -nvL'
2021-09-06 11:43:16,225 INFO: [plugin:networking] collecting output of 'iptables -vnxL'
2021-09-06 11:43:16,242 INFO: [plugin:networking] collecting output of 'ip6tables -vnxL'

The problem is that these calls create iptables compatibility built-in tables in nft, which may not be desired. 

For example:
# nft list ruleset | grep table
table ip v4 {
table ip6 v6 {
table ip6 nat {
# ip6tables -t mangle -nvL > /dev/null
# nft list ruleset | grep table
table ip v4 {
table ip6 v6 {
table ip6 nat {
table ip6 mangle { // << NOTE

// the compat tables are added by iptables code:
int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
          int rulenum, unsigned int format)
{
    const struct nft_family_ops *ops = h->ops;
    struct nftnl_chain_list *list;
    struct nftnl_chain_list_iter *iter;
    struct nftnl_chain *c;
    bool found = false;

    nft_xt_builtin_init(h, table);
    nft_assert_table_compatible(h, table, chain);
...

static int nft_xt_builtin_init(struct nft_handle *h, const char *table)
{
    const struct builtin_table *t;
    t = nft_table_builtin_find(h, table);
    if (t == NULL)
        return -1;
    if (nft_table_initialized(h, t->type))
        return 0;
    if (nft_table_builtin_add(h, t) < 0)
        return -1;
    nft_chain_builtin_init(h, t);
    h->cache->table[t->type].initialized = true;
    return 0;
}

// so no way to bypass that - iptables/ip6tables must not be called to avoid creation of compat tables

// looking at sosreport upstream
https://github.com/sosreport/sos/blame/main/sos/report/plugins/networking.py

// uses: /usr/sbin/iptables-save (which doesn't create compat tables), the rest is in ufw plugin
// sos-4.1 removes all the iptables calls (except for iptable-save) from networking and adds ufw.py

// in 4.0:
// https://github.com/sosreport/sos/blob/4.0/sos/report/plugins/networking.py
        # collect the same for ip6tables
        try:
            ip_tables_names = open("/proc/net/ip6_tables_names").read()
        except IOError:
            ip_tables_names = "nat\nmangle\nfilter\n"
        for table in ip_tables_names.splitlines():
            self.collect_ip6table(table)


// the collects have predicates on ip*table_* kernel module
    def collect_iptable(self, tablename):
        """ Collecting iptables rules for a table loads either kernel module
        of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
        If neither module is present, the rules must be empty."""

        modname = "iptable_" + tablename
        cmd = "iptables -t " + tablename + " -nvL"
        self.add_cmd_output(
            cmd,
            pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))

    def collect_ip6table(self, tablename):
        """ Same as function above, but for ipv6 """

        modname = "ip6table_" + tablename
        cmd = "ip6tables -t " + tablename + " -nvL"
        self.add_cmd_output(
            cmd,
            pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))

// the default predicate is 'any' so we run the command if ip[6]table_xyz OR nf_tables is installed

Customer monitors nftables and doesn't want the compat tables. They get security alert on every sosreport run. Plus compat tables may have (little) performance impact (as they connect some chain to hooks and have some counters too).
https://wiki.nftables.org/wiki-nftables/index.php/Main_differences_with_iptables

I believe sosreport should avoid calling iptables command if iptables are not really in use. 

We do have possible workaround - skip networking plugin or replace iptables binary (removing is not an option due to dependencies). But the current behavior is incorrect and should be addressed properly.
Comment 1 Christian Horn 2021-09-17 05:14:45 UTC 
I looked at freshly deployed rhel8 without nft rules,
after running "sosreport":

- rhel8.1 and 8.2: still have no nft tables ("nft list tables" has no output)
- rhel8.3 and 8.3: have nft tables.

We have customers who monitor their rules, so this is not desired.
One of the rules of sosreport is also "leave no trace", IIRC.
Comment 2 Pavel Moravec 2021-09-17 16:15:28 UTC 
(I expect the root cause here is *different* than in https://bugzilla.redhat.com/show_bug.cgi?id=2001096#c10 ?)

Be aware, firewall_tables in upstream has separated the iptables commands from networking plugin. I expect the same applies there?

> I believe sosreport should avoid calling iptables command if iptables are not really in use. 

How to identify that "iptables are not really in use"? By some nft command output ("nft list ruleset | grep table ip")? Such that we should extend the predicate of an ip[|6]tables command by *also* that command output match?
Comment 3 Christian Horn 2021-09-21 07:59:18 UTC 
Doh, how could I oversee bz2001096 ?  It's in the same area indeed..

> How to identify that "iptables are not really in use"? By some nft
> command output ("nft list ruleset | grep table ip")? Such that we
> should extend the predicate of an ip[|6]tables command by *also*
> that command output match?

So my idea was to check as specific as possible before
each of the commands, like this in pseudo code:

if ( nft list ruleset | grep -q 'table ip6 mangle' )
  ip6tables -t mangle -nvL
if ( nft list ruleset | grep -q 'table ip6 net' )
  ip6tables -t nat -nvL
if ( nft list ruleset | grep -q 'table ip6 filter' )
  ip6tables -t filter -nvL
if ( nft list ruleset | grep -q 'table ip filter' )
  iptables -t filter -nvL
if ( nft list ruleset | grep -q 'table ip6 filter' )
  ip6tables -t filter -nvL

Our partner points out that grep -q is sometimes not
nice, so this here is more direct and might be less
error prone for checking (i.e. when table filter2
also exists, and we do not want to use regex:

[root@rhel8 ~]# nft -nt list table ip mangle
table ip mangle {
...
[root@rhel8 ~]# echo $?
0
[root@rhel8 ~]# nft delete table ip mangle
[root@rhel8 ~]# nft -nt list table ip mangle
Error: No such file or directory; did you mean table ‘mangle’ in family ip6?
list table ip mangle
              ^^^^^^
[root@rhel8 ~]# echo $?
1

We can then redirect STDOUT and STDERR from the
"nft -nt list table" to /dev/null, as we just need the
return code.
Comment 4 Pavel Moravec 2021-10-11 07:56:54 UTC 
Hm.. I see the later test similarly error-prone (can be subjective opinion) and also more difficult to implement due to the way how sos handles stderr (it cant distinguish it, just either ignore it or collect altogether with stdout). For the first test, we can "grep -q 'table ip6 mangle {'" to deal with tables named filter2 or similar. Also, it will be faster to check as we already call "nft list ruleset" cmdoutput (well, in a different plugin which might be a problem to carry over), that we can just properly parse.

Until there is objection to this, I will propose a PR in few days.
Comment 5 Christian Horn 2021-10-12 02:37:33 UTC 
Let's go with your approach then.
Comment 6 Pavel Moravec 2021-10-14 21:13:56 UTC 
I created https://github.com/sosreport/sos/pull/2724 for this, kindly please review (i.e. if "nft list ruleset" does not list a table, we wont call iptables -t <table> for it, right? or does this apply to mangle and filter *only*?)
Comment 7 Christian Horn 2021-10-15 06:52:11 UTC 
(In reply to Pavel Moravec from comment #6)
> I created https://github.com/sosreport/sos/pull/2724 for this, kindly please
> review (i.e. if "nft list ruleset" does not list a table, we wont call
> iptables -t <table> for it, right? or does this apply to mangle and filter
> *only*?)

We know that to the least also table "ip6 nat" gets created from running
sosreport on rhel8.
I will wait for a build on https://github.com/sosreport/sos/pull/2724 to
succeed and then give it a try.
Comment 8 Christian Horn 2021-10-25 08:52:20 UTC 
We have a fix in upstream which looks good on Fed34/35 and rhel8.4/8.5.
Comment 9 Christian Horn 2021-10-25 09:49:38 UTC 
(In reply to Christian Horn from comment #8)
> We have a fix in upstream which looks good on Fed34/35 and rhel8.4/8.5.
s/in upstream/in the upstream issue/
Comment 10 Pavel Moravec 2021-11-03 12:05:07 UTC 
Reproducer steps:

nft flush ruleset   # to flush all nft rules

nft list ruleset   # to see no rules remained

sos report

nft list ruleset   # to see no rules were added
Comment 16 errata-xmlrpc 2022-05-10 15:16:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1998
Note You need to log in before you can comment on or make changes to this bug.