Bug 2228642

Summary: sosreport does not gather iptables or nftables rules
Product: Red Hat Enterprise Linux 8 Reporter: Jonathan Maxwell <jmaxwell>
Component: sosAssignee: Jose Castillo <jcastillo>
Status: NEW --- QA Contact: Supportability QE <supportability-qe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.8CC: agk, jcastillo, jjansky, plambri, pmoravec, sbradley, theute
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jonathan Maxwell 2023-08-03 00:51:16 UTC
Description of problem:

Generating a sosreport does not gather iptables/nftables rules. 

Version-Release number of selected component (if applicable):

RHEl8 and RHEL9

How reproducible:

Always

Steps to Reproduce:
1. Add an iptables rule:

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

This will result in iptables and nftables rules:

# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere            

# nft list ruleset
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		oifname "eth0" counter packets 0 bytes 0 masquerade 
	}
}


2. Generate a sosreport. 
3. The sosreport/sos_commands/networking will not contain iptables/nftables rules. This used to work in RHEL7 and below.

Actual results:

No iptables/nftables rules.

Expected results:

if there iptables/nftables rule include them in sos_commands/networking

Additional info:

I suspect this because of iptables-nft.

# iptables -v
iptables v1.8.4 (nf_tables): no command specified

RHEL9 has the same results.

Comment 1 Jose Castillo 2023-08-03 08:42:02 UTC
Thank you for the bug report Jonathan! These rules may have been gathered by the plugin firewall_tables:


# cd sos_commands/firewall_tables/
[root@localhost firewall_tables]# ll
total 40
-rw-r--r--. 1 root root   422 Aug  3 04:38 ip6tables_-t_filter_-nvL
-rw-r--r--. 1 root root   718 Aug  3 04:38 ip6tables_-t_mangle_-nvL
-rw-r--r--. 1 root root   446 Aug  3 04:38 ip6tables_-vnxL
-rw-r--r--. 1 root root   424 Aug  3 04:38 iptables_-t_filter_-nvL
-rw-r--r--. 1 root root   720 Aug  3 04:38 iptables_-t_mangle_-nvL
-rw-r--r--. 1 root root   462 Aug  3 04:38 iptables_-vnxL
-rw-r--r--. 1 root root 13475 Aug  3 04:38 nft_list_ruleset

Could you check that directory and tell us if you see the missing rules there?

Comment 2 Jonathan Maxwell 2023-08-09 22:57:08 UTC
(In reply to Jose Castillo from comment #1)
> Thank you for the bug report Jonathan! These rules may have been gathered by
> the plugin firewall_tables:
> 
> 
> # cd sos_commands/firewall_tables/
> [root@localhost firewall_tables]# ll
> total 40
> -rw-r--r--. 1 root root   422 Aug  3 04:38 ip6tables_-t_filter_-nvL
> -rw-r--r--. 1 root root   718 Aug  3 04:38 ip6tables_-t_mangle_-nvL
> -rw-r--r--. 1 root root   446 Aug  3 04:38 ip6tables_-vnxL
> -rw-r--r--. 1 root root   424 Aug  3 04:38 iptables_-t_filter_-nvL
> -rw-r--r--. 1 root root   720 Aug  3 04:38 iptables_-t_mangle_-nvL
> -rw-r--r--. 1 root root   462 Aug  3 04:38 iptables_-vnxL
> -rw-r--r--. 1 root root 13475 Aug  3 04:38 nft_list_ruleset
> 
> Could you check that directory and tell us if you see the missing rules
> there?

Thanks Jose, yes my bad I didn't realize that directory had changed.

One thing I noticed is that it doesn't gather "iptables -t nat -nvL" can that be added?

Regards

Jon

Comment 3 Jose Castillo 2023-08-10 07:23:04 UTC
Yes it can be added. I'll send a PR upstream with this new command.

Comment 4 Pavel Moravec 2023-08-10 07:46:46 UTC
(In reply to Jonathan Maxwell from comment #2)
> One thing I noticed is that it doesn't gather "iptables -t nat -nvL" can
> that be added?

It should be added, usually. See https://github.com/sosreport/sos/blob/main/sos/report/plugins/firewall_tables.py#L31-L40 .

Two possible reasons why it was not collected:
1) iptable_nat or nf_tables kernel module was not loaded. Then sos can't call "iptables -t nat -nvL" as that would load the missing kmod, and sos should not alter system. If you want to allow that, use sosreport's option --allow-system-changes .

2) sos recognized there is no "nat" table, *maybe* due to an exception hit, see https://github.com/sosreport/sos/blob/main/sos/report/plugins/firewall_tables.py#L78-L85 . Is e.g. "nat" contained in /proc/net/ip_tables_names ?

Comment 5 Jose Castillo 2023-08-10 08:26:10 UTC
(In reply to Pavel Moravec from comment #4)
> (In reply to Jonathan Maxwell from comment #2)
> > One thing I noticed is that it doesn't gather "iptables -t nat -nvL" can
> > that be added?
> 
> It should be added, usually. See
> https://github.com/sosreport/sos/blob/main/sos/report/plugins/
> firewall_tables.py#L31-L40 .
> 
> Two possible reasons why it was not collected:
> 1) iptable_nat or nf_tables kernel module was not loaded. Then sos can't
> call "iptables -t nat -nvL" as that would load the missing kmod, and sos
> should not alter system. If you want to allow that, use sosreport's option
> --allow-system-changes .
> 
> 2) sos recognized there is no "nat" table, *maybe* due to an exception hit,
> see
> https://github.com/sosreport/sos/blob/main/sos/report/plugins/
> firewall_tables.py#L78-L85 . Is e.g. "nat" contained in
> /proc/net/ip_tables_names ?

Jon, do you have a sos that we can look at to see if it's one of these reasons?

Comment 6 Jonathan Maxwell 2023-08-11 04:58:42 UTC
(In reply to Pavel Moravec from comment #4)
> (In reply to Jonathan Maxwell from comment #2)
> > One thing I noticed is that it doesn't gather "iptables -t nat -nvL" can
> > that be added?
> 
> It should be added, usually. See
> https://github.com/sosreport/sos/blob/main/sos/report/plugins/
> firewall_tables.py#L31-L40 .
> 
> Two possible reasons why it was not collected:
> 1) iptable_nat or nf_tables kernel module was not loaded. Then sos can't
> call "iptables -t nat -nvL" as that would load the missing kmod, and sos
> should not alter system. If you want to allow that, use sosreport's option
> --allow-system-changes .

It still doesn't take iptables_-t_nat_-nvL

ip6tables_-t_filter_-nvL  ip6tables_-vnxL          iptables_-t_mangle_-nvL  nft_list_ruleset
ip6tables_-t_mangle_-nvL  iptables_-t_filter_-nvL  iptables_-vnxL

> 
> 2) sos recognized there is no "nat" table, *maybe* due to an exception hit,
> see
> https://github.com/sosreport/sos/blob/main/sos/report/plugins/
> firewall_tables.py#L78-L85 . Is e.g. "nat" contained in
> /proc/net/ip_tables_names ?

This is a RHEL8 system that uses the nftables backend.

# cat /proc/net/ip_tables_name
sat: /proc/net/ip_tables_names: No such file or directory

# grep nat lsmod
nft_chain_nat          16384  8
nf_nat                 45056  2 ipt_MASQUERADE,nft_chain_nat
nf_conntrack          172032  4 xt_conntrack,nf_nat,ipt_MASQUERADE,nf_conntrack_netlink
nf_tables             180224  110 nft_compat,nft_counter,nft_chain_nat
libcrc32c              16384  4 nf_conntrack,nf_nat,nf_tables,xfs

Comment 9 Jose Castillo 2023-08-11 08:51:36 UTC
The problem is indeed here:

        default_ip_tables = "mangle\nfilter\n"
        try:
            ip_tables_names = open("/proc/net/ip_tables_names").read()
        except IOError:
            ip_tables_names = default_ip_tables


Two possible solutions:

1) we add 'nat' to default_ip_tables (tested, works perfectly, but not a fan of this one), or
2) We find another way to get ip_tables_names (which may be better, I think). 

Regarding 2), actually we collect the list from non-existent /proc/net/ip_tables_names, but we then use this info to iterate through the output of nft_list:

        for table in ip_tables_names.splitlines():
            if nft_list['status'] == 0 and table in nft_ip_tables['ip6']:
                self.collect_ip6table(table)

So we fall back into default_ip_tables always. I'm testing now a possible fix - more on this soon.

Comment 10 Jose Castillo 2023-08-11 10:34:46 UTC
Just wondering - will there be a situation where the content of /proc/net/ip_tables_names and 'nft tables list' would differ?
In fact, checking in fedora, /proc/net/ip_tables_names exists but it's empty - is it deprecated?

Comment 11 Jose Castillo 2023-08-11 12:27:57 UTC
PR submitted https://github.com/sosreport/sos/pull/3335

The fix works and it's very simple, basically:

diff --git a/sos/report/plugins/firewall_tables.py b/sos/report/plugins/firewall_tables.py
index bec97438..4538aebb 100644
--- a/sos/report/plugins/firewall_tables.py
+++ b/sos/report/plugins/firewall_tables.py
@@ -72,24 +72,12 @@ class firewall_tables(Plugin, IndependentPlugin):
             if len(words) == 3 and words[0] == 'table' and \
                     words[1] in nft_ip_tables.keys():
                 nft_ip_tables[words[1]].append(words[2])
-        # collect iptables -t for any existing table, if we can't read the
-        # tables, collect 2 default ones (mangle, filter)
-        # do collect them only when relevant nft list ruleset exists
-        default_ip_tables = "mangle\nfilter\n"
-        try:
-            ip_tables_names = open("/proc/net/ip_tables_names").read()
-        except IOError:
-            ip_tables_names = default_ip_tables
-        for table in ip_tables_names.splitlines():
-            if nft_list['status'] == 0 and table in nft_ip_tables['ip']:
+        # collect iptables -t for any existing table
+        if nft_list['status'] == 0:
+            for table in nft_ip_tables['ip']:
                 self.collect_iptable(table)
-        # collect the same for ip6tables
-        try:
-            ip_tables_names = open("/proc/net/ip6_tables_names").read()
-        except IOError:
-            ip_tables_names = default_ip_tables
-        for table in ip_tables_names.splitlines():
-            if nft_list['status'] == 0 and table in nft_ip_tables['ip6']:
+            # collect the same for ip6tables
+            for table in nft_ip_tables['ip6']:
                 self.collect_ip6table(table)
 
         # When iptables is called it will load:


Jon, can you test it as well?