Bug 2011538

+++ This bug was initially created as a clone of Bug #2001096 +++

I have an sosreport from a customer gathered on "Red Hat Enterprise Linux
CoreOS release 4.7" (according to etc/redhat-release). The described issue
should apply to plain RHEL8+, though.

Installed sos is 4.0-11.el8, I hope this ticket is still relevant.

Curious why iptables dumps are missing from init-ns (there are logs for 'ip netns exec <something> iptables-save') I looked at sos.log:

| 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat.
| 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'ip6tables -t mangle -nvL': required kmods missing: ip6table_mangle, nf_tables.
| 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'ip6tables -t filter -nvL': required kmods missing: nf_tables, ip6table_filter.
| 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'nft list ruleset': required kmods missing: nf_tables.
| 2021-09-02 15:48:03,938 WARNING: [plugin:networking] skipped command 'iptables -vnxL': required kmods missing: nf_tables, iptable_filter.
| 2021-09-02 15:48:03,938 WARNING: [plugin:networking] skipped command 'ip6tables -vnxL': required kmods missing: nf_tables, ip6table_filter.

There are two problems with the above:

1) nf_tables module is in fact loaded, sos_commands/kernel/lsmod lists it.

2) The other modules are not required at all: In RHEL8 onwards, iptables-nft is
   used which internally creates an nftables ruleset calling legacy iptables
   extensions via special nftables expressions. In fact, a module like
   iptable_filter would indicate use of legacy iptables (in a container or so),
   but RHEL8 doesn't ship legacy iptables so there's no way to gather that data
   for sos.

In general, I don't quite understand the need to check for kernel modules in
the first place. It is true that 'iptables -vnL' may cause module loading which
is unwanted, but using iptables-save does not and it is a better way of
gathering the installed ruleset.

So I suggest replacing all the above iptables/ip6tables calls by single calls to
iptables-save and ip6tables-save.

If module loading is to be avoided, checking nf_tables is needed: 'nft list
ruleset' causes the kernel to load it and so do ip*tables-save on RHEL8+ due to
the nftables backend.

Note that despite of the above, iptables-save is called within the detected
network namespaces. As said, this would cause nf_tables module to be loaded if
not present already.

--- Additional comment from Pavel Moravec on 2021-09-04 10:16:44 UTC ---

Hello,
we indeed guard execution of some commands if they can load a kernel module (or enable a service or similar). Since one key request/feature to sos is _not_ altering the system at all.

1) "sos prevents 'nft list ruleset' claiming nf_tables isnt loaded, but lsmod shows it" 
There seems to be a bug in either detection of the module, or sos executed a command that loaded the module, or maybe the module was loaded independently on sos, during sos execution. Could you please provide the sosreport where you noticed this? (case number should sufficient)


2) "ip*table_* modules are not loaded by "ip[|6]tables -L .." commands on RHEL8+, so no reason to guard the commands that way"

This has been fixed that way in https://github.com/sosreport/sos/commit/31e0467885ef3986b476ea070941a786b426f298#diff-2de6e8c3c90c8908b74f5ba71a45dd9cf0066f1e60193da0f3a1923c32b6ef37 . That change is present in sos-4.0 and I can confirm so:

# rpm -qf /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
sos-4.0-11.el8.noarch
# cat /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
..
    def collect_iptable(self, tablename):
        """ Collecting iptables rules for a table loads either kernel module
        of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
        If neither module is present, the rules must be empty."""

        modname = "iptable_" + tablename
        cmd = "iptables -t " + tablename + " -nvL"
        self.add_cmd_output(
            cmd,
            pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
..

And I can confirm that behaviour (well, on RHEL8, not CoreOS) that having nf_tables loaded but not ip*tables_* ones, iptables are collected.

Was really sos-4.0-11 used? (grep sos installed-packages)


3) "Replacing calls by one command iptables-save"
I assume it was meant as a workaround or a way of improvement the 2) issue that should be resolved. Also I think the current format is more human-readable (until there is a request for machine processing output, to e.g. load the rules to my system to set them up..?) and splitting the output per tables (nat,filter,..) is for the sake of readiness. Anyway if you feel otherwise, let describe the benefits of the "ip*tables-save" option and we can discussi it either here or rather in upstream (https://github.com/sosreport/sos/issues) - as the change would affect also other distros.


TL;DR: please provide sosreport from point 1), points 2) and 3) should be resolved/irelevant.

--- Additional comment from Phil Sutter on 2021-09-06 12:50:51 UTC ---

Hi Pavel,

(In reply to Pavel Moravec from comment #1)
> we indeed guard execution of some commands if they can load a kernel module
> (or enable a service or similar). Since one key request/feature to sos is
> _not_ altering the system at all.

Yes, makes sense.

> 1) "sos prevents 'nft list ruleset' claiming nf_tables isnt loaded, but
> lsmod shows it" 
> There seems to be a bug in either detection of the module, or sos executed a
> command that loaded the module, or maybe the module was loaded independently
> on sos, during sos execution. Could you please provide the sosreport where
> you noticed this? (case number should sufficient)

Case number is 03000256. The file I inspected is
sosreport-tc06-5vrvn-worker-m42xlarge8-eu-de-02-l9fqx-03000256-2021-09-02-bzqreun.tar.xz,
in case there are multiple.

> 2) "ip*table_* modules are not loaded by "ip[|6]tables -L .." commands on
> RHEL8+, so no reason to guard the commands that way"
> 
> This has been fixed that way in
> https://github.com/sosreport/sos/commit/
> 31e0467885ef3986b476ea070941a786b426f298#diff-
> 2de6e8c3c90c8908b74f5ba71a45dd9cf0066f1e60193da0f3a1923c32b6ef37 . That
> change is present in sos-4.0 and I can confirm so:
> 
> # rpm -qf /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
> sos-4.0-11.el8.noarch
> # cat /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
> ..
>     def collect_iptable(self, tablename):
>         """ Collecting iptables rules for a table loads either kernel module
>         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
>         If neither module is present, the rules must be empty."""
> 
>         modname = "iptable_" + tablename
>         cmd = "iptables -t " + tablename + " -nvL"
>         self.add_cmd_output(
>             cmd,
>             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> ..
> 
> And I can confirm that behaviour (well, on RHEL8, not CoreOS) that having
> nf_tables loaded but not ip*tables_* ones, iptables are collected.

How do you prevent the same from happening on RHEL7? There may be nf_tables
loaded but ip*table_* not. Calling iptables then would load the latter.

> Was really sos-4.0-11 used? (grep sos installed-packages)

sos is not listed in 'installed-rpms'. But I found it:

| % grep sos sos_commands/yum/yum_list_installed
| sos.noarch                                    4.0-11.el8                           @RHEL-8.4.0-updates-20210803.2-BaseOS-1

> 3) "Replacing calls by one command iptables-save"
> I assume it was meant as a workaround or a way of improvement the 2) issue
> that should be resolved. Also I think the current format is more
> human-readable (until there is a request for machine processing output, to
> e.g. load the rules to my system to set them up..?) and splitting the output
> per tables (nat,filter,..) is for the sake of readiness. Anyway if you feel
> otherwise, let describe the benefits of the "ip*tables-save" option and we
> can discussi it either here or rather in upstream
> (https://github.com/sosreport/sos/issues) - as the change would affect also
> other distros.

No, if you consider -L output preferrable and are willing to deal with the
implications I won't object. To me it doesn't matter much, but I'm used to both
formats. I pointed at ip*tables-save because it was designed to not cause
module loads (OK, nf_tables breaks that) and therefore might simplify sos.

> TL;DR: please provide sosreport from point 1), points 2) and 3) should be
> resolved/irelevant.

Thanks, Phil

--- Additional comment from Pavel Moravec on 2021-09-06 16:16:32 UTC ---

Thanks for prompt feedback. And sorry that I was blind:

>     def collect_iptable(self, tablename):
>         """ Collecting iptables rules for a table loads either kernel module
>         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
>         If neither module is present, the rules must be empty."""
> 
>         modname = "iptable_" + tablename
>         cmd = "iptables -t " + tablename + " -nvL"
>         self.add_cmd_output(
>             cmd,
>             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))

so for table name "filter", we guard the "iptables -t filter -nvL" command by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I overlooked the "modname" here? /o\).

Indeed, we need to distingusih RHEL7 and RHEL8+ - or rather per kernel version(?) that changed this behaviour? Note that sos runs on more distros, so we can't decide based on RHEL version (only).


Do you know a distro-independent test whether  "ip*table_* modules will be loaded by "ip[|6]tables -L .." commands on given system? (I can ask concurrently as well)


> No, if you consider -L output preferrable and are willing to deal with the
> implications I won't object. To me it doesn't matter much, but I'm used to both
> formats. I pointed at ip*tables-save because it was designed to not cause
> module loads (OK, nf_tables breaks that) and therefore might simplify sos.

As the current code will make more complicated due to the "what module might be loaded?" decision, it makes sense to move to the ip*tables-save command - I will discuss this with my colleagues / in sos upstream.

--- Additional comment from Phil Sutter on 2021-09-07 11:08:55 UTC ---

Hi,

(In reply to Pavel Moravec from comment #3)
> Thanks for prompt feedback. And sorry that I was blind:
> 
> >     def collect_iptable(self, tablename):
> >         """ Collecting iptables rules for a table loads either kernel module
> >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> >         If neither module is present, the rules must be empty."""
> > 
> >         modname = "iptable_" + tablename
> >         cmd = "iptables -t " + tablename + " -nvL"
> >         self.add_cmd_output(
> >             cmd,
> >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> 
> so for table name "filter", we guard the "iptables -t filter -nvL" command
> by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> overlooked the "modname" here? /o\).

If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"] defaults
to "any", so either modname or 'nf_tables' should suffice. But the log
indicates that's not the case. Maybe I missed a spot where required["kmods"] is
set to "all".

> Indeed, we need to distingusih RHEL7 and RHEL8+ - or rather per kernel
> version(?) that changed this behaviour? Note that sos runs on more distros,
> so we can't decide based on RHEL version (only).
> 
> 
> Do you know a distro-independent test whether  "ip*table_* modules will be
> loaded by "ip[|6]tables -L .." commands on given system? (I can ask
> concurrently as well)

It actually depends on iptables-variant being used, not kernel version. So I
suggest testing 'iptables -V | grep -q nf_tables'. Older legacy iptables does
not contain 'legacy' in version output, so detecting iptables-nft is backwards
compatible. If that test passes, you need to check if nf_tables.ko is loaded.
If it fails, you can call iptables-save straightaway. Or keep the old report
format and continue to check for iptable_<name>.ko before calling 'iptables
-vnL -t <name>'. 

As a side-note: In Fedora, users may switch between legacy and nft iptables
using alternatives. Checking Fedora release or kernel version is obviously not
an option.

Another alternative could be to call 'iptables-legacy' and 'iptables-nft'
explicitly, with appropriate module checks in beforehand. In theory, a system
could have legacy ruleset in parallel with nft one (although that's discouraged
and almost always leads to disaster). But you'll have to prepare for
environments where only a plain 'iptables' command is available, and there
you'll have to check version output because it may be an old system with legacy
iptables only or RHEL8 on which we ship iptables-nft as iptables and don't
install the explicit name (my approach for RHEL8 is a mess in hindsight).

> > No, if you consider -L output preferrable and are willing to deal with the
> > implications I won't object. To me it doesn't matter much, but I'm used to both
> > formats. I pointed at ip*tables-save because it was designed to not cause
> > module loads (OK, nf_tables breaks that) and therefore might simplify sos.
> 
> As the current code will make more complicated due to the "what module might
> be loaded?" decision, it makes sense to move to the ip*tables-save command -
> I will discuss this with my colleagues / in sos upstream.

OK.

Cheers, Phil

--- Additional comment from Pavel Moravec on 2021-09-07 15:21:47 UTC ---

(In reply to Phil Sutter from comment #4)
> Hi,
> 
> (In reply to Pavel Moravec from comment #3)
> > Thanks for prompt feedback. And sorry that I was blind:
> > 
> > >     def collect_iptable(self, tablename):
> > >         """ Collecting iptables rules for a table loads either kernel module
> > >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> > >         If neither module is present, the rules must be empty."""
> > > 
> > >         modname = "iptable_" + tablename
> > >         cmd = "iptables -t " + tablename + " -nvL"
> > >         self.add_cmd_output(
> > >             cmd,
> > >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> > 
> > so for table name "filter", we guard the "iptables -t filter -nvL" command
> > by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> > overlooked the "modname" here? /o\).
> 
> If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"]
> defaults
> to "any", so either modname or 'nf_tables' should suffice. But the log
> indicates that's not the case. Maybe I missed a spot where required["kmods"]
> is
> set to "all".
> 

This seems to be the real culprit behind this bugzilla. The kmods is really "any", which means:

1) the kmods is set right - if either kmod is loaded, iptables will use *that* one (until one does something dirty).
2) as nf_tables was loaded (at some time) but not detected as loaded by sos (at the "right" time), sos skipped collection of various cmd outputs. Only due to this mismatch. WHy this happened?

At:
2021-09-02 15:47:59,614 INFO: [archive:TarFileArchive] initialised empty FileCacheArchive at '/host/var/tmp/sos.mga3nalv/sosreport-HOSTNAME-03000256-2021-09-02-bzqreun'

the sosreport was kicked off in the container (does the issue happen in some container only? it has the same modules like the guest, right?). Very short time later, it executes "lsmod":
https://github.com/sosreport/sos/blob/7d025933c2cdc7bf07a7b6b9c0d8b2637ade372b/sos/policies/distros/__init__.py#L139

to get the list of loaded kernel modules to compare in SoSPreset.

At:
2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat.

it is the 1st time when sos _reported_ the kmod isnt loaded.

At:
2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'

the plugin was already loaded.


So either there is some tricky bug in sos code, or the kernel module was loaded between 15:47:59 and 15:48:32 ? dmesg does not mention anything..

--- Additional comment from Phil Sutter on 2021-09-07 16:44:16 UTC ---

(In reply to Pavel Moravec from comment #5)
> (In reply to Phil Sutter from comment #4)
> > Hi,
> > 
> > (In reply to Pavel Moravec from comment #3)
> > > Thanks for prompt feedback. And sorry that I was blind:
> > > 
> > > >     def collect_iptable(self, tablename):
> > > >         """ Collecting iptables rules for a table loads either kernel module
> > > >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> > > >         If neither module is present, the rules must be empty."""
> > > > 
> > > >         modname = "iptable_" + tablename
> > > >         cmd = "iptables -t " + tablename + " -nvL"
> > > >         self.add_cmd_output(
> > > >             cmd,
> > > >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> > > 
> > > so for table name "filter", we guard the "iptables -t filter -nvL" command
> > > by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> > > overlooked the "modname" here? /o\).
> > 
> > If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"]
> > defaults
> > to "any", so either modname or 'nf_tables' should suffice. But the log
> > indicates that's not the case. Maybe I missed a spot where required["kmods"]
> > is
> > set to "all".
> > 
> 
> This seems to be the real culprit behind this bugzilla. The kmods is really
> "any", which means:
> 
> 1) the kmods is set right - if either kmod is loaded, iptables will use
> *that* one (until one does something dirty).
> 2) as nf_tables was loaded (at some time) but not detected as loaded by sos
> (at the "right" time), sos skipped collection of various cmd outputs. Only
> due to this mismatch. WHy this happened?
> 
> At:
> 2021-09-02 15:47:59,614 INFO: [archive:TarFileArchive] initialised empty
> FileCacheArchive at
> '/host/var/tmp/sos.mga3nalv/sosreport-HOSTNAME-03000256-2021-09-02-bzqreun'
> 
> the sosreport was kicked off in the container (does the issue happen in some
> container only? it has the same modules like the guest, right?). Very short
> time later, it executes "lsmod":
> https://github.com/sosreport/sos/blob/
> 7d025933c2cdc7bf07a7b6b9c0d8b2637ade372b/sos/policies/distros/__init__.
> py#L139
> 
> to get the list of loaded kernel modules to compare in SoSPreset.
> 
> At:
> 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command
> 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat.
> 
> it is the 1st time when sos _reported_ the kmod isnt loaded.

And then there is:

| 2021-09-02 15:48:03,980 INFO: [plugin:networking] added cmd output 'ip netns exec c4e2b27a-a821-41ca-bf4a-fec375e935a1 iptables-save'

If nf_tables.ko wasn't loaded before, that should have caused it. It is not the
first per-netns iptables-save call, but the first one which returns data
instead of error due to invalid netns name.

But you can't close the case yet, the output from that command shows there are
rules present. So nf_tables.ko was very likely loaded already by the time sos
was called.

> At:
> 2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'
> 
> the plugin was already loaded.
> 
> 
> So either there is some tricky bug in sos code, or the kernel module was
> loaded between 15:47:59 and 15:48:32 ? dmesg does not mention anything..

Loading nf_tables.ko is silent in dmesg, so nothing to gain there. :(

--- Additional comment from Pavel Moravec on 2021-09-08 09:58:27 UTC ---

It was / (still is) tricky:

All other predicates were evaluated by the sos execution properly. "just" nf_tables was mis-detected. This supports the theory the kernel module was loaded even after starting sosreport.

And indeed, commands like

2021-09-02 15:48:53,945 INFO: [plugin:networking] collecting output of 'ip netns exec 1924b01d-71e2-4e34-8174-f1ce3e7efa4b iptables-save'

do load the kmod:

# ip netns add test
# lsmod | grep nf
nfnetlink              16384  0
binfmt_misc            20480  1
# ip netns exec test iptables-save
# lsmod | grep nf
nf_tables             147456  0
nfnetlink              16384  1 nf_tables
binfmt_misc            20480  1
#


BUT.. not yet done! Since why the kernel module was already loaded at:

2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'

when sos_commands/kernel/lsmod DOES contain :

nf_tables             167936  15986 nft_compat,nft_counter,nft_chain_nat

? Some _previous_ command had to load it.


$ grep "collecting output of" sos_logs/sos.log | cut -d\' -f2 | grep -n lsmod
554:lsmod
$

So the first 553 commands have to do so (until some other routine did so, see below).

$ grep -m553 "collecting output of" sos_logs/sos.log | cut -d\' -f2
atomic host status
..
rpm -V audit
$

(none of the command loads the kmod on my system)

If I rule out commands like "uname" or "rpm", here are possible "sinners":

- iscsiadm commands? (I doubt so)
- crictl [inspect|inspecti|inspectp|logs -t] ? could inspecting a kubernetes container load a kernel module? (I dont have simple means to test it)
- "ip netns exec 1924b01d-71e2-4e34-8174-f1ce3e7efa4b bpftool net list" - probably not (cant repro myself)
- some bpftool commands? can't reproduce
- some gluster command? probably not, I dont have means to verify it
- "atomic host status" - probably not


Apart of that, sos report invokes few other commands not directly logged - neither sound a probable one, though:
lsmod
systemctl list-unit-files --type=service
podman ps
podman images --format {{lower .Repository}}:{{lower .Tag}} {{lower .ID}}
podman volume ls
rpm -qa --queryformat '%{NAME}|%{VERSION}|%{RELEASE}\\n'
scl -l
scl -l
scl -l
abrt-cli list
alternatives --list
lsblk -f -a -l
dnf --assumeno module list --installed
bpftool -j prog list
bpftool -j map list
ip netns
gluster volume info
grub2-mkconfig --help
kpatch list
kubectl get namespaces
kubectl get nodes
ldconfig -p -N -X
vgdisplay -h
ip netns
nmcli --terse --fields RUNNING general status
nmcli --terse --fields NAME con
nmcli --terse --fields DEVICE dev
podman network ls
podman ps -a
runc list -q
/usr/sap/hostctrl/exe/saphostctrl -function ListInstances
/usr/sap/hostctrl/exe/saphostctrl -function ListDatabases
getenforce
systemctl status sssd
sssctl domain-list
systemctl status systemd-resolved
ip -o link
rpm -qal
vdo list --all
virt-what

(some few other cmds might be executed, depending on the given system - this is almost impossible to enumerate precisely, though).


TL;DR:
- "ip netns exec <namespace> iptables-save" commands must be guarded by nf_tables kmod (always? since some ip/nftables/kernel version?)
- there is most probably something else among sos commands being collected, that loads that kernel module, but I am unable to find it; please provide some reproducer or more hints to troubleshoot here

--- Additional comment from Pavel Moravec on 2021-09-08 11:09:32 UTC ---

> - "ip netns exec <namespace> iptables-save" commands must be guarded by nf_tables kmod (always? since some ip/nftables/kernel version?)

RHEL7 does not load a kernel module when executing that command, so we must come up with condition when the cmd can load the kmod / when to guard the cmd by that kmod predicate. Any idea?\


(btw for a reproducer on RHEL8, I had to stop firewalld service, remove the package and reboot; even then I got nf_tables kmod removed - worth to note for a reproducer)

--- Additional comment from Pavel Moravec on 2021-09-08 12:33:59 UTC ---

(on a similar topic, one related PR raised: https://github.com/sosreport/sos/pull/2679 )

--- Additional comment from Phil Sutter on 2021-09-08 13:41:08 UTC ---

Hi,

There is not only nf_tables loaded but also nft_compat, nft_counter and
nft_chain_nat which depend on it. I find it very unlikely that one of the sos
commands not only causes nf_tables to be loaded but also adds a rule to a chain
in nat table.

The problem reported around the discussed sosreport is that OCP calls 'iptables
-C' too often in a specific situation. Another reason why nf_tables very likely
was loaded already by the time sos got called.

I can't put my finger onto it, but am quite sure this is a bug within sos not
properly detecting nf_tables as loaded. I had thought about the underscore
being removed or something, but the code in sos/utilities.py and
sos/policies/distros/__init__.py looks OK to me.

'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
'iptables -V' output contains 'nf_tables'. This is the best way to find out
what variant is used.

Looking at 'iptables -V' output is my recommendation as elaborated on in
comment 4.

Cheers, Phil

--- Additional comment from Pavel Moravec on 2021-09-16 15:29:05 UTC ---

(In reply to Phil Sutter from comment #10)
> 'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
> 'iptables -V' output contains 'nf_tables'. This is the best way to find out
> what variant is used.

On a similar note: shouldn't we apply the same to ip6tables-save as well? I.e. "if 'ip6tables -V' contains nf_tables, then guard the 'ip netns exec <foo> ip6tables save' command by nf_tables kmod"?

--- Additional comment from Phil Sutter on 2021-09-16 16:11:05 UTC ---

(In reply to Pavel Moravec from comment #11)
> (In reply to Phil Sutter from comment #10)
> > 'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
> > 'iptables -V' output contains 'nf_tables'. This is the best way to find out
> > what variant is used.
> 
> On a similar note: shouldn't we apply the same to ip6tables-save as well?
> I.e. "if 'ip6tables -V' contains nf_tables, then guard the 'ip netns exec
> <foo> ip6tables save' command by nf_tables kmod"?

Yes, please, sorry for being a bit sloppy. The same holds for arptables and
ebtables, BTW. I don't recall if sos collects those, too.

--- Additional comment from Pavel Moravec on 2021-09-17 14:01:06 UTC ---

[root@pmoravec-rhel8 ~]# ip netns ls
test
[root@pmoravec-rhel8 ~]# lsmod | grep nf
[root@pmoravec-rhel8 ~]# ip netns exec test iptables-save
[root@pmoravec-rhel8 ~]# lsmod | grep nf
nf_tables             172032  0
nfnetlink              16384  1 nf_tables
libcrc32c              16384  2 nf_tables,xfs
[root@pmoravec-rhel8 ~]# rmmod nf_tables; lsmod | grep nf
nfnetlink              16384  0
[root@pmoravec-rhel8 ~]# ip netns exec test iptables-save
[root@pmoravec-rhel8 ~]# lsmod | grep nf
nf_tables             172032  0
nfnetlink              16384  1 nf_tables
libcrc32c              16384  2 nf_tables,xfs
[root@pmoravec-rhel8 ~]#

So the "ip netns exec .." command loads nf_tables which loads nfnetlink. So SoSPreset kmods=['nf_tables'] should be sufficient. That applies to both iptables-save and ip6tables-save.

--- Additional comment from Pavel Moravec on 2021-09-28 20:56:52 UTC ---

https://github.com/sosreport/sos/pull/2703 PR proposed to upstream.

--- Additional comment from Pavel Moravec on 2021-09-29 06:19:04 UTC ---

Reproducer steps for QE:

1) Have a RHEL8 with firewalld package removed (sometimes a reboot afterwards is required as well), which should allow you to have nf_tables kernel module unloaded:

dnf remove firewalld -y
lsmod | grep nf_tables  # shows nothing, optionally rmmod nf_tables if no reboot done

(also ensure iptables and ip6tables have nf_tables built in - this distinguishes behaviour between RHEL7 and RHEL8+ :)
iptables -V | grep nf_tables  # shows a match
ip6tables -V | grep nf_tables  # shows a match


2) Add a network namespace:

ip netns add test
ip netns ls


3) Collect networking plugin:

sos report -o networking --batch --build


4) Check that neither "ip netns test iptables-save" or "ip netns test ip6tables-save" is collected (when iptables have nf_tables built in)


5) Ensure nf_tables kmod is not loaded:

lsmod | grep nf_tables  # shows nothing


6) Manually enforce loading the kmod:

ip netns test iptables-save
lsmod | grep nf_tables  # shows the kmod loaded


7) Re-run the sos report and ensure the "ip netns test iptables-save" and "ip netns test ip6tables-save" is properly collected.