Bug 2001096

Hello,
we indeed guard execution of some commands if they can load a kernel module (or enable a service or similar). Since one key request/feature to sos is _not_ altering the system at all.

1) "sos prevents 'nft list ruleset' claiming nf_tables isnt loaded, but lsmod shows it" 
There seems to be a bug in either detection of the module, or sos executed a command that loaded the module, or maybe the module was loaded independently on sos, during sos execution. Could you please provide the sosreport where you noticed this? (case number should sufficient)


2) "ip*table_* modules are not loaded by "ip[|6]tables -L .." commands on RHEL8+, so no reason to guard the commands that way"

This has been fixed that way in https://github.com/sosreport/sos/commit/31e0467885ef3986b476ea070941a786b426f298#diff-2de6e8c3c90c8908b74f5ba71a45dd9cf0066f1e60193da0f3a1923c32b6ef37 . That change is present in sos-4.0 and I can confirm so:

# rpm -qf /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
sos-4.0-11.el8.noarch
# cat /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
..
    def collect_iptable(self, tablename):
        """ Collecting iptables rules for a table loads either kernel module
        of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
        If neither module is present, the rules must be empty."""

        modname = "iptable_" + tablename
        cmd = "iptables -t " + tablename + " -nvL"
        self.add_cmd_output(
            cmd,
            pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
..

And I can confirm that behaviour (well, on RHEL8, not CoreOS) that having nf_tables loaded but not ip*tables_* ones, iptables are collected.

Was really sos-4.0-11 used? (grep sos installed-packages)


3) "Replacing calls by one command iptables-save"
I assume it was meant as a workaround or a way of improvement the 2) issue that should be resolved. Also I think the current format is more human-readable (until there is a request for machine processing output, to e.g. load the rules to my system to set them up..?) and splitting the output per tables (nat,filter,..) is for the sake of readiness. Anyway if you feel otherwise, let describe the benefits of the "ip*tables-save" option and we can discussi it either here or rather in upstream (https://github.com/sosreport/sos/issues) - as the change would affect also other distros.


TL;DR: please provide sosreport from point 1), points 2) and 3) should be resolved/irelevant.

Hi Pavel,

(In reply to Pavel Moravec from comment #1)
> we indeed guard execution of some commands if they can load a kernel module
> (or enable a service or similar). Since one key request/feature to sos is
> _not_ altering the system at all.

Yes, makes sense.

> 1) "sos prevents 'nft list ruleset' claiming nf_tables isnt loaded, but
> lsmod shows it" 
> There seems to be a bug in either detection of the module, or sos executed a
> command that loaded the module, or maybe the module was loaded independently
> on sos, during sos execution. Could you please provide the sosreport where
> you noticed this? (case number should sufficient)

Case number is 03000256. The file I inspected is
sosreport-tc06-5vrvn-worker-m42xlarge8-eu-de-02-l9fqx-03000256-2021-09-02-bzqreun.tar.xz,
in case there are multiple.

> 2) "ip*table_* modules are not loaded by "ip[|6]tables -L .." commands on
> RHEL8+, so no reason to guard the commands that way"
> 
> This has been fixed that way in
> https://github.com/sosreport/sos/commit/
> 31e0467885ef3986b476ea070941a786b426f298#diff-
> 2de6e8c3c90c8908b74f5ba71a45dd9cf0066f1e60193da0f3a1923c32b6ef37 . That
> change is present in sos-4.0 and I can confirm so:
> 
> # rpm -qf /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
> sos-4.0-11.el8.noarch
> # cat /usr/lib/python3.6/site-packages/sos/report/plugins/networking.py 
> ..
>     def collect_iptable(self, tablename):
>         """ Collecting iptables rules for a table loads either kernel module
>         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
>         If neither module is present, the rules must be empty."""
> 
>         modname = "iptable_" + tablename
>         cmd = "iptables -t " + tablename + " -nvL"
>         self.add_cmd_output(
>             cmd,
>             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> ..
> 
> And I can confirm that behaviour (well, on RHEL8, not CoreOS) that having
> nf_tables loaded but not ip*tables_* ones, iptables are collected.

How do you prevent the same from happening on RHEL7? There may be nf_tables
loaded but ip*table_* not. Calling iptables then would load the latter.

> Was really sos-4.0-11 used? (grep sos installed-packages)

sos is not listed in 'installed-rpms'. But I found it:

| % grep sos sos_commands/yum/yum_list_installed
| sos.noarch                                    4.0-11.el8                           @RHEL-8.4.0-updates-20210803.2-BaseOS-1

> 3) "Replacing calls by one command iptables-save"
> I assume it was meant as a workaround or a way of improvement the 2) issue
> that should be resolved. Also I think the current format is more
> human-readable (until there is a request for machine processing output, to
> e.g. load the rules to my system to set them up..?) and splitting the output
> per tables (nat,filter,..) is for the sake of readiness. Anyway if you feel
> otherwise, let describe the benefits of the "ip*tables-save" option and we
> can discussi it either here or rather in upstream
> (https://github.com/sosreport/sos/issues) - as the change would affect also
> other distros.

No, if you consider -L output preferrable and are willing to deal with the
implications I won't object. To me it doesn't matter much, but I'm used to both
formats. I pointed at ip*tables-save because it was designed to not cause
module loads (OK, nf_tables breaks that) and therefore might simplify sos.

> TL;DR: please provide sosreport from point 1), points 2) and 3) should be
> resolved/irelevant.

Thanks, Phil

Thanks for prompt feedback. And sorry that I was blind:

>     def collect_iptable(self, tablename):
>         """ Collecting iptables rules for a table loads either kernel module
>         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
>         If neither module is present, the rules must be empty."""
> 
>         modname = "iptable_" + tablename
>         cmd = "iptables -t " + tablename + " -nvL"
>         self.add_cmd_output(
>             cmd,
>             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))

so for table name "filter", we guard the "iptables -t filter -nvL" command by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I overlooked the "modname" here? /o\).

Indeed, we need to distingusih RHEL7 and RHEL8+ - or rather per kernel version(?) that changed this behaviour? Note that sos runs on more distros, so we can't decide based on RHEL version (only).


Do you know a distro-independent test whether  "ip*table_* modules will be loaded by "ip[|6]tables -L .." commands on given system? (I can ask concurrently as well)


> No, if you consider -L output preferrable and are willing to deal with the
> implications I won't object. To me it doesn't matter much, but I'm used to both
> formats. I pointed at ip*tables-save because it was designed to not cause
> module loads (OK, nf_tables breaks that) and therefore might simplify sos.

As the current code will make more complicated due to the "what module might be loaded?" decision, it makes sense to move to the ip*tables-save command - I will discuss this with my colleagues / in sos upstream.

Hi,

(In reply to Pavel Moravec from comment #3)
> Thanks for prompt feedback. And sorry that I was blind:
> 
> >     def collect_iptable(self, tablename):
> >         """ Collecting iptables rules for a table loads either kernel module
> >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> >         If neither module is present, the rules must be empty."""
> > 
> >         modname = "iptable_" + tablename
> >         cmd = "iptables -t " + tablename + " -nvL"
> >         self.add_cmd_output(
> >             cmd,
> >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> 
> so for table name "filter", we guard the "iptables -t filter -nvL" command
> by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> overlooked the "modname" here? /o\).

If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"] defaults
to "any", so either modname or 'nf_tables' should suffice. But the log
indicates that's not the case. Maybe I missed a spot where required["kmods"] is
set to "all".

> Indeed, we need to distingusih RHEL7 and RHEL8+ - or rather per kernel
> version(?) that changed this behaviour? Note that sos runs on more distros,
> so we can't decide based on RHEL version (only).
> 
> 
> Do you know a distro-independent test whether  "ip*table_* modules will be
> loaded by "ip[|6]tables -L .." commands on given system? (I can ask
> concurrently as well)

It actually depends on iptables-variant being used, not kernel version. So I
suggest testing 'iptables -V | grep -q nf_tables'. Older legacy iptables does
not contain 'legacy' in version output, so detecting iptables-nft is backwards
compatible. If that test passes, you need to check if nf_tables.ko is loaded.
If it fails, you can call iptables-save straightaway. Or keep the old report
format and continue to check for iptable_<name>.ko before calling 'iptables
-vnL -t <name>'. 

As a side-note: In Fedora, users may switch between legacy and nft iptables
using alternatives. Checking Fedora release or kernel version is obviously not
an option.

Another alternative could be to call 'iptables-legacy' and 'iptables-nft'
explicitly, with appropriate module checks in beforehand. In theory, a system
could have legacy ruleset in parallel with nft one (although that's discouraged
and almost always leads to disaster). But you'll have to prepare for
environments where only a plain 'iptables' command is available, and there
you'll have to check version output because it may be an old system with legacy
iptables only or RHEL8 on which we ship iptables-nft as iptables and don't
install the explicit name (my approach for RHEL8 is a mess in hindsight).

> > No, if you consider -L output preferrable and are willing to deal with the
> > implications I won't object. To me it doesn't matter much, but I'm used to both
> > formats. I pointed at ip*tables-save because it was designed to not cause
> > module loads (OK, nf_tables breaks that) and therefore might simplify sos.
> 
> As the current code will make more complicated due to the "what module might
> be loaded?" decision, it makes sense to move to the ip*tables-save command -
> I will discuss this with my colleagues / in sos upstream.

OK.

Cheers, Phil

(In reply to Phil Sutter from comment #4)
> Hi,
> 
> (In reply to Pavel Moravec from comment #3)
> > Thanks for prompt feedback. And sorry that I was blind:
> > 
> > >     def collect_iptable(self, tablename):
> > >         """ Collecting iptables rules for a table loads either kernel module
> > >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> > >         If neither module is present, the rules must be empty."""
> > > 
> > >         modname = "iptable_" + tablename
> > >         cmd = "iptables -t " + tablename + " -nvL"
> > >         self.add_cmd_output(
> > >             cmd,
> > >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> > 
> > so for table name "filter", we guard the "iptables -t filter -nvL" command
> > by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> > overlooked the "modname" here? /o\).
> 
> If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"]
> defaults
> to "any", so either modname or 'nf_tables' should suffice. But the log
> indicates that's not the case. Maybe I missed a spot where required["kmods"]
> is
> set to "all".
> 

This seems to be the real culprit behind this bugzilla. The kmods is really "any", which means:

1) the kmods is set right - if either kmod is loaded, iptables will use *that* one (until one does something dirty).
2) as nf_tables was loaded (at some time) but not detected as loaded by sos (at the "right" time), sos skipped collection of various cmd outputs. Only due to this mismatch. WHy this happened?

At:
2021-09-02 15:47:59,614 INFO: [archive:TarFileArchive] initialised empty FileCacheArchive at '/host/var/tmp/sos.mga3nalv/sosreport-HOSTNAME-03000256-2021-09-02-bzqreun'

the sosreport was kicked off in the container (does the issue happen in some container only? it has the same modules like the guest, right?). Very short time later, it executes "lsmod":
https://github.com/sosreport/sos/blob/7d025933c2cdc7bf07a7b6b9c0d8b2637ade372b/sos/policies/distros/__init__.py#L139

to get the list of loaded kernel modules to compare in SoSPreset.

At:
2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat.

it is the 1st time when sos _reported_ the kmod isnt loaded.

At:
2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'

the plugin was already loaded.


So either there is some tricky bug in sos code, or the kernel module was loaded between 15:47:59 and 15:48:32 ? dmesg does not mention anything..

(In reply to Pavel Moravec from comment #5)
> (In reply to Phil Sutter from comment #4)
> > Hi,
> > 
> > (In reply to Pavel Moravec from comment #3)
> > > Thanks for prompt feedback. And sorry that I was blind:
> > > 
> > > >     def collect_iptable(self, tablename):
> > > >         """ Collecting iptables rules for a table loads either kernel module
> > > >         of the table name (for kernel <= 3), or nf_tables (for kernel >= 4).
> > > >         If neither module is present, the rules must be empty."""
> > > > 
> > > >         modname = "iptable_" + tablename
> > > >         cmd = "iptables -t " + tablename + " -nvL"
> > > >         self.add_cmd_output(
> > > >             cmd,
> > > >             pred=SoSPredicate(self, kmods=[modname, 'nf_tables']))
> > > 
> > > so for table name "filter", we guard the "iptables -t filter -nvL" command
> > > by presence/loaded of plugins modname="iptable_filter" and nf_tables (why I
> > > overlooked the "modname" here? /o\).
> > 
> > If that "and" is actually an "OR", yes: SoSPredicate.required["kmods"]
> > defaults
> > to "any", so either modname or 'nf_tables' should suffice. But the log
> > indicates that's not the case. Maybe I missed a spot where required["kmods"]
> > is
> > set to "all".
> > 
> 
> This seems to be the real culprit behind this bugzilla. The kmods is really
> "any", which means:
> 
> 1) the kmods is set right - if either kmod is loaded, iptables will use
> *that* one (until one does something dirty).
> 2) as nf_tables was loaded (at some time) but not detected as loaded by sos
> (at the "right" time), sos skipped collection of various cmd outputs. Only
> due to this mismatch. WHy this happened?
> 
> At:
> 2021-09-02 15:47:59,614 INFO: [archive:TarFileArchive] initialised empty
> FileCacheArchive at
> '/host/var/tmp/sos.mga3nalv/sosreport-HOSTNAME-03000256-2021-09-02-bzqreun'
> 
> the sosreport was kicked off in the container (does the issue happen in some
> container only? it has the same modules like the guest, right?). Very short
> time later, it executes "lsmod":
> https://github.com/sosreport/sos/blob/
> 7d025933c2cdc7bf07a7b6b9c0d8b2637ade372b/sos/policies/distros/__init__.
> py#L139
> 
> to get the list of loaded kernel modules to compare in SoSPreset.
> 
> At:
> 2021-09-02 15:48:03,937 WARNING: [plugin:networking] skipped command
> 'ip6tables -t nat -nvL': required kmods missing: nf_tables, ip6table_nat.
> 
> it is the 1st time when sos _reported_ the kmod isnt loaded.

And then there is:

| 2021-09-02 15:48:03,980 INFO: [plugin:networking] added cmd output 'ip netns exec c4e2b27a-a821-41ca-bf4a-fec375e935a1 iptables-save'

If nf_tables.ko wasn't loaded before, that should have caused it. It is not the
first per-netns iptables-save call, but the first one which returns data
instead of error due to invalid netns name.

But you can't close the case yet, the output from that command shows there are
rules present. So nf_tables.ko was very likely loaded already by the time sos
was called.

> At:
> 2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'
> 
> the plugin was already loaded.
> 
> 
> So either there is some tricky bug in sos code, or the kernel module was
> loaded between 15:47:59 and 15:48:32 ? dmesg does not mention anything..

Loading nf_tables.ko is silent in dmesg, so nothing to gain there. :(

It was / (still is) tricky:

All other predicates were evaluated by the sos execution properly. "just" nf_tables was mis-detected. This supports the theory the kernel module was loaded even after starting sosreport.

And indeed, commands like

2021-09-02 15:48:53,945 INFO: [plugin:networking] collecting output of 'ip netns exec 1924b01d-71e2-4e34-8174-f1ce3e7efa4b iptables-save'

do load the kmod:

# ip netns add test
# lsmod | grep nf
nfnetlink              16384  0
binfmt_misc            20480  1
# ip netns exec test iptables-save
# lsmod | grep nf
nf_tables             147456  0
nfnetlink              16384  1 nf_tables
binfmt_misc            20480  1
#


BUT.. not yet done! Since why the kernel module was already loaded at:

2021-09-02 15:48:32,745 INFO: [plugin:kernel] collecting output of 'lsmod'

when sos_commands/kernel/lsmod DOES contain :

nf_tables             167936  15986 nft_compat,nft_counter,nft_chain_nat

? Some _previous_ command had to load it.


$ grep "collecting output of" sos_logs/sos.log | cut -d\' -f2 | grep -n lsmod
554:lsmod
$

So the first 553 commands have to do so (until some other routine did so, see below).

$ grep -m553 "collecting output of" sos_logs/sos.log | cut -d\' -f2
atomic host status
..
rpm -V audit
$

(none of the command loads the kmod on my system)

If I rule out commands like "uname" or "rpm", here are possible "sinners":

- iscsiadm commands? (I doubt so)
- crictl [inspect|inspecti|inspectp|logs -t] ? could inspecting a kubernetes container load a kernel module? (I dont have simple means to test it)
- "ip netns exec 1924b01d-71e2-4e34-8174-f1ce3e7efa4b bpftool net list" - probably not (cant repro myself)
- some bpftool commands? can't reproduce
- some gluster command? probably not, I dont have means to verify it
- "atomic host status" - probably not


Apart of that, sos report invokes few other commands not directly logged - neither sound a probable one, though:
lsmod
systemctl list-unit-files --type=service
podman ps
podman images --format {{lower .Repository}}:{{lower .Tag}} {{lower .ID}}
podman volume ls
rpm -qa --queryformat '%{NAME}|%{VERSION}|%{RELEASE}\\n'
scl -l
scl -l
scl -l
abrt-cli list
alternatives --list
lsblk -f -a -l
dnf --assumeno module list --installed
bpftool -j prog list
bpftool -j map list
ip netns
gluster volume info
grub2-mkconfig --help
kpatch list
kubectl get namespaces
kubectl get nodes
ldconfig -p -N -X
vgdisplay -h
ip netns
nmcli --terse --fields RUNNING general status
nmcli --terse --fields NAME con
nmcli --terse --fields DEVICE dev
podman network ls
podman ps -a
runc list -q
/usr/sap/hostctrl/exe/saphostctrl -function ListInstances
/usr/sap/hostctrl/exe/saphostctrl -function ListDatabases
getenforce
systemctl status sssd
sssctl domain-list
systemctl status systemd-resolved
ip -o link
rpm -qal
vdo list --all
virt-what

(some few other cmds might be executed, depending on the given system - this is almost impossible to enumerate precisely, though).


TL;DR:
- "ip netns exec <namespace> iptables-save" commands must be guarded by nf_tables kmod (always? since some ip/nftables/kernel version?)
- there is most probably something else among sos commands being collected, that loads that kernel module, but I am unable to find it; please provide some reproducer or more hints to troubleshoot here

> - "ip netns exec <namespace> iptables-save" commands must be guarded by nf_tables kmod (always? since some ip/nftables/kernel version?)

RHEL7 does not load a kernel module when executing that command, so we must come up with condition when the cmd can load the kmod / when to guard the cmd by that kmod predicate. Any idea?\


(btw for a reproducer on RHEL8, I had to stop firewalld service, remove the package and reboot; even then I got nf_tables kmod removed - worth to note for a reproducer)

(on a similar topic, one related PR raised: https://github.com/sosreport/sos/pull/2679 )

Hi,

There is not only nf_tables loaded but also nft_compat, nft_counter and
nft_chain_nat which depend on it. I find it very unlikely that one of the sos
commands not only causes nf_tables to be loaded but also adds a rule to a chain
in nat table.

The problem reported around the discussed sosreport is that OCP calls 'iptables
-C' too often in a specific situation. Another reason why nf_tables very likely
was loaded already by the time sos got called.

I can't put my finger onto it, but am quite sure this is a bug within sos not
properly detecting nf_tables as loaded. I had thought about the underscore
being removed or something, but the code in sos/utilities.py and
sos/policies/distros/__init__.py looks OK to me.

'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
'iptables -V' output contains 'nf_tables'. This is the best way to find out
what variant is used.

Looking at 'iptables -V' output is my recommendation as elaborated on in
comment 4.

Cheers, Phil

(In reply to Phil Sutter from comment #10)
> 'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
> 'iptables -V' output contains 'nf_tables'. This is the best way to find out
> what variant is used.

On a similar note: shouldn't we apply the same to ip6tables-save as well? I.e. "if 'ip6tables -V' contains nf_tables, then guard the 'ip netns exec <foo> ip6tables save' command by nf_tables kmod"?

(In reply to Pavel Moravec from comment #11)
> (In reply to Phil Sutter from comment #10)
> > 'ip netns exec <foo> iptables-save' must be guarded by nf_tables kmod if
> > 'iptables -V' output contains 'nf_tables'. This is the best way to find out
> > what variant is used.
> 
> On a similar note: shouldn't we apply the same to ip6tables-save as well?
> I.e. "if 'ip6tables -V' contains nf_tables, then guard the 'ip netns exec
> <foo> ip6tables save' command by nf_tables kmod"?

Yes, please, sorry for being a bit sloppy. The same holds for arptables and
ebtables, BTW. I don't recall if sos collects those, too.

[root@pmoravec-rhel8 ~]# ip netns ls
test
[root@pmoravec-rhel8 ~]# lsmod | grep nf
[root@pmoravec-rhel8 ~]# ip netns exec test iptables-save
[root@pmoravec-rhel8 ~]# lsmod | grep nf
nf_tables             172032  0
nfnetlink              16384  1 nf_tables
libcrc32c              16384  2 nf_tables,xfs
[root@pmoravec-rhel8 ~]# rmmod nf_tables; lsmod | grep nf
nfnetlink              16384  0
[root@pmoravec-rhel8 ~]# ip netns exec test iptables-save
[root@pmoravec-rhel8 ~]# lsmod | grep nf
nf_tables             172032  0
nfnetlink              16384  1 nf_tables
libcrc32c              16384  2 nf_tables,xfs
[root@pmoravec-rhel8 ~]#

So the "ip netns exec .." command loads nf_tables which loads nfnetlink. So SoSPreset kmods=['nf_tables'] should be sufficient. That applies to both iptables-save and ip6tables-save.

https://github.com/sosreport/sos/pull/2703 PR proposed to upstream.

Reproducer steps for QE:

1) Have a RHEL8 with firewalld package removed (sometimes a reboot afterwards is required as well), which should allow you to have nf_tables kernel module unloaded:

dnf remove firewalld -y
lsmod | grep nf_tables  # shows nothing, optionally rmmod nf_tables if no reboot done

(also ensure iptables and ip6tables have nf_tables built in - this distinguishes behaviour between RHEL7 and RHEL8+ :)
iptables -V | grep nf_tables  # shows a match
ip6tables -V | grep nf_tables  # shows a match


2) Add a network namespace:

ip netns add test
ip netns ls


3) Collect networking plugin:

sos report -o networking --batch --build


4) Check that neither "ip netns test iptables-save" or "ip netns test ip6tables-save" is collected (when iptables have nf_tables built in)


5) Ensure nf_tables kmod is not loaded:

lsmod | grep nf_tables  # shows nothing


6) Manually enforce loading the kmod:

ip netns test iptables-save
lsmod | grep nf_tables  # shows the kmod loaded


7) Re-run the sos report and ensure the "ip netns test iptables-save" and "ip netns test ip6tables-save" is properly collected.

Hello,
would you be able to verify a candidate build, once that is available?

(and ideally also against 9.0 GA clone bz2011538 ?)

Hi Pavel,

(In reply to Pavel Moravec from comment #16)
> Hello,
> would you be able to verify a candidate build, once that is available?
> 
> (and ideally also against 9.0 GA clone bz2011538 ?)

Sure! Please just ping me once the build is ready for testing.

Hi!

I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two things:

* 'nft list ruleset' is not called for detected network namespaces. This is not
  a new problem and rather an RFE but worth mentioning IMO. Feel free to clone
  a new BZ for this.

* With no firewalling active (and no related modules loaded), if I manually
  load iptable_filter.ko (this provides filter table for legacy iptables), sos
  calls 'iptables -vnL' which causes loading of nf_tables.ko. This is a
  somewhat constructed case, iptable_filter.ko shouldn't be used on RHEL8.
  Though AFAIK, we allow RHEL7 containers on RHEL8 so it might indeed happen in
  practice.

Cheers, Phil

(In reply to Phil Sutter from comment #20)
> Hi!
> 
> I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two
> things:

Hello, first, thanks for the testing. As neither below thing is directly related to the original BZ, does it mean the bugfix is correct? (i.e. can be the BZ put to verified?)?
> 
> * 'nft list ruleset' is not called for detected network namespaces. This is
> not
>   a new problem and rather an RFE but worth mentioning IMO. Feel free to
> clone
>   a new BZ for this.
> 


You mean https://github.com/sosreport/sos/blob/7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/__init__.py#L417-L431 where we we generate the underlying list of amespaces queried e.g. at https://github.com/sosreport/sos/blob/7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/plugins/networking.py#L186 ?

When the __init__.py#L417-L431 should rely on "ip" command and when on "nft"? On what architectures and/or distros or their versions? (or is "ip" command being deprecated (and everytime in favour of "nft") on any arch/distro since some release?)?

Thanks in advance for the answers.

Hi Pavel,

(In reply to Pavel Moravec from comment #21)
> (In reply to Phil Sutter from comment #20)
> > Hi!
> > 
> > I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two
> > things:
> 
> Hello, first, thanks for the testing. As neither below thing is directly
> related to the original BZ, does it mean the bugfix is correct? (i.e. can be
> the BZ put to verified?)?

I am not entirely happy about my second item above, it is quite related to this
BZ but the current solution is probably good enough for RHEL.

> > * 'nft list ruleset' is not called for detected network namespaces. This is
> > not
> >   a new problem and rather an RFE but worth mentioning IMO. Feel free to
> > clone
> >   a new BZ for this.
> > 
> 
> 
> You mean
> https://github.com/sosreport/sos/blob/
> 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/__init__.py#L417-L431
> where we we generate the underlying list of amespaces queried e.g. at
> https://github.com/sosreport/sos/blob/
> 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/plugins/networking.
> py#L186 ?
> 
> When the __init__.py#L417-L431 should rely on "ip" command and when on
> "nft"? On what architectures and/or distros or their versions? (or is "ip"
> command being deprecated (and everytime in favour of "nft") on any
> arch/distro since some release?)?
> 
> Thanks in advance for the answers.

This does not compile, sorry: 'nft' can't be used to detect network namespaces.
So using 'ip netns list' is perfectly fine. But sos does not call
'ip netns exec <NS> nft list ruleset'. So we don't record the nftables ruleset
which may be present in a network namespace.

Cheers, Phil

(In reply to Phil Sutter from comment #22)
> Hi Pavel,
> 
> (In reply to Pavel Moravec from comment #21)
> > (In reply to Phil Sutter from comment #20)
> > > Hi!
> > > 
> > > I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two
> > > things:
> > 
> > Hello, first, thanks for the testing. As neither below thing is directly
> > related to the original BZ, does it mean the bugfix is correct? (i.e. can be
> > the BZ put to verified?)?
> 
> I am not entirely happy about my second item above, it is quite related to
> this
> BZ but the current solution is probably good enough for RHEL.

If you can re-formulate the issue to "sos terminology", I am happy to improve the code.

"sos terminology" - to describe when and what to collect, like "collect iptables -nvL only if (some old kernel OR (nf_tables loaded AND iptable_filter loaded)" - I am bit lost now when exactly we should (not) collect the command.

> 
> > > * 'nft list ruleset' is not called for detected network namespaces. This is
> > > not
> > >   a new problem and rather an RFE but worth mentioning IMO. Feel free to
> > > clone
> > >   a new BZ for this.
> > > 
> > 
> > 
> > You mean
> > https://github.com/sosreport/sos/blob/
> > 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/__init__.py#L417-L431
> > where we we generate the underlying list of amespaces queried e.g. at
> > https://github.com/sosreport/sos/blob/
> > 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/plugins/networking.
> > py#L186 ?
> > 
> > When the __init__.py#L417-L431 should rely on "ip" command and when on
> > "nft"? On what architectures and/or distros or their versions? (or is "ip"
> > command being deprecated (and everytime in favour of "nft") on any
> > arch/distro since some release?)?
> > 
> > Thanks in advance for the answers.
> 
> This does not compile, sorry: 'nft' can't be used to detect network
> namespaces.
> So using 'ip netns list' is perfectly fine. But sos does not call
> 'ip netns exec <NS> nft list ruleset'. So we don't record the nftables
> ruleset
> which may be present in a network namespace.

Ah I see. So you mean add to the list https://github.com/sosreport/sos/blob/58902e8f08d3c432541f6fc037999daafb9d5b4f/sos/report/plugins/networking.py#L205-L214  (where ns_cmd_prefix = "ip netns exec NAMESPACE ") also subcommand "nft list rulest" ?

(In reply to Pavel Moravec from comment #23)
> (In reply to Phil Sutter from comment #22)
> > (In reply to Pavel Moravec from comment #21)
> > > (In reply to Phil Sutter from comment #20)
> > > > Hi!
> > > > 
> > > > I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two
> > > > things:
> > > 
> > > Hello, first, thanks for the testing. As neither below thing is directly
> > > related to the original BZ, does it mean the bugfix is correct? (i.e. can be
> > > the BZ put to verified?)?
> > 
> > I am not entirely happy about my second item above, it is quite related to
> > this
> > BZ but the current solution is probably good enough for RHEL.
> 
> If you can re-formulate the issue to "sos terminology", I am happy to
> improve the code.

Sorry that this stuff is so complicated. Avoiding iptables to load a module is
apparently harder than anticipated.

> "sos terminology" - to describe when and what to collect, like "collect
> iptables -nvL only if (some old kernel OR (nf_tables loaded AND
> iptable_filter loaded)" - I am bit lost now when exactly we should (not)
> collect the command.

collect iptables -vnL ONLY IF
	('iptables --version' CONTAINS 'nf_tables' AND nf_tables.ko is loaded) OR
	('iptables --version' NOT CONTAINS 'nf_tables' AND iptable_filter.ko is loaded)

Since the above gets duplicated for every table and ip6tables, the following is simpler:

IF 'iptables --version' CONTAINS 'nf_tables':
	IF nf_tables.ko is loaded:
		collect iptables -vnL
		collect iptables -vnL -t nat
		...
		collect ip6tables -vnL
		collect ip6tables -vnL -t nat
		...
ELSE:
	collect iptables -vnL ONLY IF iptable_filter.ko is loaded
	collect iptables -vnL -t nat ONLY IF iptable_nat.ko is loaded
	...
	collect ip6tables -vnL ONLY IF ip6table_filter.ko is loaded
	collect ip6tables -vnL -t nat ONLY IF ip6table_nat.ko is loaded
	...
ENDIF

> > > > * 'nft list ruleset' is not called for detected network namespaces. This is
> > > > not
> > > >   a new problem and rather an RFE but worth mentioning IMO. Feel free to
> > > > clone
> > > >   a new BZ for this.
> > > > 
> > > 
> > > 
> > > You mean
> > > https://github.com/sosreport/sos/blob/
> > > 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/__init__.py#L417-L431
> > > where we we generate the underlying list of amespaces queried e.g. at
> > > https://github.com/sosreport/sos/blob/
> > > 7ae47e6c0717c0b56c3368008dd99a87f7f436d5/sos/report/plugins/networking.
> > > py#L186 ?
> > > 
> > > When the __init__.py#L417-L431 should rely on "ip" command and when on
> > > "nft"? On what architectures and/or distros or their versions? (or is "ip"
> > > command being deprecated (and everytime in favour of "nft") on any
> > > arch/distro since some release?)?
> > > 
> > > Thanks in advance for the answers.
> > 
> > This does not compile, sorry: 'nft' can't be used to detect network
> > namespaces.
> > So using 'ip netns list' is perfectly fine. But sos does not call
> > 'ip netns exec <NS> nft list ruleset'. So we don't record the nftables
> > ruleset
> > which may be present in a network namespace.
> 
> Ah I see. So you mean add to the list
> https://github.com/sosreport/sos/blob/
> 58902e8f08d3c432541f6fc037999daafb9d5b4f/sos/report/plugins/networking.
> py#L205-L214  (where ns_cmd_prefix = "ip netns exec NAMESPACE ") also
> subcommand "nft list rulest" ?

Yes. :)
But please keep in mind to predicate it with a check for loaded nf_tables.ko.

Thanks, Phil

(In reply to Phil Sutter from comment #24)
> (In reply to Pavel Moravec from comment #23)
> > (In reply to Phil Sutter from comment #22)
> > > (In reply to Pavel Moravec from comment #21)
> > > > (In reply to Phil Sutter from comment #20)
> > > > > Hi!
> > > > > 
> > > > > I just tested sos-4.2-1.el8.noarch on a 1minutetip machine and noticed two
> > > > > things:
> > > > 
> > > > Hello, first, thanks for the testing. As neither below thing is directly
> > > > related to the original BZ, does it mean the bugfix is correct? (i.e. can be
> > > > the BZ put to verified?)?
> > > 
> > > I am not entirely happy about my second item above, it is quite related to
> > > this
> > > BZ but the current solution is probably good enough for RHEL.
> > 
> > If you can re-formulate the issue to "sos terminology", I am happy to
> > improve the code.
> 
> Sorry that this stuff is so complicated. Avoiding iptables to load a module
> is
> apparently harder than anticipated.

Yes. Updating networking plugin is like minefield walk-through :).

 
> collect iptables -vnL ONLY IF
> 	('iptables --version' CONTAINS 'nf_tables' AND nf_tables.ko is loaded) OR
> 	('iptables --version' NOT CONTAINS 'nf_tables' AND iptable_filter.ko is
> loaded)
> 
> Since the above gets duplicated for every table and ip6tables, the following
> is simpler:
> 
> IF 'iptables --version' CONTAINS 'nf_tables':
> 	IF nf_tables.ko is loaded:
> 		collect iptables -vnL
> 		collect iptables -vnL -t nat
> 		...
> 		collect ip6tables -vnL
> 		collect ip6tables -vnL -t nat
> 		...
> ELSE:
> 	collect iptables -vnL ONLY IF iptable_filter.ko is loaded
> 	collect iptables -vnL -t nat ONLY IF iptable_nat.ko is loaded
> 	...
> 	collect ip6tables -vnL ONLY IF ip6table_filter.ko is loaded
> 	collect ip6tables -vnL -t nat ONLY IF ip6table_nat.ko is loaded
> 	...
> ENDIF

So *similarly* like we do for per-namespace ip*tables-save : https://github.com/sosreport/sos/blob/58902e8f08d3c432541f6fc037999daafb9d5b4f/sos/report/plugins/networking.py#L190-L202 (and #L215-L220)? The "else None" should be extended to "else ip*table_filter kmod loaded", I guess.

See the *second* commit in https://github.com/sosreport/sos/pull/2760, if it applies the rules properly (and also in the networking plugin).


> > > This does not compile, sorry: 'nft' can't be used to detect network
> > > namespaces.
> > > So using 'ip netns list' is perfectly fine. But sos does not call
> > > 'ip netns exec <NS> nft list ruleset'. So we don't record the nftables
> > > ruleset
> > > which may be present in a network namespace.
> > 
> > Ah I see. So you mean add to the list
> > https://github.com/sosreport/sos/blob/
> > 58902e8f08d3c432541f6fc037999daafb9d5b4f/sos/report/plugins/networking.
> > py#L205-L214  (where ns_cmd_prefix = "ip netns exec NAMESPACE ") also
> > subcommand "nft list rulest" ?
> 
> Yes. :)
> But please keep in mind to predicate it with a check for loaded nf_tables.ko.

Please review if the *first* commit in https://github.com/sosreport/sos/pull/2760 matches your expectations. 


Both PRs will automatically land in 8.7/9.1, no plans to backport sooner.

Pavel,

I'm sorry but I can't review your code-changes. I'm not familiar with the code
in general nor the predicate paradigm in specific so it's mere guesswork
whether the code is correct or not. From my PoV it was quicker to run sos on a
system and see if it does the right thing.

Was the IF/ELSE notation you requested unclear? If so,  please let me know and
I'll try to redefine it.

A note about namespaces: There is no need to treat them like a whole separate
thing. From the point of whether to call iptables or not and what modules to
check in beforehand, there's no difference between inside namespace or not.

Cheers, Phil

(In reply to Phil Sutter from comment #26)
> Pavel,
> 
> I'm sorry but I can't review your code-changes. I'm not familiar with the
> code
> in general nor the predicate paradigm in specific so it's mere guesswork
> whether the code is correct or not. From my PoV it was quicker to run sos on
> a
> system and see if it does the right thing.
> 
> Was the IF/ELSE notation you requested unclear? If so,  please let me know
> and
> I'll try to redefine it.

Yes, it was clear, thanks for that.

> 
> A note about namespaces: There is no need to treat them like a whole separate
> thing. From the point of whether to call iptables or not and what modules to
> check in beforehand, there's no difference between inside namespace or not.
> 

This was my point - the "collect iptables -vnL ONLY IF .." rules does apply to "plain" iptables command as well as to "ip netns .. iptables" commands, right?

Any way, there is a gotcha in evaluating the rules: we can't use "iptables -V" since that already requires a kernel module:

# lsmod | grep -e nfnetlink -e nf_tables
# iptables -V
iptables v1.8.4 (nf_tables)
# lsmod | grep -e nfnetlink -e nf_tables
nfnetlink              16384  0
# 

Sigh.. Any other idea?

Hi Pavel,

(In reply to Pavel Moravec from comment #27)
> (In reply to Phil Sutter from comment #26)
[...]
> > A note about namespaces: There is no need to treat them like a whole separate
> > thing. From the point of whether to call iptables or not and what modules to
> > check in beforehand, there's no difference between inside namespace or not.
> > 
> 
> This was my point - the "collect iptables -vnL ONLY IF .." rules does apply
> to "plain" iptables command as well as to "ip netns .. iptables" commands,
> right?

Yes, exactly. While the output differs depending on netns (because netfilter
rulesets are per-netns), module loading is global so the initial check results
apply to both calls.

> Any way, there is a gotcha in evaluating the rules: we can't use "iptables
> -V" since that already requires a kernel module:
> 
> # lsmod | grep -e nfnetlink -e nf_tables
> # iptables -V
> iptables v1.8.4 (nf_tables)
> # lsmod | grep -e nfnetlink -e nf_tables
> nfnetlink              16384  0
> # 
> 
> Sigh.. Any other idea?

Argh, I missed the fact that iptables-nft does initialize nfnetlink socket
prior to parsing arguments. The next best option is to check the symlink's
target: The actual binary name is either xtables-nft-multi or not (depending on
distribution the alternative is either xtables-legacy-multi or xtables-multi).
/usr/sbin/iptables is a symlink to it, sometimes with alternatives as
indirection. Long story short:

| # readlink -f /usr/sbin/iptables
| /usr/sbin/xtables-nft-multi

(Option '-f' follows all intermediate symlinks.) If the result is
'xtables-nft-multi', you need to check for nf_tables.ko. If not, check
iptable_filter.ko (etc).

Does that work for you?

Cheers, Phil

TODO on myself: modify https://github.com/sosreport/sos/pull/2760 per above comments.

Removing this bug from 8.6 scope.

Already backported changes will remain in the 8.6 candidate build, so most of the required improvement is achieved. But the

(In reply to Pavel Moravec from comment #29)
> TODO on myself: modify https://github.com/sosreport/sos/pull/2760 per above
> comments.

will not be resolved in time for 8.6. Thus we agreed to complete the BZ once the #2760 is fixed / in 8.7.

Removing from 8.7 scope as the upstream issue hasnt been fixed (and we should revisit the networking plugin first, imho).

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.