Bug 1668007

Hi Michele,

(In reply to Michele Baldessari from comment #0)
> Description of problem:
> We're trying to port openstack to rhel8 and I noticed the following output:
> iptables -nvL
> ...<snip a bunch of rules>...
> # Warning: iptables-legacy tables present, use iptables-legacy to see them

Hmm. Looks like legacy iptables kernel modules were loaded. This message is
printed if /proc/net/ip_tables_names exists and contains at least one table
name. I know this check is not quite sufficient, but assuming that relevant
kernel modules are auto-loaded it should be relatively safe to assume they are
not loaded for no reason. Does OpenStack explicitly load netfilter kernel
modules?

> The above warning about iptables-legacy is quite confusing because we seem
> to have decided to not ship those legacy binaries. So for a user the
> question becomes how do I see those rules that are present in legacy tables?

That's a valid question, but without iptables-legacy there shouldn't be any
rules, either.

Cheers, Phil

Hi Phil,

thanks a bunch for your help here, much appreciated

(In reply to Phil Sutter from comment #1)
> (In reply to Michele Baldessari from comment #0)
> > Description of problem:
> > We're trying to port openstack to rhel8 and I noticed the following output:
> > iptables -nvL
> > ...<snip a bunch of rules>...
> > # Warning: iptables-legacy tables present, use iptables-legacy to see them
> 
> Hmm. Looks like legacy iptables kernel modules were loaded. This message is
> printed if /proc/net/ip_tables_names exists and contains at least one table
> name. I know this check is not quite sufficient, but assuming that relevant
> kernel modules are auto-loaded it should be relatively safe to assume they
> are
> not loaded for no reason. Does OpenStack explicitly load netfilter kernel
> modules?

It could be, I'll need to dig around a bit more (it might be some underlying puppet
module that is doing that under the hood). Here is some relevant output:
[root@win1 ~]# cat /proc/net/ip_tables_names 
filter
[root@win1 ~]# lsmod 
Module                  Size  Used by
ebtable_filter         16384  0
ebtables               36864  1 ebtable_filter
ip6table_filter        16384  0
ip6_tables             32768  1 ip6table_filter
iptable_filter         16384  0
ip_tables              28672  1 iptable_filter
devlink                81920  0
scsi_transport_iscsi   106496  1
overlay               126976  9
nf_log_ipv6            16384  1
nf_log_ipv4            16384  1
nf_log_common          16384  2 nf_log_ipv4,nf_log_ipv6
nft_limit              16384  2
xt_LOG                 16384  2
xt_limit               16384  0
xt_multiport           16384  2
xt_comment             16384  1
binfmt_misc            20480  1
ip6t_REJECT            16384  1
nf_reject_ipv6         16384  1 ip6t_REJECT
nft_counter            16384  73
ipt_REJECT             16384  1
nf_reject_ipv4         16384  1 ipt_REJECT
xt_state               16384  0
xt_conntrack           16384  1
nft_compat             20480  178
nf_tables             143360  207 nft_compat,nft_counter,nft_limit
nfnetlink              16384  2 nft_compat,nf_tables
loop                   32768  1
openvswitch           155648  3
nf_conntrack_ipv6      20480  33
nf_nat_ipv6            16384  1 openvswitch
nf_conntrack_ipv4      16384  34
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 openvswitch
nf_conncount           16384  1 openvswitch
nf_defrag_ipv6         20480  2 nf_conntrack_ipv6,openvswitch
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_ipv4,openvswitch
nf_conntrack          155648  9 xt_conntrack,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,xt_state,nf_nat_ipv6,nf_nat_ipv4,openvswitch,nf_conncount
cirrus                 28672  1
sb_edac                24576  0
ttm                   131072  1 cirrus
kvm_intel             241664  0
drm_kms_helper        200704  1 cirrus
kvm                   745472  1 kvm_intel
syscopyarea            16384  1 drm_kms_helper
irqbypass              16384  1 kvm
sysfillrect            16384  1 drm_kms_helper
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
sysimgblt              16384  1 drm_kms_helper
sunrpc                434176  1
ghash_clmulni_intel    16384  0
fb_sys_fops            16384  1 drm_kms_helper
joydev                 24576  0
drm                   520192  4 drm_kms_helper,cirrus,ttm
i2c_piix4              24576  0
pcspkr                 16384  0
virtio_balloon         20480  0
xfs                  1470464  2
libcrc32c              16384  4 nf_conntrack,nf_nat,openvswitch,xfs
ata_generic            16384  0
ata_piix               36864  0
virtio_net             53248  0
libata                274432  2 ata_piix,ata_generic
net_failover           24576  1 virtio_net
virtio_console         32768  1
serio_raw              16384  0
virtio_blk             20480  4
crc32c_intel           24576  1
failover               16384  1 net_failover

What should we investigate here? Will making sure that we avoid loading the legacy iptables modules be enough?
Which ones are the legacy ones? all the xt_* and ipt_* ones?
 
> > The above warning about iptables-legacy is quite confusing because we seem
> > to have decided to not ship those legacy binaries. So for a user the
> > question becomes how do I see those rules that are present in legacy tables?
> 
> That's a valid question, but without iptables-legacy there shouldn't be any
> rules, either.

Thanks,
Michele

Ok so the reason why we had the loaded iptables legacy modules was due to using a libvirt container which was likely a bit old and was still loading the older things. Shouldn't we either not ship these older iptables kernel modules or at the very least blacklist them by default in the distro?

Thanks,
Michele

Hi Michele,

(In reply to Michele Baldessari from comment #3)
> Ok so the reason why we had the loaded iptables legacy modules was due to
> using a libvirt container which was likely a bit old and was still loading
> the older things. Shouldn't we either not ship these older iptables kernel
> modules or at the very least blacklist them by default in the distro?

That's an interesting idea. We can't drop any of the xt_* or ipt_* modules as
they are used by nftables-based iptables, too. But we might remove/blacklist the
modules providing the different tables, like {eb,ip,ip6}table_filter in your
case.

Can you reproduce the situation? I wonder what happens if you blacklist those
modules and start that container. Might happen that it refuses to operate, which
I guess is a no go then. OTOH that legacy tables check is certainly subject to
improve - it should trigger only if there actually are rules present.

Thanks, Phil

Hi Phil,

(In reply to Phil Sutter from comment #4)
> (In reply to Michele Baldessari from comment #3)
> > Ok so the reason why we had the loaded iptables legacy modules was due to
> > using a libvirt container which was likely a bit old and was still loading
> > the older things. Shouldn't we either not ship these older iptables kernel
> > modules or at the very least blacklist them by default in the distro?
> 
> That's an interesting idea. We can't drop any of the xt_* or ipt_* modules as
> they are used by nftables-based iptables, too. But we might remove/blacklist
> the
> modules providing the different tables, like {eb,ip,ip6}table_filter in your
> case.
> 
> Can you reproduce the situation? I wonder what happens if you blacklist those
> modules and start that container. Might happen that it refuses to operate,
> which
> I guess is a no go then. OTOH that legacy tables check is certainly subject
> to
> improve - it should trigger only if there actually are rules present.

thanks for the idea. I will give it a shot (will take a bit). Is the complete list
of legacy kernel modules that might populate the /proc/net/ip[6]_tables_names files
the following:
[root@rhel8 kernel]# find /lib/modules/$(uname -r)/kernel -iname '*_filter*' -type f
/lib/modules/4.18.0-57.el8.x86_64/kernel/net/bridge/netfilter/ebtable_filter.ko.xz
/lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv4/netfilter/arptable_filter.ko.xz
/lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv4/netfilter/iptable_filter.ko.xz
/lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv6/netfilter/ip6table_filter.ko.xz

Or should are there other legacy modules that could/should be blacklisted?

Thanks,
Michele

Hi Michele,


(In reply to Michele Baldessari from comment #5)
[...]
> thanks for the idea. I will give it a shot (will take a bit). Is the
> complete list
> of legacy kernel modules that might populate the
> /proc/net/ip[6]_tables_names files
> the following:
> [root@rhel8 kernel]# find /lib/modules/$(uname -r)/kernel -iname '*_filter*'
> -type f
> /lib/modules/4.18.0-57.el8.x86_64/kernel/net/bridge/netfilter/ebtable_filter.
> ko.xz
> /lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv4/netfilter/arptable_filter.
> ko.xz
> /lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv4/netfilter/iptable_filter.
> ko.xz
> /lib/modules/4.18.0-57.el8.x86_64/kernel/net/ipv6/netfilter/ip6table_filter.
> ko.xz
> 
> Or should are there other legacy modules that could/should be blacklisted?

The full list is as follows:

iptable_nat.ko
iptable_raw.ko
iptable_mangle.ko
iptable_security.ko
iptable_filter.ko
ip6table_nat.ko
ip6table_raw.ko
ip6table_mangle.ko
ip6table_security.ko
ip6table_filter.ko
arptable_filter.ko
ebtable_nat.ko
ebtable_filter.ko
ebtable_broute.ko

Thanks, Phil

Hi Michele,

(In reply to Michele Baldessari from comment #5)
> (In reply to Phil Sutter from comment #4)
> > (In reply to Michele Baldessari from comment #3)
> > > Ok so the reason why we had the loaded iptables legacy modules was due to
> > > using a libvirt container which was likely a bit old and was still loading
> > > the older things. Shouldn't we either not ship these older iptables kernel
> > > modules or at the very least blacklist them by default in the distro?
> > 
> > That's an interesting idea. We can't drop any of the xt_* or ipt_* modules as
> > they are used by nftables-based iptables, too. But we might remove/blacklist
> > the
> > modules providing the different tables, like {eb,ip,ip6}table_filter in your
> > case.
> > 
> > Can you reproduce the situation? I wonder what happens if you blacklist those
> > modules and start that container. Might happen that it refuses to operate,
> > which
> > I guess is a no go then. OTOH that legacy tables check is certainly subject
> > to
> > improve - it should trigger only if there actually are rules present.
> 
> thanks for the idea. I will give it a shot (will take a bit). Is the

Did you find time already to give it a test?

Cheers, Phil

Hi Phil,

sorry this took way longer than I had hoped. So I think we're fundamentally okay here:
1) We did not trigger the issue whenever we used rhel8 containers on rhel8 os and hopefully we will never hit a mixed version situation (we do not yet know how the upgrade story will look like, so there is potential for this to hit us)
2) We have a plan B) which we did not merge yet here https://review.opendev.org/#/c/637171/, in case 1) will still bite us somehow.

I still think we should not even ship these deprecated modules, but I guess that ship has sailed some time ago ;)

I'll close this for the time being. Thanks for your help, much appreciated.

cheers,
Michele

Hi Michele,

(In reply to Michele Baldessari from comment #8)
> sorry this took way longer than I had hoped. So I think we're fundamentally
> okay here:
> 1) We did not trigger the issue whenever we used rhel8 containers on rhel8
> os and hopefully we will never hit a mixed version situation (we do not yet
> know how the upgrade story will look like, so there is potential for this to
> hit us)
> 2) We have a plan B) which we did not merge yet here
> https://review.opendev.org/#/c/637171/, in case 1) will still bite us
> somehow.
> 
> I still think we should not even ship these deprecated modules, but I guess
> that ship has sailed some time ago ;)

AFAICT, RHEL7 containers on RHEL8 is an expected scenario. I'm pretty sure they
shouldn't (be able to) modify host's firewall ruleset, but even listing
iptables rules is enough to trigger the problem.

BTW, I just tested blacklisting on a local VM - it is not effective, iptables
explicitly calls modprobe. With that in mind, I wonder how that module loading
from a container is possible at all: Does /lib/modules stay shared between host
and container?

So removing the problematic modules is the only option, but that means
customers won't have an easy fix if a container refuses to operate because
'iptables -vnL' returned non-zero. Not sure if we can demand tolerance to
missing legacy iptables from containers.

Maybe we could at least move the unneeded modules to kernel-modules-extra, but
I fear everyone will just install that package by default anyway.

Cheers, Phil

Thank you for the analysis — especially about /proc/net/ip_tables_names — and the blacklist. I need it for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963012 to work around bugs in iptables-persistent (it basically forces iptables-legacy); unsure whether that bug also applies to Red Hat, but cross-linking the issues won’t hurt.

Hey Thorsten,

Long time no see! :)

(In reply to Thorsten Glaser from comment #10)
> Thank you for the analysis — especially about /proc/net/ip_tables_names —
> and the blacklist. I need it for
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963012 to work around bugs
> in iptables-persistent (it basically forces iptables-legacy); unsure whether
> that bug also applies to Red Hat, but cross-linking the issues won’t hurt.

Since iptables-persistent does not exist in RHEL, I guess we're fine here. For
iptables-services, I simply hard-coded the list of available table names.
Implementing a real detection is not as easy, nftables doesn't reveal its
capabilities that openly to the user.

Cheers, Phil

Hi Phil,

ah, I was more-than-elbow-deep in a customer project while debugging this and didn’t notice the name.

As for the feedback; I guess we’re all good then. Thanks!