This is apparently a side-effect of backported commit 200bc39965149 ("nft:
cache: Fix iptables-save segfault under stress"). Due to other changes,
upstream doesn't suffer from the problem anymore.

While it wasn't easy to figure what exactly goes wrong, the fix is rather
trivial: If cache is rebuilt, the stored generation ID must be reset to zero as
it is never updated when fetching cache if it is non-zero. The change is a one-liner:

--- a/iptables/nft-cache.c
+++ b/iptables/nft-cache.c
@@ -629,6 +629,7 @@ void nft_rebuild_cache(struct nft_handle *h)
        if (h->cache_level)
                __nft_flush_cache(h);

+       h->nft_genid = 0;
        h->cache_level = NFT_CL_NONE;
        __nft_build_cache(h, level, NULL, NULL, NULL);
 }

(In reply to Phil Sutter from comment #1)
> This is apparently a side-effect of backported commit 200bc39965149 ("nft:
> cache: Fix iptables-save segfault under stress"). Due to other changes,
> upstream doesn't suffer from the problem anymore.
> 
> While it wasn't easy to figure what exactly goes wrong, the fix is rather
> trivial: If cache is rebuilt, the stored generation ID must be reset to zero
> as
> it is never updated when fetching cache if it is non-zero. The change is a
> one-liner:
> 
> --- a/iptables/nft-cache.c
> +++ b/iptables/nft-cache.c
> @@ -629,6 +629,7 @@ void nft_rebuild_cache(struct nft_handle *h)
>         if (h->cache_level)
>                 __nft_flush_cache(h);
> 
> +       h->nft_genid = 0;
>         h->cache_level = NFT_CL_NONE;
>         __nft_build_cache(h, level, NULL, NULL, NULL);
>  }


Thank you for the update.  Will this fix be part of 8.2?  We use "iptables-restore --test" and this is causing considerable headaches.

Hi,

(In reply to Dan from comment #2)
> (In reply to Phil Sutter from comment #1)
> > This is apparently a side-effect of backported commit 200bc39965149 ("nft:
> > cache: Fix iptables-save segfault under stress"). Due to other changes,
> > upstream doesn't suffer from the problem anymore.
> > 
> > While it wasn't easy to figure what exactly goes wrong, the fix is rather
> > trivial: If cache is rebuilt, the stored generation ID must be reset to zero
> > as
> > it is never updated when fetching cache if it is non-zero. The change is a
> > one-liner:
> > 
> > --- a/iptables/nft-cache.c
> > +++ b/iptables/nft-cache.c
> > @@ -629,6 +629,7 @@ void nft_rebuild_cache(struct nft_handle *h)
> >         if (h->cache_level)
> >                 __nft_flush_cache(h);
> > 
> > +       h->nft_genid = 0;
> >         h->cache_level = NFT_CL_NONE;
> >         __nft_build_cache(h, level, NULL, NULL, NULL);
> >  }
> 
> 
> Thank you for the update.  Will this fix be part of 8.2?  We use
> "iptables-restore --test" and this is causing considerable headaches.

Not by default. This ticket will address 8.3, backporting into 8.2 requires an explicit request. I'll discuss this with QA as they'll need to have capacity for it.

Cheers, Phil

(In reply to Phil Sutter from comment #3)
> Hi,
> 
> (In reply to Dan from comment #2)
> > (In reply to Phil Sutter from comment #1)
> > > This is apparently a side-effect of backported commit 200bc39965149 ("nft:
> > > cache: Fix iptables-save segfault under stress"). Due to other changes,
> > > upstream doesn't suffer from the problem anymore.
> > > 
> > > While it wasn't easy to figure what exactly goes wrong, the fix is rather
> > > trivial: If cache is rebuilt, the stored generation ID must be reset to zero
> > > as
> > > it is never updated when fetching cache if it is non-zero. The change is a
> > > one-liner:
> > > 
> > > --- a/iptables/nft-cache.c
> > > +++ b/iptables/nft-cache.c
> > > @@ -629,6 +629,7 @@ void nft_rebuild_cache(struct nft_handle *h)
> > >         if (h->cache_level)
> > >                 __nft_flush_cache(h);
> > > 
> > > +       h->nft_genid = 0;
> > >         h->cache_level = NFT_CL_NONE;
> > >         __nft_build_cache(h, level, NULL, NULL, NULL);
> > >  }
> > 
> > 
> > Thank you for the update.  Will this fix be part of 8.2?  We use
> > "iptables-restore --test" and this is causing considerable headaches.
> 
> Not by default. This ticket will address 8.3, backporting into 8.2 requires
> an explicit request. I'll discuss this with QA as they'll need to have
> capacity for it.
> 
> Cheers, Phil

Yes please - this causes a pretty nasty 100% CPU condition and the command is basically broken.
Do I have to do anything to request it be backported to 8.2?

Dan

(In reply to Dan from comment #5)
> Yes please - this causes a pretty nasty 100% CPU condition and the command
> is basically broken.
> Do I have to do anything to request it be backported to 8.2?

Opening a customer case linking to this ticket is always helpful in order to
justify capacities. Backporting to 8.2 is sufficient for you and we can leave
8.1 as-is?

(In reply to Phil Sutter from comment #6)
> (In reply to Dan from comment #5)
> > Yes please - this causes a pretty nasty 100% CPU condition and the command
> > is basically broken.
> > Do I have to do anything to request it be backported to 8.2?
> 
> Opening a customer case linking to this ticket is always helpful in order to
> justify capacities. Backporting to 8.2 is sufficient for you and we can leave
> 8.1 as-is?

Yes - we would need this backported to the earliest RHEL 8 version it effects.
When I tried in ECS using RHEL 8.0 and 8.1 it reproduced on both.

I assume that is because when I installed iptables they both installed:
iptables-1.8.4-10.el8.x86_64

So I assume this is tied to iptables or iptables-lib rpm? 

Do you know the version of the the rpms that are effected?

Thanks,
Dan

Customer case 02666328 has been opened referring to this issue.

Hi Dan,

(In reply to Dan from comment #12)
> (In reply to Phil Sutter from comment #6)
> > (In reply to Dan from comment #5)
> > > Yes please - this causes a pretty nasty 100% CPU condition and the command
> > > is basically broken.
> > > Do I have to do anything to request it be backported to 8.2?
> > 
> > Opening a customer case linking to this ticket is always helpful in order to
> > justify capacities. Backporting to 8.2 is sufficient for you and we can leave
> > 8.1 as-is?
> 
> Yes - we would need this backported to the earliest RHEL 8 version it
> effects.
> When I tried in ECS using RHEL 8.0 and 8.1 it reproduced on both.
> 
> I assume that is because when I installed iptables they both installed:
> iptables-1.8.4-10.el8.x86_64
> 
> So I assume this is tied to iptables or iptables-lib rpm? 
> 
> Do you know the version of the the rpms that are effected?

Yes, it is a bug in iptables RPM, iptables-1.8.4-10.el8 is the first one being
affected. I just checked and the iptables-restore shipped in RHEL8.1 doesn't
expose the problem, hence backporting to RHEL8.2 is sufficient. Sorry for the
confusion, I assumed the problematic patch was in RHEL8 for longer than that.

Cheers, Phil

Just to throw some more weight behind backporting to 8.2, we've also been bitten by this one. Please see customer case 02672005.


Thanks
Jerry

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (iptables bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4518