Bug 1917398

I looked ad the (non-gating) failures in CI for the rebase:

Sanity/upstream-testsuite:
- from tests/py, kernel lacks:
  - support for netfilter egress hook
  - support for TCP option removal in exthdr expression
  - support for inner header matching in payload expression
- in tests/shell:
  - tests/shell/testcases/chains/0021prio_0 uses egress hook
  - tests/shell/testcases/maps/typeof_integer_0 uses inner header expression
    (actually, inner header offset in raw payload expression)
  - tests/shell/testcases/maps/typeof_raw_0 and
    tests/shell/testcases/sets/typeof_raw_0 also use inner header

Regression/rhel-8-1-rebase-patches:
- "segtree debug crashing" test checks segtree debug output for an anonymous
  set with ranges which is empty - segtree code got a rewrite, probably debug
  output is less verbose now
- There is a warning in dmesg output[1] (net/core/flow_dissector.c:985) - this
  needs investigation, but is probably a kernel bug

Regression/Fix-for-poor-performance-of-JSON-API-echo-support:
- This was written for bug 1835300, the fix claims reduction in restore
  run-time from 27s down to 1.4s.
- Same restore takes 1m35s on a 1minutetip VM even without --echo.
- According to perf[2], most of the time is spent sorting the set elements.
- So we're regressing, but for a different bug.


[1] http://artifacts.osci.redhat.com/baseos-ci/brew-build/45/87/87/45878743/https___baseos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com-ci-artemis/28541/tmp1fnuUn.01/recipes/1/tasks/13/results/1654794774/logs/resultoutputfile.log
[2] FTR, here's the top of perf report:
  23.16%  nft      libnftables.so.1.1.0  [.] expr_msort_value
  16.59%  nft      libnftables.so.1.1.0  [.] list_expr_sort
  14.97%  nft      libgmp.so.10.4.1      [.] __gmpn_copyi_x86_64
   7.27%  nft      libnftables.so.1.1.0  [.] set_to_range
   6.26%  nft      libc.so.6             [.] _int_free
   4.68%  nft      libgmp.so.10.4.1      [.] __gmpz_set
   4.33%  nft      libnftables.so.1.1.0  [.] interval_set_eval.lto_priv.0
   3.76%  nft      libc.so.6             [.] malloc
   3.15%  nft      libnftables.so.1.1.0  [.] range_expr_value_high
   2.59%  nft      libc.so.6             [.] cfree.5
   1.82%  nft      libgmp.so.10.4.1      [.] __gmpz_cmp

(In reply to Phil Sutter from comment #3)
> Regression/Fix-for-poor-performance-of-JSON-API-echo-support:
> - This was written for bug 1835300, the fix claims reduction in restore
>   run-time from 27s down to 1.4s.
> - Same restore takes 1m35s on a 1minutetip VM even without --echo.
> - According to perf[2], most of the time is spent sorting the set elements.
> - So we're regressing, but for a different bug.

I reported the issue upstream. It is caused by new overlap detection and
element auto-merging code which causes significant overhead for each individual
'add element' command. If these commands are combined into a single one adding
all elements at once, the overhead is not noticeable.

To avoid the problem, Pablo implemented automatic command coalescing for 'add
element' commands. Hence I will backport these commits:

8fafe4e6b5b30 ("tests: shell: runtime set element automerge")
498a5f0c219d8 ("rule: collapse set element commands")
87ba510fc704f ("intervals: do not report exact overlaps for new elements")

There are two related fixes to be backported as well:

d434de8b50dcf ("intervals: do not empty cache for maps")
59e3a59221fb8 ("intervals: Do not sort cached set elements over and over again")

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nftables bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8381