Related to bug 1983105. When userspace is rebased it often includes features that are not available in the kernel. This leads to testsuite failures. Instead of masking these tests we should mark them as _expected_ failures. Then down the road when the kernel features make their way into RHEL the tests will be unexpected passes. This is our interrupt to revive the userpaces (nft) documentation and focus testing on the new features. The end goal is a clearer definition of what is supported by both userspace and the kernel.
To implement this detection, it should be sufficient to compare testsuite stderr output with a stored expected output: * Passing tests are ignored, regardless of whether known or added by a recent backport. * Failing tests are assured to remain failing, so added kernel features are detected. * Previously passing tests which start to fail are detected as well, so original testsuite purpose remains intact. In nftables, json_echo and monitor need a patch, py and shell are fine. In iptables, xlate-test.py and iptabltes-test.py need a patch, shell is fine. Expected results should live in dist-git as extra source files. %prep stage will install them into the same folder as the respective testsuite, named after the testsuite binary with extension '.stderr.expect'. So for nftables' py testsuite: 'tests/py/nft-test.stderr.expect'.
(In reply to Phil Sutter from comment #2) > To implement this detection, it should be sufficient to compare testsuite > stderr output with a stored expected output: > > * Passing tests are ignored, regardless of whether known or added by a recent > backport. > > * Failing tests are assured to remain failing, so added kernel features are > detected. > > * Previously passing tests which start to fail are detected as well, so > original testsuite purpose remains intact. > > In nftables, json_echo and monitor need a patch, py and shell are fine. In > iptables, xlate-test.py and iptabltes-test.py need a patch, shell is fine. > > Expected results should live in dist-git as extra source files. %prep stage > will install them into the same folder as the respective testsuite, named > after > the testsuite binary with extension '.stderr.expect'. So for nftables' py > testsuite: 'tests/py/nft-test.stderr.expect'. ACK. Thanks for coming up with a simple solution. Sounds like a solid plan to me.
This needs to revert RHEL-only patch "tests: Disable tests known to fail on RHEL8" The following backports from upstream are required: f74f277a3a965 ("tests: json_echo: Print errors to stderr") 7e1e90591a1c2 ("tests: monitor: Print errors to stderr") 2577939cc5758 ("tests: monitor: Continue on error") f561a0ccb0524 ("tests: monitor: Hide temporary file names from error output") 6fa4ff5638583 ("tests: py: Don't colorize output if stderr is redirected") 2674c39bfdb77 ("tests: py: add missing JSON output.") 4eb71b2fe5ca0 ("tests: py: add missing test JSON output for TCP flag tests.") 6c84577b0d23d ("evaluate: add range specified flag setting (missing NF_NAT_RANGE_PROTO_SPECIFIED)") e53fd42e4aee4 ("tests: py: update nat expressions payload to include proto flags") 741a06ac15d2b ("mergesort: find base value expression type via recursion") c06f4bff1f45c ("tests: py: correct order of set elements in test JSON output.") Commit 6c84577b0d23d fixes an actual bug aside from influencing netlink debug output. Without commit 741a06ac15d2b, py testsuite run with '-j' triggers segfault in nft.
Bug 2073472 this ticket depends on missed RHEL8.7, therefore proposing this one for RHEL8.8 as well. Resetting qa_ack+ and thereby requesting explicit commitment for ITM-5.
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.