Bug 1983108

Summary:	nft: tests: mark tests for features not in kernel as expected failures
Product:	Red Hat Enterprise Linux 8	Reporter:	Eric Garver <egarver>
Component:	nftables	Assignee:	Phil Sutter <psutter>
Status:	CLOSED WONTFIX	QA Contact:	Jiri Peska <jpeska>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.5	CC:	jpeska, todoleza
Target Milestone:	beta	Keywords:	FutureFeature, Reopened, Triaged
Target Release:	8.8
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2073472 (view as bug list)		Environment:
Last Closed:	2023-07-17 07:28:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2073472
Bug Blocks:	1983105

Description Eric Garver 2021-07-16 14:23:04 UTC

Related to bug 1983105.

When userspace is rebased it often includes features that are not available in the kernel. This leads to testsuite failures. Instead of masking these tests we should mark them as _expected_ failures. Then down the road when the kernel features make their way into RHEL the tests will be unexpected passes. This is our interrupt to revive the userpaces (nft) documentation and focus testing on the new features.

The end goal is a clearer definition of what is supported by both userspace and the kernel.

Comment 2 Phil Sutter 2021-08-11 12:32:58 UTC

To implement this detection, it should be sufficient to compare testsuite
stderr output with a stored expected output:

* Passing tests are ignored, regardless of whether known or added by a recent
  backport.

* Failing tests are assured to remain failing, so added kernel features are
  detected.

* Previously passing tests which start to fail are detected as well, so
  original testsuite purpose remains intact.

In nftables, json_echo and monitor need a patch, py and shell are fine. In
iptables, xlate-test.py and iptabltes-test.py need a patch, shell is fine.

Expected results should live in dist-git as extra source files. %prep stage
will install them into the same folder as the respective testsuite, named after
the testsuite binary with extension '.stderr.expect'. So for nftables' py
testsuite: 'tests/py/nft-test.stderr.expect'.

Comment 3 Eric Garver 2021-08-11 16:44:46 UTC

(In reply to Phil Sutter from comment #2)
> To implement this detection, it should be sufficient to compare testsuite
> stderr output with a stored expected output:
> 
> * Passing tests are ignored, regardless of whether known or added by a recent
>   backport.
> 
> * Failing tests are assured to remain failing, so added kernel features are
>   detected.
> 
> * Previously passing tests which start to fail are detected as well, so
>   original testsuite purpose remains intact.
> 
> In nftables, json_echo and monitor need a patch, py and shell are fine. In
> iptables, xlate-test.py and iptabltes-test.py need a patch, shell is fine.
> 
> Expected results should live in dist-git as extra source files. %prep stage
> will install them into the same folder as the respective testsuite, named
> after
> the testsuite binary with extension '.stderr.expect'. So for nftables' py
> testsuite: 'tests/py/nft-test.stderr.expect'.

ACK. Thanks for coming up with a simple solution. Sounds like a solid plan to me.

Comment 5 Phil Sutter 2022-04-08 15:23:10 UTC

This needs to revert RHEL-only patch "tests: Disable tests known to fail on RHEL8"

The following backports from upstream are required:

f74f277a3a965 ("tests: json_echo: Print errors to stderr")
7e1e90591a1c2 ("tests: monitor: Print errors to stderr")
2577939cc5758 ("tests: monitor: Continue on error")
f561a0ccb0524 ("tests: monitor: Hide temporary file names from error output")
6fa4ff5638583 ("tests: py: Don't colorize output if stderr is redirected")
2674c39bfdb77 ("tests: py: add missing JSON output.")
4eb71b2fe5ca0 ("tests: py: add missing test JSON output for TCP flag tests.")
6c84577b0d23d ("evaluate: add range specified flag setting (missing NF_NAT_RANGE_PROTO_SPECIFIED)")
e53fd42e4aee4 ("tests: py: update nat expressions payload to include proto flags")
741a06ac15d2b ("mergesort: find base value expression type via recursion")
c06f4bff1f45c ("tests: py: correct order of set elements in test JSON output.")

Commit 6c84577b0d23d fixes an actual bug aside from influencing netlink debug
output. Without commit 741a06ac15d2b, py testsuite run with '-j' triggers
segfault in nft.

Comment 10 Phil Sutter 2022-08-31 17:17:16 UTC

Bug 2073472 this ticket depends on missed RHEL8.7, therefore proposing this one for RHEL8.8 as well. Resetting qa_ack+ and thereby requesting explicit commitment for ITM-5.

Comment 12 RHEL Program Management 2023-01-16 07:27:38 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 15 RHEL Program Management 2023-07-17 07:28:13 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.