We write our CI tests with python behave in [1]

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci


The DHCP plugin example
========================

This is an example where this matters.

NetworkManager supports `[main].dhcp=` plugins (relevant choices are "internal" and "dhclient"). In many aspects, both plugins should just work the same, and the user should notice a difference.

In NetworkManager-ci, some tests explicitly select one or the other plugin. But most tests just leave the system default. The system default depends on the distro, but as we no longer run latest NM-ci tests on rhel7, effectively all tested versions use the internal plugin. That means, our coverage for dhclient is uneven at best, in practice it's severely lacking.

It does not scale to just start copying some tests and add a tag to use one or the other DHCP plugin. Instead, we need a convenient way so that a certain test can run twice, with both DHCP plugins -- without fully cloning the test.


The General Problem
===================

Our current approach for running a test under two slightly different setups is to clone it. That does not scale. For the DHCP plugin above, we want that a large number of DHCP related tests runs with both plugins (all the time).

There are behave scenario outlines ([2]), maybe those could be used. However, they don't seem very suited to me. It should be very low-effort to say that a test should run with different setups.

[2] https://behave.readthedocs.io/en/stable/gherkin.html#scenario-outlines

I imaging a new tag like "@dhcp-any" which means that the test runs twice, with both plugins.
Or better, something more generic like: "@matrix:dhcp=internal:dhclient" which would have the effect that it runs the test twice, setting some context variable "dhcp", once to "internal" and once to "dhclient". 

that could then be combined with "@matrix:dhcp=internal:dhclient @matrix:dns:default-systemd-resolved" which would then possibly run all 4 combinations. There could also be a "@skip-matrix:dhcp=internal,dns=default" tag to skip certain combinations.

Anyway, you get the idea...