Bug 954265

(In reply to comment #0)
> 2. Roles are populated correctly (for both host and guest recipes)

There's probably no good way to test this one, since the correct values depend on the FQDNs of the systems in the recipe set (which will vary across Beaker installations and job runs). But you can test the rhts-sync-* commands work using the role env vars.

> 3. rhts-sync-block works correctly (between separate recipes and between
> host and guest recipes)

This one exists already, more or less:

http://git.beaker-project.org/cgit/beaker/tree/Tasks/distribution/beaker/Sanity/sync-set_block-tests

Explicitly reducing the scope of this one to just a single simple reference task, and the instructions for running it as a basic sanity check on a Beaker installation.

Since the sync-set_block tests sanity check task already exists and covers items on 1, 2 and 3 on the list, it probably makes sense to document using that as the reference task.

The other thing we really need a sanity test for is rhts-reboot (bug 1320320).

I think we can go one better than just documentation here... We already have bkr harness-test, which is a workflow command to just fire off a recipe for every distro arch variant combination that exists in Beaker. The idea of that one is to validate that the harness works in every possible environment, although it doesn't actually *assert* anything or do anything besides run /distribution/install.

One flaw with machine-test's approach is that it tries to figure out what recipes to run based on distros already imported into Beaker, which means it suffers from the problem where Beaker giving the wrong answer about distros could cause it to produce a false positive by not running enough tests. I would rather see a hardcoded matrix of all the combinations that we actually expect to work. We would probably also leave off some things which are currently picked up by harness-test, like EOL'ed Fedora releases, which we support on a best-effort basis only and which we probably don't need to cover in regression testing for every single Beaker release.

There are some other dimensions that we might want to eventually add into the matrix too, like single host vs. multi-host, beah vs. restraint, rhts-compat enabled vs. disabled, ...

I'm picturing something a little like OpenStack Tempest, where you say "here is my Beaker environment, go do stuff and make sure that everything works".

We could have a new workflow command (probably maintained outside of the main Beaker source tree, so that it's not tied to the same release cycle as Beaker itself) called something like workflow-selftest. Similar to harness-test it would submit a big fat job which tries to exercise all the provisioning combinations that we care about and executes all the self-test sanity tasks we have (starting with /distribution/beaker/Sanity/rhts-sync_block-tests plus one for rhts-reboot).

It would assume that the environment is set up correctly, with Beaker installed and configured and harness packages available etc. The Beaker sanity tasks would also need to have been uploaded. The workflow command could provide some help with that, for example by first checking that the necessary tasks are in Beaker and perhaps uploading them if not(?).

Ideally it should be easy to run this against an environment created by the beaker-in-a-box Vagrant+Ansible scripts.

Then testing a new Beaker version (or new harness package, or new environment deployment or configuration change, etc) would mean running beaker workflow-selftest and waiting for the job to give a Pass result.

In future if we start to add some sanity checks where the expected result is different (for example, testing that the EWD aborts a recipe) we could devise some simple conventions, like indicating the expected result of the recipe in its whiteboard, and then we would have a companion command which can inspect a finished workflow-selftest job and ensures that the results match the expectations.

Just noticed that bug 961239 was deliberately cloned to be the broader "complete suite of self-tests" RFE, so all my ideas above probably belong on that... Regardless I think it is worth implementing the bkr workflow-selftest for just the two sanity checks here (sync/block and rhts-reboot) and bug 961239 can continue to expand on the tests done by workflow-selftest.

Forgot to set this to ASSIGNED. I have been working on this, will have something to push soon.

I've set up a new git repo here, with an initial version of the workflow-selftest command:

https://git.beaker-project.org/cgit/beaker-workflow-selftest/

At least initially we will want to evolve it faster than Beaker itself, eventually we could look at moving it into beaker-client as another normal workflow command.

What we have now is good enough to consider this done, I think.

There is the old existing /distribution/beaker/Sanity/sync-set_block-tests:

https://git.beaker-project.org/cgit/beaker-meta-tasks/tree/Sanity/sync-set_block-tests

and a new task /distribution/beaker/Sanity/reboot-tests:

https://git.beaker-project.org/cgit/beaker-meta-tasks/commit/?id=3725e7e3ba00a29a323372a83655b4bb2fd64a99

and the workflow-selftest command to schedule them:

https://git.beaker-project.org/cgit/beaker-workflow-selftest/

You can run that from a git checkout by using:

PYTHONPATH=src bkr workflow-selftest

assuming you also have this bkr patch to make it load subcommands from entry points:

https://git.beaker-project.org/cgit/beaker/commit/?id=ddbe32c1feb3dd2248708dacbc12f6e645e9f2a6

Lastly, we have a Jenkins job to tie all this together, which we intended to use for acceptance testing of new Beaker releases (assuming we can get it running stably enough...):

https://git.beaker-project.org/cgit/beaker-jenkins-jobs/commit/?id=01b2a9b6b94cfbd8ac2093d143aa3b4f505fc847