Description of problem: At this point in time testing provisioning is still a manual job when it comes to acceptance testing in Beaker. This backlog item is to create automated provisioning tests to avoid long manual testing for finding regressions.
I think this is essentially done, at this point. For bug 954265 I wrote bkr workflow-selftest, which produces a matrix of each supported combination (or least, the top tier most important) of distro and arch. The recipes include our existing modest selection of automated self-tests (currently just reboot and multihost sanity, see bug 961239 for expanding that set of tests). The other piece of the puzzle is a Jenkins job to kick off the workflow and ensure that it comes back passing. We have that too, and have been using it the last few releases for automated acceptance testing: https://git.beaker-project.org/cgit/beaker-jenkins-jobs/commit/?id=01b2a9b6b94cfbd8ac2093d143aa3b4f505fc847 Of course right now it always fails, because it's quite hard to get all our hardware to reliably boot and run. But the solution there is to just keep reducing the failure rate, whether it's fixing system configs, excluding unreliable distros/machines, or ultimately implementing automatic re-scheduling of aborted recipes: https://beaker-project.org/dev/proposals/rescheduling-capability.html