Bug 1299722

Summary: Automate acceptance testing: automate provisioning
Product: [Retired] Beaker Reporter: Roman Joost <rjoost>
Component: testsAssignee: beaker-dev-list
Status: CLOSED UPSTREAM QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: dcallagh, mjia, rjoost
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-26 04:58:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Joost 2016-01-19 06:15:48 UTC 
Description of problem:

At this point in time testing provisioning is still a manual job when it comes to acceptance testing in Beaker. This backlog item is to create automated provisioning tests to avoid long manual testing for finding regressions.
Comment 1 Dan Callaghan 2017-07-26 04:58:02 UTC 
I think this is essentially done, at this point. For bug 954265 I wrote bkr workflow-selftest, which produces a matrix of each supported combination (or least, the top tier most important) of distro and arch. The recipes include our existing modest selection of automated self-tests (currently just reboot and multihost sanity, see bug 961239 for expanding that set of tests).

The other piece of the puzzle is a Jenkins job to kick off the workflow and ensure that it comes back passing. We have that too, and have been using it the last few releases for automated acceptance testing:

https://git.beaker-project.org/cgit/beaker-jenkins-jobs/commit/?id=01b2a9b6b94cfbd8ac2093d143aa3b4f505fc847

Of course right now it always fails, because it's quite hard to get all our hardware to reliably boot and run. But the solution there is to just keep reducing the failure rate, whether it's fixing system configs, excluding unreliable distros/machines, or ultimately implementing automatic re-scheduling of aborted recipes:

https://beaker-project.org/dev/proposals/rescheduling-capability.html