On scheduler store tests under version control (git repo). When scheduler sends tests to the test hosts it would be a compressed tar ball. (gzip, bzip, zip). Pros: - everything is in version control. guaranteed. - it could make test administration easier - auditability, reproducability: e.g. test <x> ran at time <t> with test version <sha1> Cons: - tied to git (potentially) - can mitigate by coding some shim layer to insulate the innards exposure to git specifics. make it scm agnostic as much as possible
Please do not do this, keep it rpm-based. With rpms you can have Requires of other tests, you can Require other packages, you use the standard tools (rpm, yum) to install the tests. Yes, it would be helpful if the tests were in git instead of cvs, but the product which gets installed should still be rpm.
IF you plan to refactor the thing, it would be helpful to think about ways of making "bundles" of tests. We at Satellite frequently run long series of tests (Gather OS, arch, and other info about the system, retrieve proper config files, prepare yum repos, upgrade system to certain state, run installation of Satellite, run sanity checks) of which we prefer to think as one entity -- "Install Satellite". Currently we do it with poor-man's approach, by Requiring rpms of other tests and then just manually running runtest.sh from their /mnt/tests/... directories. However, those "child" tests are not first-class tests in Beaker logs and WebUI, and we always face the danger that it will break in the future, that starting new journal when one is already started will stop working. It would be great if we could easily make list of tests and name it as test, and if it was supported and it just worked.
What about having "Install Satellite" as a TCMS test-plan that contains testcases (Gather OS, arch, and other info about the system, retrieve proper config files, prepare yum repos, upgrade system to certain state, run installation of Satellite, run sanity checks) mapped to Beaker tests. You won't need rpm-dependencies between task rpms, child tests will be represented in job results as first-class citizens and as a bonus you will have the testing tracked in TCMS. PS: lets continue the discussion via email, it is IMHO bit off-topic to this bug.
(In reply to comment #3) > What about having "Install Satellite" as a TCMS test-plan that contains Nope. It's not testing only for which we use beaker -- we use it as a general purpose provisioning tool. So if I want to investigate an issue on Satellite 5.3.0, I can say run test which will provision the OS, install the Satellite, and reservesys it. Not just for QA but for engineering as well. I do not want TCMS pulled into the picture because that would mean I'd need to work with with yet another system.
I feel that having single GIT rather than CVS + RPMs could help in a couple of ways. One, in terms of the ease with which we are able to 'share' tests with third-parties. Right now, i am dealing with partner's new tasks/old/patches via email attachments/manual patching/commit to cvs/update beaker task repo. We've nearly 50 tests already from partners. There's already a relatively large amount of churn within just those test cases, causing me a bit of a headache. If we scale 'task sharing' out further, (I) we are in deep trouble. And that's only RH dealing with partner's tests. We also share our tests with partners and there's a lot of churn there too which i (and others) have to manually track and ensure remain up2date. In a mail thread I started several months ago[1], i posed the question of whether we couldn't use a new git based workflow to obtain a simple, single revision controlled repo for task sources, in combo with a build system (brew) to produce installable packages of targeted packager-dependent type (RPM, EXE, DEB...). So GIT system, according to my theory anyway, could also give us a streamlined, automatable source -> package build -> schedule workflow. But i got no response to the question. :( Within the metadata (TCMS) it should be possible (in my mind) to softly create all the necessary dependencies so that when someone schedules a test, they point to either a particular GIT version hash or just a test name and beaker would know which version of tests to pull (head/master if unspecified), sends those sources off to build (RPM, EXE, TAR...) as appropriate or use cached if builds already available, determine dependencies (build them too if not available), then build repositories on the fly. I suppose it could work if we pointed to partner GIT repo's too, seamlessly. But then again, maybe this wouldn't work. But that, unfortunately, isn't something I'd be able to answer. I hardly have a clue about what i'm talking about here :) [1] http://post-office.corp.redhat.com/mailman/private/qe-dept-list/2010-June/msg00052.html
I was dreaming this morning a little about having http://gitorious.org integration. :)
(In reply to comment #5) > I feel that having single GIT rather than CVS + RPMs could help in a couple of > ways. Having single git repo would mean that if I wanted to change a single test, I'd need to git clone the whole repo, of which 99 % would be of no interest to me. While with cvs (or per-package/metapackage git repo) you'd only download what you'd be interested in. > One, in terms of the ease with which we are able to 'share' tests with > third-parties. Right now, i am dealing with partner's new tasks/old/patches via > email attachments/manual patching/commit to cvs/update beaker task repo. We've > nearly 50 tests already from partners. There's already a relatively large > amount of churn within just those test cases, causing me a bit of a headache. You can always create local git repo on top of your CVS working tree, to help you manage the changes. Please note that I'm not opposed to a change of the SCM, it's the step away from rpm as the installable test which worries me a lot.
(In reply to comment #7) > (In reply to comment #5) > > I feel that having single GIT rather than CVS + RPMs could help in a couple of > > ways. > > Having single git repo would mean that if I wanted to change a single test, I'd > need to git clone the whole repo, of which 99 % would be of no interest to me. > While with cvs (or per-package/metapackage git repo) you'd only download what > you'd be interested in. > Interesting. I guess i had envisioned each test would be its own self-contained repo. Implications of this? Could it work if we had a Beaker 'git hub' and each test mimicked a 'project'? hmm... I agree that it would be entirely pointless if all one could do is clone the entire task repo. That's terrible overhead. We wouldn't want to share the entire repo with third-parties anyway. So I'd still have to end up emailing individual test cases at the end of the day, if that's the case; fail. > > One, in terms of the ease with which we are able to 'share' tests with > > third-parties. Right now, i am dealing with partner's new tasks/old/patches via > > email attachments/manual patching/commit to cvs/update beaker task repo. We've > > nearly 50 tests already from partners. There's already a relatively large > > amount of churn within just those test cases, causing me a bit of a headache. > > You can always create local git repo on top of your CVS working tree, to help > you manage the changes. Please note that I'm not opposed to a change of the > SCM, it's the step away from rpm as the installable test which worries me a > lot. Thanks for the hint. I'm thinking that's what i'll try to do in the mean time; git repo per partner. To the point about moving away from using rpm as task packaging. How does this fit into the goal of enabling beaker to test cross platform? Maybe I'm wrong, but I figure it implies we'll have to build tasks so they're installable on OSes other than RPM based. If it's right, then it'll mean we need to support non-rpm packaging. Wouldn't it be smart for us to find a packaging neutral solution? Question; if we move away from RPM to sources only (version control), where will the 'make package' event happen? And how will the 'package type (rpm,exe,...)' be determined, assuming the above is true and we're targeting the enablement of non-rpm based tasks? Is the following feasible? * testers only work with sources under version control (git) * middle layer between SCM and scheduler determines what packaging is required based on job data and meta data * build system gets that info and builds packages and appropriate repos (using cache when available) * scheduler is sent repo info and tests run. This would obviously imply that task sources would have to support building packages for given packaging types. At first, we could officially only support 'RPM' then expand to others. Or is this not really feasible? -Chris
Binaries often behave different way on different releases. Different command line options/switches can be present, thus the invocation syntax differs and sometimes completely different coding approach must be used between releases. Tests can, of course, detect release and behave the right way using if - elif - elif - else - fi constructions or something similar. However, this may be hard to read, and unpleasant to maintain in case the code is repeating somehow. In such cases, other simple solution is to create, more tests with majority of similar code having similar name but with different suffix, e.g. -el5, -el6. This can be easily done with the current infrastructure. "Backporting" changes is much easier in this case than in the above one. If we switched to git and there was some elegant way of solving this, I would greatly appreciate it. However, just remembering which commit (hash) works on which release isn't sufficient IMO as also editing tests for older releases should be possible without changing their counterparts for newer releases. Maybe some kind of branching could help here. But this should definitely not complicate simple situations where the same test is usable for all the rhels ever released ;-)
By removing RPM will be easier to support other OS/Platforms such as Windows. Test harness should install dependencies (currently described in the Makefile/METADATA file) when installing the selected test case. If we use git (or other SCM) instead of RPM why not use git clone on the test machine instead of tarballs.
How does 'git clone' scale in time? Not good. It's extensive operation fetching repository together with all the history. It triggers considerable load on a 'provider'. Given the amount of beaker tests, git cloning/archiving of all the redhat tests, is not optimal for beaker job. We should keep in mind, that the test should be fast, very fast. It comes back to the initial problem of indivisible design of git. How can I give all-the-regression-tests for given package with revision history to the partner? How can I checkout only a few directories from git?
Just to note here. Many times, in day to day QA work experience, having tests packaged in the rpm proved to be very good decision. It helped me many times to track causation of problem in straight well-known (rpm/yum) way.
Šimon, can you describe the particular use-cases where RPM proved useful?
BTW relevant bit of recent discussion of Windows support in Beaker: On 24.6.2011 19:08, Edward Rousseau wrote: > The intent is to migrate the existing Beaker task to look asides managed > by individual groups. This provides more flexibility for groups to > revise, change and customize libraries around their individual groups' > workflows. The harness itself is designed to load whatever test assets > specified from a URI (look aside) leaving the management of those assets > to the groups themselves. Lab infrastructure will be added for groups to > store their assets. Groups themselves will need to develop policy on how > they upgrade, develop and version the test assets they use. This puts > more control in the hands of the groups using Beaker.
I dare to suggest that if someone requires having test wrapped in RPM (DEB, JAR, ZIP, whatever), one can always build it from SCM and continue as always. But forcing the use of RPM is limiting when it comes to working with something else then RHEL. And heck, even RPM format is not carved in stone (compression, signatures) open another can of worms (or compatibility issues, if you wish).
(In reply to comment #15) > I dare to suggest that if someone requires having test wrapped in RPM (DEB, > JAR, ZIP, whatever), one can always build it from SCM and continue as always. > > But forcing the use of RPM is limiting when it comes to working with something > else then RHEL. And heck, even RPM format is not carved in stone (compression, > signatures) open another can of worms (or compatibility issues, if you wish). This is not about "someone requires". This is about core infrastructure for running tests. Tests are programs. Program in RHEL and Fedora are distributed as .rpms -- that's their native format. Coming up with different packaging mechanism will mean we will be fighting instability in that mechanism for many months. You need to have some packaged form of the tests because you need for the tests to be repeatable. Again, rpm already uses the notion of EVR, it gives you the ability to rpm -V your test programs -- again, all standard tools that come with every recent RHEL. By all means, for tests that are to be run on Ubuntu, build .deb packages from the SCM.
Not about Ubuntu (OK, I asked for it by listing DEB :)), but we support running out products on Windows, I've seen some compatibility tests with Solaris and JBoss (though we don't see too much of that in Beaker). And as I said - with the hybrid approach nobody would be forcing you to use the SCM method. There is ongoing work to make Beaker harness-agnostic, which should make it even more flexible for different teams.
(In reply to comment #11) > How does 'git clone' scale in time? > > Not good. It's extensive operation fetching repository together with all > the history. It triggers considerable load on a 'provider'. This is why shallow clones exist. For the sake of grabbing some code and running it out of a git repo, there is no need to drag down the entire history. The latest commit is all you need and it is a significantly lower weight process. > Given the amount of beaker tests, git cloning/archiving of all the redhat > tests, is not optimal for beaker job. We should keep in mind, that the test > should be fast, very fast. I would not recommend "all the tests" in a single repo. That's insanity. Each package should have it's own tests repo (or branch). > It comes back to the initial problem of indivisible design of git. How can > I give all-the-regression-tests for given package with revision history to > the partner? How can I checkout only a few directories from git? You can do shallow clones, you can do an export of the files only, you can do a filter-branch to trim out things on a repo, there are options.
(In reply to comment #19) > (In reply to comment #11) > > How does 'git clone' scale in time? > > > > Not good. It's extensive operation fetching repository together with all > > the history. It triggers considerable load on a 'provider'. > > This is why shallow clones exist. For the sake of grabbing some code and > running it out of a git repo, there is no need to drag down the entire history. > The latest commit is all you need and it is a significantly lower weight > process. Can Shallow clones work without the server side doing a lot of work? I think we tried this and some option has to be turned on the server side and eng-ops didn't like the load it put on the machine. > > > Given the amount of beaker tests, git cloning/archiving of all the redhat > > tests, is not optimal for beaker job. We should keep in mind, that the test > > should be fast, very fast. > > I would not recommend "all the tests" in a single repo. That's insanity. Each > package should have it's own tests repo (or branch). > Agreed. a separate repo per package would be good. > > It comes back to the initial problem of indivisible design of git. How can > > I give all-the-regression-tests for given package with revision history to > > the partner? How can I checkout only a few directories from git? > > You can do shallow clones, you can do an export of the files only, you can do a > filter-branch to trim out things on a repo, there are options. I think maybe it was filter-branch that caused the load/memory issues. but if we have different repos per package it may not be a problem.
It is my understanding that doing a shallow clone may cause slightly more up fron CPU time on the server to deduce what to send, but ultimately far less network and cpu time to send the data over. As for filter-branch, that happens client side, you'd do a clone, then filter, then send off to outside of RHT, so that would not come into play.
Just note, that beaker tests in git should be IMO also tightly bound to their counterparts living in the TCMS system (which is a general database of beaker tests used by QA). I created a separate RFE for this bz729247. I'd like to bring this issue to discussion too.
Any recent work on supporting GIT uri's in addition to existing RPM support for tasks? Thanks
Bulk reassignment of issues as Bill has moved to another team.
We no longer have any plans to add this capability to the current beah test harness. The restraint harness already supports running tests directly from git: http://restraint.readthedocs.org/en/latest/jobs.html And here are the instructions for running from Beaker: http://restraint.readthedocs.org/en/latest/using.html However, this could be better tied into the documentation. In particular, using restraint to *test* a new version of a task directly from git before building and uploading an RPM to the main task library is a workflow that should be covered explicitly.
Patch available on gerrit: http://gerrit.beaker-project.org/4820
I've simplified the task tutorial we have in our user-guide and focused on creating/running the task using restraint+git.
The updated documentation is available here: https://beaker-project.org/docs-develop/user-guide/example-task.html
... however it doesn't really cover this suggestion from Nick: (In reply to Nick Coghlan from comment #26) > However, this could be better tied into the documentation. In particular, > using restraint to *test* a new version of a task directly from git before > building and uploading an RPM to the main task library is a workflow that > should be covered explicitly. so I'm not sure if it might need some more work?
(In reply to Dan Callaghan from comment #31) Roman pointed me at this section: https://beaker-project.org/docs-develop/user-guide/example-task.html#testing-the-task-from-a-git-repository which more or less covers it, although it doesn't explicitly talk about testing development versions of a task and then uploading it as an RPM later, it now just glosses over the existence of the task library entirely. But I think that's probably fine.
Beaker 23.0 has been released.