Description of problem:
While checking results of our tier test run, I realized, that
tests which are disabled for specific archs in test metadata were
run on these architectures as well. This results in false fails.
For example see /CoreOS/freeipmi/Sanity/initscript results in the
following matrix report:
I used bkr workflow-simple for generating the xml, which does not
support arch limiting. Is this functionality planned for simple
workflow? Or will it be supported by scheduler? (For example
skipping invalid tests-arch combinations.) Or should this be
handled solely by submitter while generating the workflow?
Will the metadata such as releases & archs be obsoleted in Beaker
and moved rather to TCMS's environments? We need to solve this
problem to be able to run our regular test tiers. So I suggest
this as a blocker.
This is a bug. I just verified this myself. The way this is suppose to work is it should filter that task out for all arches but i386 and x86_64 at submission time.
Found and fixed.
Author: Bill Peck <email@example.com>
Date: Fri Jun 4 11:05:45 2010 -0400
fix bz600353 - only add applicable tasks to recipe, fixes arch and osmajor exclude
Seems the release exclusion still has some issues. I've scheduled
python related tests using:
bkr workflow-simple --distro RHEL6.0-20100617.0 --variant
Server --whiteboard "mod_python should be excluded" --package
python --nowait --debug --prettyxml --arch x86_64
and in spite of the RHEL6 excluded in testinfo.desc:
the /CoreOS/mod_python/Sanity/smoke test is still scheduled:
The issue is we don't have the aliases in Beaker yet. I have the field for them but I haven't implemented the editor yet.
Until I can get that you could spell it out long and it will work.
Is there a bug tracking the alias support? I'd like to see it as a blocker:
1) it works in RHTS
2) we have many (maybe hundreds) of tests using the shortened syntax.
Fixed in git. We will be able to edit the OSMajor alias in the next release.
I'll have to do a mass re-import of the tasks that depend on the alias since its during the import of the task which looks for the alias.
That shouldn't be a problem though.
This has been released.
I'm currently running a test import on beaker-stage to make sure the mass import of tasks will work as expected.
After thats done I'll run it again on prod.
Works nice, thanks for the fix!
I'm using short aliases in all my tests and I still see this problem. Is re-submitting tests to beaker necessary in order to fix this problem?
Could you please update also ?
The fix is not working for me. See /CoreOS/passwd/Sanity/smoke-test, this test has been submitted to beaker yesterday and its Makefile contains the following:
@echo "Releases: RHELServer5 RHELClient5 RHEL6" >> $(METADATA)
Therefore it should run for RHEL5 and RHEL6 _only_. But I have recently scheduled job for RHEL4:
bkr workflow-simple --task /CoreOS/passwd/Sanity/smoke-test --distro RHEL4-U8
See https://beaker.engineering.redhat.com/jobs/10715 .
What families are relevant for makefiles,  or ?
I know what the problem is. Currently the code only has excludes implemented.
The majority of these are best handled via excludes. More than likely you are going to want that test to be run for rhel7 as well.
We will implement includes as well, but until its implemented there is a workaround:
@echo "Releases: -RHEL3 -RHEL4" >> $(METADATA)
From: Miroslav Vadkerti <firstname.lastname@example.org>
Date: Wed, 11 Aug 2010 10:19:32 -0400 (EDT)
I want that my test runs only for RHEL3 and RHEL4 and only for these.
With only exclude option present I would have to modify my test every time
new release is out. Of course this is not so common, but I would like to
see this option available. In your case of course it is *advised* to use
From: Zbysek MRAZ <email@example.com>
Date: Wed, 11 Aug 2010 16:25:39 +0200
+1 for includes:
In every release there is different version of package eg httpd. When
there is a CVE which affects only version 2.0.x I don't want the test to
run on RHEL4 neither on RHEL6.
+1 for excludes:
When I have different syntax of config file, package names or even
command itself changed after long period of time I want only exclude eg.
the latest release. When the test runs perfectly on RHEL3-5 I want to
exclude only the RHEL6
The problem starts when these kind of tests are included in
"multirelease" test tiers when there are failed results just because the
test is not supposed to run on the particular release.
Bill, reading the above - please put in the "inversion" solution - so when releases is set to RHEL5, do exclude everything else.
Looking with a great hope for 0.5.59 as this issue is causing a
lot of false fails in our tier runs, which then have to be handled
Changes committed to git.
*** Bug 636886 has been marked as a duplicate of this bug. ***