Red Hat Bugzilla – Bug 980423
Beah requires at least one test result
Last modified: 2013-07-02 20:53:56 EDT
Description of problem:
Tasks not reporting their success or failure via BeakerLib, but only via exit status are marked as failed.
Requiring reports with BeakerLib forces setup/teardown tasks to use it, unnecessarily complicating the code and adding dependencies, even though they don't actually run *any* tests and therefore don't need any detailed reports, just success or failure.
Steps to Reproduce:
1. Submit a task with empty executable "runtest.sh"
See example: https://beaker.engineering.redhat.com/jobs/444637 - the only failing task is the one not reporting results via BeakerLib.
the harness is reporting a fail if the task doesn't report *any* result, pass or fail. You don't have to use beakerlib, just use report_result $TEST PASS if your task did what you expected it to do and you won't get a fail reported.
"report_result" is an rhts function. That means that a task would require rhts just to report that it executed normally, where an exit status would be sufficient. Even if it was a part of a standard package, requiring explicit code for reporting success is a bit burdensome.
Wasn't there a mention of support for simple "tests" in Beaker documentation once, status of which was determined solely by their exit status? Were they deprecated?
Besides, what's wrong with a test suite producing no results? What if I wanted to have a test suite which would run a different set of tests depending on external conditions, such as environment variables specified via job XML, and certain conditions would result in *no* tests running?
Would I need to add special code to such a suite verifying that no tests ran (which can be non-trivial, if conditions were non-trivial) and then reporting a special dummy test just to keep the harness happy?
This is a requirement of the beah harness, I'm not saying I agree with it, but its been that way for quite some time.
The simple tests you mention are done via another rhts command
and that takes the return value and submits either a pass or fail to beaker.
Thank you, Bill, I now remembered some mention of "rhts-run-simple-test" in the documentation and will try to use it as a workaround for setup/teardown tasks.
However, this still leaves the empty test suites. So, can we fix this limitation of Beah? Does "test harness" mean Beah, or should I reassign this bug to some other component?
the only harness right now is beah. But we are working on some alternative ones.
The harness I'm working on will not report fail unless I get a non-zero return code.
Is Beah still maintained? Does this Bug apply to it? Can we get it fixed?
What is the ETA of the alternative harness?
(In reply to Nikolai Kondrashov from comment #7)
> Is Beah still maintained?
I'll let the beaker devs answer that one.
> Does this Bug apply to it? Can we get it fixed?
Yes it applies to it.
We can certainly make the change, the question is will this break someone else's workflow?
> What is the ETA of the alternative harness?
good question. I'm pretty busy with my new job so I don't have a lot of time to spend on it but I have made some progress in the past week. but I don't expect to have a working replacement for many months.
Thank you, Bill.
I'm changing the subject, as rhts-run-simple-test can be used to remove the need for explicit result reporting for non-test tasks, but empty test suites would still fail.
No, we won't be adding support for empty test suites to beah. There's too much risk of incorrectly marking a test as passing when it is actually failing to execute anything but still returns 0 from runtest.sh.
Thus, we will always assume failure, unless we receive affirmative indication that the result is a pass.
Other harnesses will be free to behave differently, and that's one of the big reasons we're working to support them properly (especially autotest).