This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 980423 - Beah requires at least one test result
Beah requires at least one test result
Status: CLOSED NOTABUG
Product: Beaker
Classification: Community
Component: beah (Show other bugs)
0.13
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: beaker-dev-list
tools-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-02 06:49 EDT by Nikolai Kondrashov
Modified: 2013-07-02 20:53 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 980660 (view as bug list)
Environment:
Last Closed: 2013-07-02 20:41:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nikolai Kondrashov 2013-07-02 06:49:47 EDT
Description of problem:
Tasks not reporting their success or failure via BeakerLib, but only via exit status are marked as failed.

Requiring reports with BeakerLib forces setup/teardown tasks to use it, unnecessarily complicating the code and adding dependencies, even though they don't actually run *any* tests and therefore don't need any detailed reports, just success or failure.

How reproducible:
Always

Steps to Reproduce:
1. Submit a task with empty executable "runtest.sh"

Actual results:
Task fails

Expected results:
Task passes

Additional info:
See example: https://beaker.engineering.redhat.com/jobs/444637 - the only failing task is the one not reporting results via BeakerLib.
Comment 2 Bill Peck 2013-07-02 09:49:33 EDT
the harness is reporting a fail if the task doesn't report *any* result, pass or fail.  You don't have to use beakerlib, just use report_result $TEST PASS if your task did what you expected it to do and you won't get a fail reported.
Comment 3 Nikolai Kondrashov 2013-07-02 11:05:57 EDT
"report_result" is an rhts function. That means that a task would require rhts just to report that it executed normally, where an exit status would be sufficient. Even if it was a part of a standard package, requiring explicit code for reporting success is a bit burdensome.

Wasn't there a mention of support for simple "tests" in Beaker documentation once, status of which was determined solely by their exit status? Were they deprecated?

Besides, what's wrong with a test suite producing no results? What if I wanted to have a test suite which would run a different set of tests depending on external conditions, such as environment variables specified via job XML, and certain conditions would result in *no* tests running?

Would I need to add special code to such a suite verifying that no tests ran (which can be non-trivial, if conditions were non-trivial) and then reporting a special dummy test just to keep the harness happy?
Comment 4 Bill Peck 2013-07-02 11:11:04 EDT
This is a requirement of the beah harness, I'm not saying I agree with it, but its been that way for quite some time.

The simple tests you mention are done via another rhts command

rhts-run-simple-test COMMAND

and that takes the return value and submits either a pass or fail to beaker.
Comment 5 Nikolai Kondrashov 2013-07-02 11:16:47 EDT
Thank you, Bill, I now remembered some mention of "rhts-run-simple-test" in the documentation and will try to use it as a workaround for setup/teardown tasks.

However, this still leaves the empty test suites. So, can we fix this limitation of Beah? Does "test harness" mean Beah, or should I reassign this bug to some other component?
Comment 6 Bill Peck 2013-07-02 11:20:13 EDT
the only harness right now is beah.  But we are working on some alternative ones.

The harness I'm working on will not report fail unless I get a non-zero return code.
Comment 7 Nikolai Kondrashov 2013-07-02 11:25:08 EDT
Is Beah still maintained? Does this Bug apply to it? Can we get it fixed?

What is the ETA of the alternative harness?
Comment 8 Bill Peck 2013-07-02 11:34:06 EDT
(In reply to Nikolai Kondrashov from comment #7)
> Is Beah still maintained?

I'll let the beaker devs answer that one.

> Does this Bug apply to it? Can we get it fixed?

Yes it applies to it.

We can certainly make the change, the question is will this break someone else's workflow?

> 
> What is the ETA of the alternative harness?

good question.  I'm pretty busy with my new job so I don't have a lot of time to spend on it but I have made some progress in the past week.  but I don't expect to have a working replacement for many months.
Comment 9 Nikolai Kondrashov 2013-07-02 15:51:33 EDT
Thank you, Bill.

I'm changing the subject, as rhts-run-simple-test can be used to remove the need for explicit result reporting for non-test tasks, but empty test suites would still fail.
Comment 10 Nick Coghlan 2013-07-02 20:41:45 EDT
No, we won't be adding support for empty test suites to beah. There's too much risk of incorrectly marking a test as passing when it is actually failing to execute anything but still returns 0 from runtest.sh.

Thus, we will always assume failure, unless we receive affirmative indication that the result is a pass.

Other harnesses will be free to behave differently, and that's one of the big reasons we're working to support them properly (especially autotest).

Note You need to log in before you can comment on or make changes to this bug.