Bug 1324607

Summary: Add support to beaker for SKIP value for results
Product: [Retired] Beaker Reporter: Scott Poore <spoore>
Component: generalAssignee: Dan Callaghan <dcallagh>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 22CC: dapospis, dcallagh, dowang, mjia, rjoost
Target Milestone: 24.0Keywords: FutureFeature, Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Triaged
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-21 18:50:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1324614    
Attachments:
Description Flags
example screenshot showing Skip result none

Description Scott Poore 2016-04-06 18:56:06 UTC
It would be useful if beaker could support a SKIP result in addition to PASS, FAIL, and WARN.

This would help clearly identify test cases where the case is "skipped" because of outside factors.  Allowing a test to set a result to SKIP would clearly tell the tester or anyone viewing the results that a test didn't just fail or pass but, it was skipped.

Comment 1 matt jia 2016-05-19 05:18:34 UTC
Do the progress bars need to show completed tasks with Skip result? Would be confused to show them as completed tasks with no result?

Comment 2 Dan Callaghan 2016-05-20 21:00:32 UTC
Yes I think we have to include them in the progress bar like all other tasks. I guess they would be coloured the same as New result (plain grey).

Comment 3 Scott Poore 2016-05-23 14:25:10 UTC
I would expect a different color than New just to know that something isn't still expected to run.

Right now the bar uses orange for warning right?  Would yellow for skipped be too close to that?

Or maybe purple?  I think that's what Polarion uses for blocked tests.  I'm not sure how universal that is.

Comment 4 matt jia 2016-05-25 04:39:51 UTC
Currently we show completed tasks with None or New result in the progress bar as plain grey. To use a different colour for skipped tests, I guess we need to add a new column 'stasks' in Job, Recipe Set and Recipe table to record how many tests have been skipped in a job. Not sure which colour is the best one for this. May be white which indicates nothing has been run?

Comment 5 Scott Poore 2016-05-25 16:30:11 UTC
Using white would work, I think.

Comment 6 Dalibor Pospíšil 2016-06-21 15:01:31 UTC
How is the progress here? We have depending bug on beakerlib too.

Comment 7 Dan Callaghan 2016-07-05 02:50:59 UTC
Matt is on PTO so I can take this.

Purple works. We can't use white, it would be indistinguishable from the background of the progress bar (which is what shows incomplete tasks).

Comment 8 Dan Callaghan 2016-07-05 02:52:55 UTC
Should be very similar to the change for adding None result in bug 915128 (which incidentally, does not appear to be properly supported in the web UI right now...)

https://git.beaker-project.org/cgit/beaker/commit/?id=67cf7f9688272d4406a41254857768a239287f0f

Comment 9 Dan Callaghan 2016-07-05 03:07:18 UTC
This is a patch for the Beaker side of things:

http://gerrit.beaker-project.org/5023

Skip will only be visible properly on the new job page (23.0+). The old job page will not show Skipped tasks in the progress bar.

Some more will be needed to make rhts and beah aware of the new result type. I *think* restraint already supports it though.

Comment 10 Dan Callaghan 2016-07-06 06:14:13 UTC
Created attachment 1176737 [details]
example screenshot showing Skip result

Comment 12 Dan Callaghan 2016-07-07 04:27:22 UTC
Plus a Beaker task to exercise the new result type:
http://gerrit.beaker-project.org/5031

and a patch to include it in bkr workflow-selftest:
http://gerrit.beaker-project.org/5032

Comment 13 Scott Poore 2016-07-07 13:16:32 UTC
Dan,

Will that also work at the phase level?

Thanks,
Scott

Comment 14 Dan Callaghan 2016-07-07 22:45:10 UTC
(In reply to Scott Poore from comment #13)
> Will that also work at the phase level?

Not sure what you mean "at the phase level". Beaker doesn't know anything about phases, I guess that's a question for the beakerlib devs?

What this bug gives is the ability to do:

rhts-report-result $TEST SKIP 0

Comment 15 Scott Poore 2016-07-07 23:37:13 UTC
Ok.  I think this is what I'm looking for.   I'll see how the beakerlib bug/rfe comes along for this.

Thanks.

Comment 19 Dan Callaghan 2017-02-21 18:50:33 UTC
Beaker 24.0 has been released.