Bug 1324614
| Summary: | Add support to beakerlib for SKIP value for results | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Scott Poore <spoore> |
| Component: | beakerlib | Assignee: | Dalibor Pospíšil <dapospis> |
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | low | ||
| Version: | 31 | CC: | azelinka, dapospis, fsumsal, mkyral, ohudlick, spoore, xiawu |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-24 19:51:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1324607 | ||
| Bug Blocks: | 508650 | ||
I think this is duplicate of conditional phases. As a workaround I have made library(distribution/ConditionalPhases) which is able to fulfill it partially. In every case we are limited by BASH so we need to have test code prepared with blocks so they can be actually skipped. *** This bug has been marked as a duplicate of bug 508650 *** I think the ConditionalPhases are useful but, as I was able to use them, they do not quite provide exactly what I'm looking for. It looks like they completely skip the tests altogether. I am looking for a way to not run the tests yet in the reports/results (especially any job xml logs) show that the tests were skipped. Is there already a way to do that? I have noted you requested this feature on beaker side, bz1324607. I think once this is available in beaker I can change the library accordingly to actually create the phase and report it as skipped. The downside is that creation of phase is slow process and if there is a lot of phases which are meant to be skipped, the overhead would be significant. I've read the bug once again and I think there is still possibility I did not understand what you mean precisely. Can you give me more examples how and under what circumstances would you use the SKIP feature? I do not completely understand how would you define when it should be skipped and when properly processed. I suppose you would not use 'rlPhaseStart SKIP "test description that may be skipped"' statically as it could be removed right away. To determine if a phase should be skipped, we could run any number of checks/functions. This could include something like a function to confirm if an install completed properly in a previous phase. Or, check in some way if a bug is present. If conditions aren't as expected by that function, return non-zero or set a variable to indicate a skip.
Then if the phase is configured in some way to be skipped if that non-zero return code (or variable) is seen, do not run the phase code. This is very much like the ConditionalPhases with the Black List. The difference is, I also want to see in the reports (journal.xml) from the job that it was indeed skipped.
I'll use the ConditionalPhases as an example:
check_installer
if [ $? -ne 0 ]; then
export CONDITIONAL_PHASES_BL="tc001 run test"
ConditionalPhasesLibraryLoaded
fi
rlPhaseStartTest "tc001 run test" && {
rlRun "echo test now"
rlPhaseEnd; }
But, with this I see:
<message severity="INFO">replacing rlPhaseStart by modified function with conditional phases implemented</message>
<message severity="WARNING">phase 'tc001 run test' should be skipped as it is defined in $CONDITIONAL_PHASES_BL='tc001 run test'</message>
<message severity="INFO">phase 'ipa-test-template cleanup' will be executed as 'setup' and 'cleanup' phases are allowed by default, these can be black-listed</message>
What I would like to see is something like this:
<phase endtime="2016-04-12 13:17:00 CDT" name="tc001 run test" result="SKIP" score="0" starttime="2016-04-12 13:17:00 CDT" type="FAIL">
<...same message lines as above.../>
</phase>
We want to show skipped tests so it's obvious to anyone looking that a case is a part of the test but it was skipped. Why could be left to drill down. Maybe a message added like an rlLog to the phase? I'm not sure what the best way might be.
Thanks,
Scott
Dalibor, Did comment #5 answer your questions? Thanks, Scott I think yes.
As I'm thinking about it I would say we can use this bug as a tool for conditional phases instead.
Still there is undecided how to handle situation where the phase should be skipped but author of the test did not use block brackets so the phase code was actually executed. It that case we should probably report rather FAIL than SKIP as it may break thinks up.
So the actual code should look like
[[ USER_DEFINED_CONDITION ]] && rlSkipNextPhase; # rise a flag for skipping next phase
rlPhaseStartTest && { # rlPhaseStart* would log information about requested skipping and return 1 if the phase should be skipped
}; rlPhaseEnd; # try to guess if the phase was actually skipped or not (probably based on logged messages) and return desired status to for whole phase
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle. Changing version to '25'. Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle. Changing version to '28'. This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle. Changing version to '31'. This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle. Changing version to 31. This message is a reminder that Fedora 31 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '31'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 31 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |
It would be useful if beakerlib supported returning a SKIP result in addition to PASS, FAIL, and WARN. At least one area this would help is in Phases. For testers that define test cases per Phase, supporting SKIP would help indicate that a test was intentionally NOT run. This is just an example but, not necessarily what I'm looking for: rlPhaseStart SKIP "test description that may be skipped" some_check_function if [ $? -eq 3 ]; then return 3 else rlRun "test" fi rlPhaseEnd I am looking for a way to have phases show as skipped instead of just PASS, FAIL, or WARN. If useful at other levels of beakerlib that would be great too. I suspect rlReport should also support a result of SKIP as well.