Bug 705844

Summary: interactive tests run in --mode auto should fail
Product: [Retired] Red Hat Hardware Certification Program Reporter: Greg Nichols <gnichols>
Component: Test Suite (harness)Assignee: Greg Nichols <gnichols>
Status: CLOSED ERRATA QA Contact: Guangze Bai <gbai>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.3CC: czhang, gbai, rlandry, yshao
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
An enhancement has been made in v7 1.4 that when trying to run interactive tests with --mode auto option, the test will fail, instead of showing "AUTO" or being logged as "PASS".
 Story Points: ---
Clone Of:
: 747990 (view as bug list) Environment:
Last Closed: 2011-11-08 15:44:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 747990    
Attachments:
Description Flags
patch to fail interactive tests run in auto mode none

Description Greg Nichols 2011-05-18 17:09:14 UTC 
Description of problem:

Tests that require interaction currently are logged as passing, then marked as failing when the results rpm is built.   Prior to building the results rpm, "v7 print" will show the test result as AUTO.

The intent of --mode auto in v7 1.3 is to run tests in automated fashion.   Tests fall into three catalogies, and should be treated accordingly:

1) interactive tests (example: cdrom, audio)
These tests can not be run without user-interaction.   These tests should be marked as failures if run in --mode auto

2) tests that have user prompts for usability/effectiveness, but can be run with out user intervention. (example: reboot)

3) non-interactive tests (info, core, etc.)
--mode auto has no effect.

Category 1) test code should be changed to fail the tests.  Also, code in v7 to make interactive tests as failures during results submission (rpm or otherwise) should be removed.

Version-Release number of selected component (if applicable):

v7 1.3 R45
Comment 1 Greg Nichols 2011-07-11 17:18:21 UTC 
Created attachment 512264 [details]
patch to fail interactive tests run in auto mode
Comment 4 Caspar Zhang 2011-09-02 10:13:59 UTC 
(In reply to comment #0)

>  Also, code in v7 to
> make interactive tests as failures during results submission (rpm or otherwise)
> should be removed.

This code seems not be removed yet in latest version?
Comment 6 Caspar Zhang 2011-10-21 14:48:06 UTC 
(In reply to comment #4)
> (In reply to comment #0)
> 
> >  Also, code in v7 to
> > make interactive tests as failures during results submission (rpm or otherwise)
> > should be removed.
> 
> This code seems not be removed yet in latest version?

The following code will never be reached:

in reporttohtml.py:

159                    if result == Constants.PASS and run.getMode() == Constants.auto and test.isInteractive():                            
160                        result = "auto"  

in report.py:

117                     if test.isInteractive() and run.getMode() == Constants.auto and run.getSummary() == Constants.PASS:                 
118                         summary = "- AUTO"

Code still exist.

Shall we fix it in 1.4 or defer the code cleaning to 1.5?
Comment 7 Caspar Zhang 2011-10-21 14:48:06 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
An enhancement has been made in v7 1.4 that when trying to run interactive tests with --mode auto option, the test will fail, instead of showing "AUTO" or being logged as "PASS".
Comment 8 Greg Nichols 2011-10-21 15:10:17 UTC 
(In reply to comment #6)

> 
> Shall we fix it in 1.4 or defer the code cleaning to 1.5?

defer.
Comment 9 Caspar Zhang 2011-10-21 16:11:39 UTC 
(In reply to comment #8)
> (In reply to comment #6)
> 
> > 
> > Shall we fix it in 1.4 or defer the code cleaning to 1.5?
> 
> defer.

Bug 747990 reported for tracking code clean up process in 1.5.
Comment 10 errata-xmlrpc 2011-11-08 15:44:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1436.html
Comment 11 errata-xmlrpc 2011-11-08 18:30:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1436.html