Bug 447769 - HTS does not allow manual scheduling of device-independant tests
HTS does not allow manual scheduling of device-independant tests
Status: CLOSED CURRENTRELEASE
Product: Red Hat Hardware Certification Program
Classification: Red Hat
Component: Test Suite (harness) (Show other bugs)
5
All Linux
low Severity low
: ---
: ---
Assigned To: Greg Nichols
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-21 14:09 EDT by Greg Nichols
Modified: 2009-02-24 13:09 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-02-24 13:09:55 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
hts/hardwaretest.py patch to change plan --add command (3.38 KB, patch)
2008-08-22 14:51 EDT, Greg Nichols
no flags Details | Diff
hts/test.py patch to change Test.getDevice to return None if no device (340 bytes, patch)
2008-08-22 14:52 EDT, Greg Nichols
no flags Details | Diff

  None (edit)
Description Greg Nichols 2008-05-21 14:09:47 EDT
Description of problem:

HTS does not allow manual scheduling of device-independant tests, such as cpu
scaling, giving an error that --device or --udi options are required.

Workaround:

Supply a bogus device name via --device

Version-Release number of selected component (if applicable):

HTS 5.2 R13
Comment 1 Greg Nichols 2008-08-22 14:50:10 EDT
The following patch removes code that requires either --udi or --device to add a test, and replaces it with a warning and a prompt to confirm.
Comment 2 Greg Nichols 2008-08-22 14:51:25 EDT
Created attachment 314830 [details]
hts/hardwaretest.py patch to change plan --add command
Comment 3 Greg Nichols 2008-08-22 14:52:38 EDT
Created attachment 314831 [details]
hts/test.py patch to change Test.getDevice to return None if no device
Comment 4 Rob Landry 2008-08-26 10:12:40 EDT
It looks like this will allow more broken tests planning than it fixes cleanly planning for tests which don't require a device; unless I've missed where the test can say that it does or doesn't require a device?
Comment 5 Greg Nichols 2008-08-26 22:10:10 EDT
It is a manual override, and will allow tests to be scheduled that will fail for lack of a device.

An alternative would be to implement a mechanism where tests could specify whether or not they require a device, so that the "hts plan -add" code could refuse to add a test if a device was required, and not supplied.  Note that even with this new mechanism, the user could supply an inappropriate device.

For example: 

    hts plan --add --test storage --device eth0

In summary, "hts plan --add" is a manual override provided to work around 
automatic test planning failures.  There's no way for HTS 
prevent user error when supplying a device (or not supplying one),
since it's already failed.
Comment 6 Rob Landry 2008-08-27 17:50:07 EDT
We're probably best off implementing the feature where tests would require some # of params.  Using the network test as an example, --device won't quite cut it you also need a server.  Lots of effort might go into that depending on the number and types of params (ATM I believe there's only device and storage); an alternate way to look at this might be to consider if it is ever proper for a test not to require a device?
Comment 7 Greg Nichols 2008-08-27 21:22:49 EDT
  The network test already checks to see that --server is set to a system running the HTS server.

  Current code requires a device be supplied (either --device or --udi).   The 
validity of the device is not checked, since implementing such a check would make the command useless as a work-around.   For example, if HTS fails to schedule a test on a storage device, then later checks during the manual addition of a storage test would fail as well.    

  The current code does check if the supplied device is found, but only Warns if it can't find the device, and makes no attempt to see if the device is valid for the test being added.

- With this patch, if the user adds a test that requires a device, and fails to supply one (again, via --device or --udi), the test will fail, leaving the user no worse of than before, since manual addition of tests is a work-around for a different failure.

- the only real benefit to the inclusion of this patch is to fix the situation that has arisen where we've had to recommend users add bogus devices to get a test to manually scheduled - most recently cases were bugs where the usb test was not scheduled.   If this benefit is seen as incomplete, I'd recommend we simply not address this bug (WILL NOT FIX).  We'd be better served by improvements to HTS test planning so this kind of work-around isn't needed.
Comment 8 Rob Landry 2008-08-28 14:54:24 EDT
ok, for now we can take this as close enough, we are slightly better off (in not needing to provide bugus hardware options) but that better creates new issues (eg. more paramete validation the suite should be doing) which we can consider adding as later feature.

Note You need to log in before you can comment on or make changes to this bug.