Description of problem: v7's command set and options need revision to address usability issues, including the following: - OS/v7 configuration problems should be detected earlier - v7 must handle tests beyond the scope of required hardware certification tests - users need better control for unattended test runs - attended test runs could benefit from increased interative prompting. Version-Release number of selected component (if applicable): v7 1.2
Proposed commands/options changes compared to v7 1.2: general options: --mode normal | auto The --mode option is clarified to mean that v7 should run unattended, to the extent possible, for which ever command is running. It has no direct connotations about running interactve vs. non-interactive tests. The default is "normal". command: plan ---------------------------------------------------------------------- option: --tags | all The --tests options effects which collection of tests are scheduled (depending on detected hardware). Tests will have tags associated with them, initially to distinguish between tests required for certification (tagged, "certification"), and other tests intended for RHEL QA efforts (tagged "RHEL"). Other tags may be added. v7 must warn the user that if "certification" tag is not included, the results will not be valid for certification, and the submit command should mark results accordingly. The default is to include only the "certification" tag. command: verify ---------------------------------------------------------------------- Verify is a new command that runs configuration verification code for every planned test. For example, the network test will verify that the test server is configured and reachable, and the info test will check that selinux is set to enforcing. In the default --mode normal, tests and/or the harness can include code to prompt the user to correct these issues. command: certify ---------------------------------------------------------------------- The certify command will be changed to include the verification step in --mode normal. Also, test tags may be used to filter tests selected for the run. The tags "interactive" and "non-interactive" may be used to control whether or not interactive tests are run. By default, certify will run all tests in the plan that have not yet passed. The --tags option can filter the tests to be run further. command: run ----------------------------------------------------------------------- The new --tags option can be use to manually select all tests with the tag(s).
Overall I like the above, one question... The way I read the above, tests can have multiple tags but some tags are may be mutually exclusive? The current complete list of tags is: interactive _or_ noninteractive RHELQA _or_ certification Some infrastructure will need to know which tags are mutually exclusive and the --tags command would be positive only, meaning include all tests with interactive and RHELQA tags vs. some form of positive and netgative, include all tests with the interactive and without the certification tags?
The tests will be manually tagged, so situations like multiple confilicting tags won't happen on that end. We just wont't tag a test with two mutually-exclusive tags. A command line like: v7 certify --tags interactive,noninteractive,RHELQA,certification is perfectly fine. Or am I missing something?
Nope, that sounds fine.
Created attachment 468453 [details] Patch implementing the verify command and test tags This patch revises the v7 command set to add a "verify" step. The command "v7 verify" will run verification code on each test intended to verify important configuration settings before running of the full test. For example, this patch checks SELinux. The patch also adds test tagging, so tests may be chosen by various criteria, such as "interactive" vs. "non-interactive", "realtime", etc. The "--mode" option is redefined to indicate whether testing is run in an automated environment ("auto"), or a normal, attended environment ("normal", the default).
If the user supplies: > v7 certify --mode auto Should the tests selected by limited to tests tagged "non-interactive", or should the only effect of supplying --mode auto be skipping the new verify step?
I think it wouldn't do either of those? Comment #1 states that mode does not impact interactive vs. non-interactive, nor would it not-run the validation step that is required by certify, again according to the description in comment #1. I think instead each test would need to be altered to have a default value used when mode=auto is selected for any user prompts? This would allow tests like dvd or cdrom to proceed if they are capable of doing so without the user but bail and fail the test if they require the user to complete successfully. It could also skip any tests that fail the validation step; again where any question that arise during validation defaults are provided instead of prompts; for example if $server hasn't been specified validate might prompt the user to provide it, where as in auto mode that prompt would not appear but have a value that fails removing the network test from the list of tests to run, or even get lucky if we set the default to the dhcp server and that system is also a v7 server?
I've run into a bit of a snag with use of tags in the plan command. v7 re-runs planning when performing a run or certify command. This is done to detect hardware changes. If tags are used during with the plan command, then other tags are used for run/certify, the plan command will detect this as a change, and ask the user to either update the plan, or cancel the operation. Options for handling this issue: 0) Require plan update: Essentially leave the behavior described above. This means that you'd have to keep the tags the same if you used them in the plan command. 1) Store the plan tags: v7 could store the tags used to plan the tests (in results.xml). When checking the plan during run/certify, it could use the previous tags to check the plan. 3) Don't use tags when planning v7 would plan tests ignoring tags, so the test plan would show all tests relevant to the hardware, whether or not they are required for certification ("certification" vs. "osqa"). Since the "certification" is default, then unless "osqa" was added, only certification tests would run. I'd recommend 3), and we not use tags in planning.
*** Bug 476303 has been marked as a duplicate of this bug. ***
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: A series of enhancements has been made in v7 1.3 to make command set for better usability and control of testing. Now users are able to use tags to run specified test sets.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0497.html