Description of problem:
python2-tempest-18.0.0-2.el7ost.noarch contains two tests with an identical
source code error (ie `TypeError`) that cause Tempest to stop executing any
further tests and exit "successfully" (ie return code "0").
Affected tests:
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_unshelve_server[id-77eba8e0-036e-4635-944b-f7a8f3b78dc9,image]
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_shelve_shelved_server[id-443e4f9b-e6bf-4389-b601-3a710f15fddd,negative]
Version-Release number of selected component (if applicable):
python2-tempest-18.0.0-2.el7ost.noarch
How reproducible:
100% if you use the --serial option it will always stop at the same point. If you don't use the --serial option, then the amount of tests completed before exit varies.
Steps to Reproduce:
To just run the affected tests:
[stack@director tempest]$ rm tempest.log
[stack@director tempest]$ tempest run --regex 'tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_(shelve_unshelve|stop_start)_server'
{0} tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_unshelve_server [341.664794s] ... ok
======
Totals
======
Ran: 1 tests in 407.0000 sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 341.6648 sec.
==============
Worker Balance
==============
- Worker 0 (1 tests) => 0:05:41.664794
[stack@director tempest]$ echo $?
0
[stack@director tempest]$ # huh? Tempest only ran one test and exited saying everything was alright.
[stack@director tempest]$ # does Tempest's log agree with its output?
[stack@director tempest]$ grep -E ' (CRITICAL|ERROR) tempest ' tempest.log
2018-10-18 12:46:27.838 31096 CRITICAL tempest [-] Unhandled error: TypeError: _unshelve_server() takes no arguments (1 given)
2018-10-18 12:46:27.838 31096 ERROR tempest Traceback (most recent call last):
2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
2018-10-18 12:46:27.838 31096 ERROR tempest "__main__", fname, loader, pkg_name)
2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
2018-10-18 12:46:27.838 31096 ERROR tempest exec code in run_globals
2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib/python2.7/site-packages/subunit/run.py", line 149, in <module>
2018-10-18 12:46:27.838 31096 ERROR tempest main()
Actual results:
Tempest exits prematurely, but shows return code of 0, which would indicate to an automated test system that it actually passed all tests.
Expected results:
1. Fix this unhandled error:
Unhandled error: TypeError: _unshelve_server() takes no arguments (1 given)
2. These tests should cause an RC of 1 if they fail
It looks like there are upstream patches for this issue
https://github.com/openstack/tempest/commit/8c9d4367d95bd9595c91402461b132d4b0add1ffhttps://bugs.launchpad.net/tempest/+bug/1741554https://review.openstack.org/558417
Additional info:
sosreport and tempest logs at:
http://collab-shell.usersys.redhat.com/02233140/
Hi Tim,
I'm not able to reproduce the issue. Was your deployment somehow specific? I tried to run the tests using the same version of Tempest on my undercloud and it worked well.
Have you tried to apply that patch on your tempest and rerun?
Once we make sure, that the review fixes the issue, we have no problem to backport it.
The change, which resolves the issue, is part of openstack-tempest-18.0.0-5.el7ost package, which is available in the latest puddles and also in the latest symlink.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:0088
Description of problem: python2-tempest-18.0.0-2.el7ost.noarch contains two tests with an identical source code error (ie `TypeError`) that cause Tempest to stop executing any further tests and exit "successfully" (ie return code "0"). Affected tests: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_unshelve_server[id-77eba8e0-036e-4635-944b-f7a8f3b78dc9,image] tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_shelve_shelved_server[id-443e4f9b-e6bf-4389-b601-3a710f15fddd,negative] Version-Release number of selected component (if applicable): python2-tempest-18.0.0-2.el7ost.noarch How reproducible: 100% if you use the --serial option it will always stop at the same point. If you don't use the --serial option, then the amount of tests completed before exit varies. Steps to Reproduce: To just run the affected tests: [stack@director tempest]$ rm tempest.log [stack@director tempest]$ tempest run --regex 'tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_(shelve_unshelve|stop_start)_server' {0} tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_unshelve_server [341.664794s] ... ok ====== Totals ====== Ran: 1 tests in 407.0000 sec. - Passed: 1 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 341.6648 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:05:41.664794 [stack@director tempest]$ echo $? 0 [stack@director tempest]$ # huh? Tempest only ran one test and exited saying everything was alright. [stack@director tempest]$ # does Tempest's log agree with its output? [stack@director tempest]$ grep -E ' (CRITICAL|ERROR) tempest ' tempest.log 2018-10-18 12:46:27.838 31096 CRITICAL tempest [-] Unhandled error: TypeError: _unshelve_server() takes no arguments (1 given) 2018-10-18 12:46:27.838 31096 ERROR tempest Traceback (most recent call last): 2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main 2018-10-18 12:46:27.838 31096 ERROR tempest "__main__", fname, loader, pkg_name) 2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code 2018-10-18 12:46:27.838 31096 ERROR tempest exec code in run_globals 2018-10-18 12:46:27.838 31096 ERROR tempest File "/usr/lib/python2.7/site-packages/subunit/run.py", line 149, in <module> 2018-10-18 12:46:27.838 31096 ERROR tempest main() Actual results: Tempest exits prematurely, but shows return code of 0, which would indicate to an automated test system that it actually passed all tests. Expected results: 1. Fix this unhandled error: Unhandled error: TypeError: _unshelve_server() takes no arguments (1 given) 2. These tests should cause an RC of 1 if they fail It looks like there are upstream patches for this issue https://github.com/openstack/tempest/commit/8c9d4367d95bd9595c91402461b132d4b0add1ff https://bugs.launchpad.net/tempest/+bug/1741554 https://review.openstack.org/558417 Additional info: sosreport and tempest logs at: http://collab-shell.usersys.redhat.com/02233140/