Created attachment 1608665 [details] config file in ~/.config/openstack Description of problem: 'tempest cleanup' fails with an authentication error when you try to run a cleanup after running tempest tests Version-Release number of selected component (if applicable): openstack-tempest 21.0.0 How reproducible: always Steps to Reproduce: 1. create ~/mytempest directory, change to directory 2. set up tempest with discover-tempest-config command (detailed below) 3. run 'tempest cleanup --init-saved-state' to get configuration 4. run various tempest test runs (tempest run ....) 5. after tempest testcases are run, use 'tempest cleanup' from ~/mytempest (where the saved_state.json file generated in #3 is located) Actual results: 'tempest cleanup' fails with apparent authentication errors, exits without cleaning up leftover projects, tasks, etc from tempest runs. --------------------- tempest.lib.exceptions.Unauthorized: Unauthorized Details: The request you have made requires authentication. --------------------- Expected results: 'tempest cleanup' removes residual test run elements (projects, ports, instances, and other tasks) from overcloud, returns state of the system to how it was before the test runs Additional info: command parameters for discover-tempest-config: ---------------- discover-tempest-config --deployer-input ~/tempest-deployer-input.conf --debug --create identity.uri http://192.168.77.46:5000 identity.admin_password <Password_Went_Here> image.http_image http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-ppc64le-disk.img --image cirros-0.4.0-ppc64le-disk --network-id 5d1933c1-bcfa-4551-956f-34e910bade9b ---------------- Error output from 'tempest cleanup' --------------- $ tempest cleanup Begin cleanup Process 2 projects Cleaning project: c7dc6af158de4942b703952b0ff92c3d-c1136efc-5b5a-4acc-9952-72fce6c Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 102, in take_action self._cleanup() File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 159, in _cleanup self._clean_project(project) File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 201, in _clean_project **kwargs)) File "/usr/lib/python3.6/site-packages/tempest/common/credentials_factory.py", line 299, in get_credentials **params) File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 648, in get_credentials creds = auth_provider.fill_credentials() File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 124, in fill_credentials auth_data = self.get_auth() File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 150, in get_auth self.set_auth() File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 159, in set_auth self.cache = self._get_auth() File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 314, in _get_auth token, auth_data = auth_func(**auth_params) File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 189, in get_token body = self.auth(**kwargs) File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 140, in auth resp, body = self.post(self.auth_url, body=body) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 169, in request raise exceptions.Unauthorized(resp_body['error']['message']) tempest.lib.exceptions.Unauthorized: Unauthorized Details: The request you have made requires authentication. verified env var set before running: ---------------------- NOVA_VERSION=1.1 COMPUTE_API_VERSION=1.1 OS_USERNAME=admin OS_PROJECT_NAME=admin OS_USER_DOMAIN_NAME=Default OS_PROJECT_DOMAIN_NAME=Default OS_NO_CACHE=True OS_CLOUDNAME=overcloud no_proxy=,192.168.77.46,192.168.77.46 PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available' OS_AUTH_TYPE=password OS_AUTH_URL=http://192.168.77.46:5000 OS_IDENTITY_API_VERSION=3 OS_IMAGE_API_VERSION=2 OS_VOLUME_API_VERSION=3 OS_REGION_NAME=regionOne (OS_PASSWORD was also set)
Created attachment 1608666 [details] stester.conf file
Created attachment 1608667 [details] tempest.conf generated by discover-tempest-config
Hi James, did the tests you ran pass? Just to confirm that the credentials are correct. I've seen this issue multiple times over the last ~2 years (I mean I saw bug reports, I've never hit the issue myself), for example there's an upstream bug for this but it hasn't been reproduced yet: https://bugs.launchpad.net/tempest/+bug/1766582 Due to the fact it hasn't been reproduced yet we could assume it's a very flaky issue probably related to a specific project or a service. The deletion of c7dc6af158de4942b703952b0ff92c3d-c1136efc-5b5a-4acc-9952-72fce6c project failed - do you know what test created this project? Or which plugin? What plugins did you have installed and what kind of tests you were running? For better chances in reproducing the issue we need to identify the services which create resources which are failing to be removed. Regards, Martin
I ran into exact issue this with OSP 13. Tempest version tempest-18.0.0-10 It occurs for me when I run the heat plugin tests.
We hit this issue with heat.
Info how to reproduce the issue is described in the upstream bug: https://bugs.launchpad.net/tempest/+bug/1766582/comments/1 The exact reason why and a way how to fix this are still being investigated.
Created attachment 1666174 [details] test_fix_bug_1746091.txt Tested with: - package with the bug (openstack-tempest-21.0.0-0.20190726140456.702b21c.el8ost) - package with the fix (openstack-tempest-21.0.0-0.20191209200453.702b21c.el8ost) The bug was not reproduced when tested with the package with the fix as shown in the attached log file (test_fix_bug_1746091.log).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0710