Bug 1746091

Summary: 'tempest cleanup' reports authorization error when running cleanup after test runs
Product: Red Hat OpenStack Reporter: James E. LaBarre <jlabarre>
Component: openstack-tempestAssignee: Lukas Piwowarski <lpiwowar>
Status: CLOSED ERRATA QA Contact: Lukas Piwowarski <lpiwowar>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: apevec, dhill, harsh.kotak, lhh, slinaber, udesale
Target Milestone: z2Keywords: Triaged, ZStream
Target Release: 15.0 (Stein)Flags: mkopec: needinfo-
Hardware: ppc64le   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tempest-21.0.0-0.20191209200453.702b21c.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-05 12:02:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1755132    
Attachments:
Description Flags
config file in ~/.config/openstack
none
stester.conf file
none
tempest.conf generated by discover-tempest-config
none
test_fix_bug_1746091.txt none

Description James E. LaBarre 2019-08-27 16:02:55 UTC
Created attachment 1608665 [details]
config file in ~/.config/openstack

Description of problem:
'tempest cleanup' fails with an authentication error when you try to run a cleanup after running tempest tests

Version-Release number of selected component (if applicable):
openstack-tempest 21.0.0

How reproducible:
always

Steps to Reproduce:
1. create ~/mytempest directory, change to directory
2. set up tempest with discover-tempest-config command (detailed below)
3. run 'tempest cleanup --init-saved-state' to get configuration
4. run various tempest test runs (tempest run ....)
5. after tempest testcases are run, use 'tempest cleanup' from ~/mytempest (where the saved_state.json file generated in #3 is located)

Actual results:
'tempest cleanup' fails with apparent authentication errors, exits without cleaning up leftover projects, tasks, etc from tempest runs.
---------------------
tempest.lib.exceptions.Unauthorized: Unauthorized
Details: The request you have made requires authentication.
---------------------

Expected results:
'tempest cleanup' removes residual test run elements (projects, ports, instances, and other tasks) from overcloud, returns state of the system to how it was before the test runs


Additional info:


command parameters for discover-tempest-config:
----------------
discover-tempest-config --deployer-input ~/tempest-deployer-input.conf --debug --create identity.uri http://192.168.77.46:5000 identity.admin_password <Password_Went_Here> image.http_image http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-ppc64le-disk.img --image cirros-0.4.0-ppc64le-disk --network-id 5d1933c1-bcfa-4551-956f-34e910bade9b 
----------------


Error output from 'tempest cleanup'
---------------
$ tempest cleanup
Begin cleanup
Process 2 projects
Cleaning project: c7dc6af158de4942b703952b0ff92c3d-c1136efc-5b5a-4acc-9952-72fce6c 
Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 102, in take_action
 self._cleanup()
 File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 159, in _cleanup
 self._clean_project(project)
 File "/usr/lib/python3.6/site-packages/tempest/cmd/cleanup.py", line 201, in _clean_project
 **kwargs))
 File "/usr/lib/python3.6/site-packages/tempest/common/credentials_factory.py", line 299, in get_credentials
 **params)
 File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 648, in get_credentials
 creds = auth_provider.fill_credentials()
 File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 124, in fill_credentials
 auth_data = self.get_auth()
 File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 150, in get_auth
 self.set_auth()
 File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 159, in set_auth
 self.cache = self._get_auth()
 File "/usr/lib/python3.6/site-packages/tempest/lib/auth.py", line 314, in _get_auth
 token, auth_data = auth_func(**auth_params)
 File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 189, in get_token
 body = self.auth(**kwargs)
 File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 140, in auth
 resp, body = self.post(self.auth_url, body=body)
 File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post
 return self.request('POST', url, extra_headers, headers, body, chunked)
 File "/usr/lib/python3.6/site-packages/tempest/lib/services/identity/v3/token_client.py", line 169, in request
 raise exceptions.Unauthorized(resp_body['error']['message'])
tempest.lib.exceptions.Unauthorized: Unauthorized
Details: The request you have made requires authentication.


verified env var set before running:
----------------------
NOVA_VERSION=1.1
COMPUTE_API_VERSION=1.1
OS_USERNAME=admin
OS_PROJECT_NAME=admin
OS_USER_DOMAIN_NAME=Default
OS_PROJECT_DOMAIN_NAME=Default
OS_NO_CACHE=True
OS_CLOUDNAME=overcloud
no_proxy=,192.168.77.46,192.168.77.46
PYTHONWARNINGS='ignore:Certificate has no, ignore:A true SSLContext object is not available'
OS_AUTH_TYPE=password
OS_AUTH_URL=http://192.168.77.46:5000
OS_IDENTITY_API_VERSION=3
OS_IMAGE_API_VERSION=2
OS_VOLUME_API_VERSION=3
OS_REGION_NAME=regionOne

(OS_PASSWORD was also set)

Comment 1 James E. LaBarre 2019-08-27 16:05:55 UTC
Created attachment 1608666 [details]
stester.conf file

Comment 2 James E. LaBarre 2019-08-27 16:09:39 UTC
Created attachment 1608667 [details]
tempest.conf generated by discover-tempest-config

Comment 4 Martin Kopec 2019-09-05 14:17:54 UTC
Hi James,

did the tests you ran pass? Just to confirm that the credentials are correct.

I've seen this issue multiple times over the last ~2 years (I mean I saw bug reports, I've never hit the issue myself), for example there's an upstream bug for this but it hasn't been reproduced yet:
https://bugs.launchpad.net/tempest/+bug/1766582
Due to the fact it hasn't been reproduced yet we could assume it's a very flaky issue probably related to a specific project or a service. 
The deletion of c7dc6af158de4942b703952b0ff92c3d-c1136efc-5b5a-4acc-9952-72fce6c project failed - do you know what test created this project? Or which plugin? What plugins did you have installed and what kind of tests you were running? 
For better chances in reproducing the issue we need to identify the services which create resources which are failing to be removed.

Regards,
Martin

Comment 5 Harsh 2019-09-13 17:32:34 UTC
I ran into exact issue this with OSP 13.

Tempest version tempest-18.0.0-10

It occurs for me when I run the heat plugin tests.

Comment 6 David Hill 2019-10-01 17:19:54 UTC
We hit this issue with heat.

Comment 7 Martin Kopec 2019-10-01 17:28:43 UTC
Info how to reproduce the issue is described in the upstream bug: 
https://bugs.launchpad.net/tempest/+bug/1766582/comments/1

The exact reason why and a way how to fix this are still being investigated.

Comment 12 Lukas Piwowarski 2020-02-27 10:57:49 UTC
Created attachment 1666174 [details]
test_fix_bug_1746091.txt

Tested with:
  - package with the bug (openstack-tempest-21.0.0-0.20190726140456.702b21c.el8ost)
  - package with the fix (openstack-tempest-21.0.0-0.20191209200453.702b21c.el8ost)

The bug was not reproduced when tested with the package with the fix as shown in the attached log file (test_fix_bug_1746091.log).

Comment 14 errata-xmlrpc 2020-03-05 12:02:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0710