Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1755132

Summary: 'tempest cleanup' reports authorization error when running cleanup after test runs
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-tempestAssignee: Chandan Kumar <chkumar>
Status: CLOSED ERRATA QA Contact: Martin Kopec <mkopec>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: apevec, chkumar, gkadam, kholtz, kurt_hey, lhh, marjones, mkopec, slinaber, udesale
Target Milestone: z11Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: mkopec: needinfo-
mkopec: needinfo-
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tempest-18.0.0-12.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-10 11:52:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1746091    
Bug Blocks: 1581398, 1588541    
Attachments:
Description Flags
verification output none

Description David Hill 2019-09-24 20:31:16 UTC
Description of problem:
'tempest cleanup' reports authorization error when running cleanup after test runs

What problem/issue/behavior are you having trouble with?  What do you expect to see?

'tempest cleanup' reports authorization error when running cleanup after test runs. Only noticed to occur when running tests using the heat plugin.

(engcloud) [stack@director tempest]$ tempest cleanup
Begin cleanup
Process 18 projects
Cleaning project:  d5da6cf0b2d8442586875880559d4481-fc5be417-89c4-471e-bff4-edddd33
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 112, in take_action
    self._cleanup()
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 175, in _cleanup
    self._clean_project(project)
  File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 217, in _clean_project
    **kwargs))
  File "/usr/lib/python2.7/site-packages/tempest/common/credentials_factory.py", line 298, in get_credentials
    **params)
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 648, in get_credentials
    creds = auth_provider.fill_credentials()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 124, in fill_credentials
    auth_data = self.get_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 150, in get_auth
    self.set_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 159, in set_auth
    self.cache = self._get_auth()
  File "/usr/lib/python2.7/site-packages/tempest/lib/auth.py", line 314, in _get_auth
    token, auth_data = auth_func(**auth_params)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 181, in get_token
    body = self.auth(**kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 132, in auth
    resp, body = self.post(self.auth_url, body=body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 279, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/identity/v3/token_client.py", line 161, in request
    raise exceptions.Unauthorized(resp_body['error']['message'])
Unauthorized: Unauthorized
Details: The request you have made requires authentication.

(engcloud) [stack@director tempest]$ openstack project list --long
+----------------------------------+------------------------------------------------------------------+----------------------------------+-----------------------------------+---------+
| ID                               | Name                                                             | Domain ID                        | Description                       | Enabled |
+----------------------------------+------------------------------------------------------------------+----------------------------------+-----------------------------------+---------+
| 01de83ce02004f4da8ef29a227189ad9 | d5da6cf0b2d8442586875880559d4481-fc5be417-89c4-471e-bff4-edddd33 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 19bc9ab03bd74df0aae536a6b2299207 | d5da6cf0b2d8442586875880559d4481-5041044f-5420-4815-8733-1ddbcf8 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 25edc75d1146485981f9072ef27a6c48 | d5da6cf0b2d8442586875880559d4481-ec74e469-c36c-45fb-9760-48cc970 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 29845792d8654202afe11cc02525ab75 | admin                                                            | default                          | admin tenant                      | True    |
| 2fa4c6d547004df1871efe83e86b9617 | d5da6cf0b2d8442586875880559d4481-93285bd5-080c-46a9-8255-d48ed1a | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 3dfbd8e3141149c3bdbb902e658422b4 | d5da6cf0b2d8442586875880559d4481-6da0b1b1-7629-4c88-a44d-f09eab8 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 3ec266e0162d447c805ee7a3ac2f4bc2 | d5da6cf0b2d8442586875880559d4481-189c0eb5-281d-4d24-bf1b-8491de9 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 65415adbcaac42d1b67620717e6401bd | d5da6cf0b2d8442586875880559d4481-60ba1f8b-5465-4e1b-88e9-886d985 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 672529ea19d14f7582a703c163646803 | service                                                          | default                          | Tenant for the openstack services | True    |
| 77d733d1e0774173b691fb628545466f | project2                                                         | bde7454715144b3d875f6e2b8b9af6ce |                                   | True    |
| 864645bbf2e94b679ec319601297d652 | project1                                                         | bde7454715144b3d875f6e2b8b9af6ce |                                   | True    |
| 8aa289cd6c024708a45a6b10a01e1fff | d5da6cf0b2d8442586875880559d4481-4ae0652d-929b-4d59-8600-da047b3 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| 8d502b85a0e843bf94c638f994a2b903 | alt_demo                                                         | default                          | Project for Tempest alt_demo user | True    |
| 979d0c6b4e9a468e9074f1e24edb91a1 | d5da6cf0b2d8442586875880559d4481-5516a755-30b3-4c81-bcfd-02a3d51 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| a127a899896746c3a82c5dfdad20f157 | d5da6cf0b2d8442586875880559d4481-2c2f0cd5-329b-4fbc-8609-b6238b1 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| abb89fdb5a084defafae9910737440af | d5da6cf0b2d8442586875880559d4481-d98d99af-d6ea-4aa0-b4b8-65da437 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| ad17e1ca2d444da7b9368acf93ec1ead | d5da6cf0b2d8442586875880559d4481-55960d7b-d5af-4dcd-8592-2b864ba | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| be0fad4925b048949586652ffebb510c | d5da6cf0b2d8442586875880559d4481-f9957699-ef1d-45ab-a75d-ede04f3 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| bfd70eec072f41cebc460dbdb1982ee1 | d5da6cf0b2d8442586875880559d4481-8a7ee36d-9522-4cc1-8bb2-a28bf0c | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| c3416b1b9c2d42a9a2ce48a5cd060188 | d5da6cf0b2d8442586875880559d4481-b15903c3-b293-4996-9b24-85a86a6 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| c57483997a8743f2a98d54e99846c24e | d5da6cf0b2d8442586875880559d4481-ac9fca08-816a-4ff5-9673-bb3086d | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| cc80d87c98af4f8bb8e0777c1fa58ca7 | d5da6cf0b2d8442586875880559d4481-c50fa6f6-ceec-46df-8393-18acac6 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
| d5da6cf0b2d8442586875880559d4481 | demo                                                             | default                          | Project for Tempest demo user     | True    |
| eb8770dc05d044859d21b5f6cca43066 | d5da6cf0b2d8442586875880559d4481-64b07b14-ffdd-4219-b2aa-f49f563 | 22cbc8fbc95842f389af5f665d4c9898 | Heat stack user project           | True    |
+----------------------------------+------------------------------------------------------------------+----------------------------------+-----------------------------------+---------+

Where are you experiencing the behavior?  What environment?

OSP 13.0.8.
Tempest installed on the director and the tests are executed on the overcloud.

(engcloud) [stack@director-eng eng-tempest]$ rpm -qa | grep tempest
python2-neutron-tests-tempest-0.0.1-0.20180425142847.02a5e2b.el7ost.noarch
python2-keystone-tests-tempest-0.0.1-1.0.564670bgit.el7ost.noarch
python2-tempest-18.0.0-10.el7ost.noarch
python2-heat-tests-tempest-0.3.0-1.el7ost.noarch
python2-novajoin-tests-tempest-0.0.1-0.20180219103837.59d7ba6.el7ost.noarch
python2-cinder-tests-tempest-0.0.1-1.0.aafbdf8git.el7ost.noarch
python2-horizon-tests-tempest-0.0.1-0.20180219094157.a23f407.el7ost.noarch
openstack-tempest-18.0.0-10.el7ost.noarch
python2-tempestconf-2.2.0-1.el7ost.noarch

When does the behavior occur? Frequently?  Repeatedly?   At certain times?
Repeatedly

Comment 5 Martin Kopec 2019-09-30 21:10:13 UTC
*** Bug 1747513 has been marked as a duplicate of this bug. ***

Comment 7 Martin Kopec 2019-10-01 17:29:29 UTC
Info how to reproduce the issue is described in the upstream bug: 
https://bugs.launchpad.net/tempest/+bug/1766582/comments/1

The exact reason why and a way how to fix this are still being investigated.

Comment 15 Martin Kopec 2020-02-11 09:04:51 UTC
Created attachment 1662407 [details]
verification output

The 'Fixed in version' contains the fix for the issue, verified on a 13 env using 2020-02-06.2 puddle. Verification output attached.

Comment 17 errata-xmlrpc 2020-03-10 11:52:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0769