Bug 2111902

Summary: Can not perform PUT against RGW: Unauthorized
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Filip Hubík <fhubik>
Component: RGWAssignee: Marcus Watts <mwatts>
Status: CLOSED ERRATA QA Contact: Alfredo <alfrgarc>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.2CC: afazekas, alfrgarc, aoconnor, cbodley, ceph-eng-bugs, cephqe-warriors, derekh, fpantano, gfidente, johfulto, jparoly, kbader, kdreyer, kkeithle, ltoscano, mbenjamin, mwatts, tserlin, yrabl, zaitcev
Target Milestone: ---Keywords: Regression
Target Release: 5.2-async1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-16.2.8-85.el8cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2116926 (view as bug list) Environment:
Last Closed: 2022-08-31 16:02:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1820257, 2112000, 2116926    
Attachments:
Description Flags
logs from controller during the failure none

Description Filip Hubík 2022-07-28 12:34:20 UTC 
Description of problem:
Unauthorized response is received when executing PUT against the swift endpoint. Could it be possibly an issue between keystone/swift/rgw related to 5.2 container? Example of failing test follows, its very similar amongst many failing Tempest tests:

testtools.testresult.real._StringException: pythonlogging:'': {{{
2022-07-28 10:38:29,380 153118 INFO     [tempest.lib.common.rest_client] Request (ContainerQuotasTest:setUp): 401 PUT https://10.0.0.108:13808/swift/v1/AUTH_3507c05b92c8461c9260010ad7d41264/tempest-TestContainer-1602425073 0.018s
2022-07-28 10:38:29,380 153118 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-length': '12', 'x-trans-id': 'tx0000027a746bc8f8d93f5-0062e26725-37ae-default', 'x-openstack-request-id': 'tx0000027a746bc8f8d93f5-0062e26725-37ae-default', 'accept-ranges': 'bytes', 'content-type': 'text/plain; charset=utf-8', 'date': 'Thu, 28 Jul 2022 10:38:29 GMT', 'connection': 'close', 'status': '401', 'content-location': 'https://10.0.0.108:13808/swift/v1/AUTH_3507c05b92c8461c9260010ad7d41264/tempest-TestContainer-1602425073'}
        Body: b'AccessDenied'
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_container_quotas.py", line 41, in setUp
    self.container_name = self.create_container()
  File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/base.py", line 106, in create_container
    cls.container_client.update_container(container_name)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/container_client.py", line 45, in update_container
    resp, body = self.put(url, body=None, headers=headers)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 363, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 816, in _error_checker
    raise exceptions.Unauthorized(resp_body, resp=resp)
tempest.lib.exceptions.Unauthorized: Unauthorized
Details: b'AccessDenied'

I see the following during the failed test run:
 1360905cbb8d4212919757bb8f04696f | regionOne | swift        | object-store   | True    | public    | https://10.0.0.108:13808/swift/v1/AUTH_%(project_id)s |

(haproxy.cfg)
listen ceph_rgw
  bind 10.0.0.108:13808 transparent ssl crt
  bind 172.17.3.81:8080 transparent...

and others point to rgw container, but only this error in basic logs during the failure:
./containers/keystone/keystone.log... Authorization failed. The request you have made requires authentication. from 172.17.1.58: keystone.exception.Unauthorized: The request you have made requires authentication.

ceph log last "*"
# doesn't seem to show anything useful

Additional info:
rhceph 5-259 container
RHOS-17.0-RHEL-9-20220727.n.0 OSP content
Comment 2 Filip Hubík 2022-07-28 12:44:32 UTC 
Created attachment 1899947 [details]
logs from controller during the failure
Comment 3 Luigi Toscano 2022-07-28 12:54:55 UTC 
This is an RGW issue.
Comment 5 RHEL Program Management 2022-07-28 13:25:24 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 36 Francesco Pantano 2022-08-03 07:10:03 UTC 
*** Bug 2113066 has been marked as a duplicate of this bug. ***
Comment 73 Yogev Rabl 2022-08-15 15:01:46 UTC 
Verified
Comment 75 errata-xmlrpc 2022-08-31 16:02:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.2 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6273