Description of problem: The tempest test tempest.api.object_storage.test_container_acl_negative.ObjectACLsNegativeTest.test_delete_object_without_using_creds[id-af85af0b-a025-4e72-a90e-121babf55720,negative] failes with HTTP 404 error: testtools.testresult.real._StringException: pythonlogging:'': {{{ 2021-06-11 09:02:19,350 450931 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:setUp): 201 PUT http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625 0.022s 2021-06-11 09:02:19,351 450931 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'content-length': '0', 'x-trans-id': 'tx000000000000000001d3a-0060c3269b-1376-default', 'x-openstack-request-id': 'tx000000000000000001d3a-0060c3269b-1376-default', 'accept-ranges': 'bytes', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 11 Jun 2021 09:02:19 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625'} Body: b'' 2021-06-11 09:02:19,365 450931 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:test_delete_object_without_using_creds): 201 PUT http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625/tempest-Object-1772998575 0.013s 2021-06-11 09:02:19,365 450931 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: data Response - Headers: {'etag': '8d777f385d3dfec8815d20f7496026dc', 'last-modified': 'Fri, 11 Jun 2021 09:02:19 GMT', 'x-trans-id': 'tx000000000000000001d3a-0060c3269b-13a2-default', 'x-openstack-request-id': 'tx000000000000000001d3a-0060c3269b-13a2-default', 'content-type': 'application/json; charset=utf-8', 'content-length': '0', 'date': 'Fri, 11 Jun 2021 09:02:19 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625/tempest-Object-1772998575'} Body: b'' 2021-06-11 09:02:19,376 450931 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:test_delete_object_without_using_creds): 404 DELETE http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625/tempest-Object-1772998575 0.010s 2021-06-11 09:02:19,377 450931 DEBUG [tempest.lib.common.rest_client] Request - Headers: {} Body: None Response - Headers: {'content-length': '12', 'x-trans-id': 'tx000000000000000001d3b-0060c3269b-13a7-default', 'x-openstack-request-id': 'tx000000000000000001d3b-0060c3269b-13a7-default', 'accept-ranges': 'bytes', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 11 Jun 2021 09:02:19 GMT', 'connection': 'close', 'status': '404', 'content-location': 'http://10.0.0.121:8080/swift/v1/AUTH_8ac090c77da34fedab34ab666f6373db/tempest-TestContainer-230605625/tempest-Object-1772998575'} Body: b'NoSuchBucket' }}} Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tempest/api/object_storage/test_container_acl_negative.py", line 80, in test_delete_object_without_using_creds self.container_name, object_name) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 485, in assertRaises self.assertThat(our_callable, matcher) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 496, in assertThat mismatch_error = self._matchHelper(matchee, matcher, message, verbose) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 547, in _matchHelper mismatch = matcher.match(matchee) File "/usr/lib/python3.6/site-packages/testtools/matchers/_exception.py", line 108, in match mismatch = self.exception_matcher.match(exc_info) File "/usr/lib/python3.6/site-packages/testtools/matchers/_higherorder.py", line 62, in match mismatch = matcher.match(matchee) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 475, in match reraise(*matchee) File "/usr/lib/python3.6/site-packages/testtools/_compat3x.py", line 16, in reraise raise exc_obj.with_traceback(exc_tb) File "/usr/lib/python3.6/site-packages/testtools/matchers/_exception.py", line 101, in match result = matchee() File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 1049, in __call__ return self._callable_object(*self._args, **self._kwargs) File "/usr/lib/python3.6/site-packages/tempest/lib/services/object_storage/object_client.py", line 58, in delete_object resp, body = self.delete(url, headers={}) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 331, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 704, in request self._error_checker(resp, resp_body) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 810, in _error_checker raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: b'NoSuchBucket' Version-Release number of selected component (if applicable): ceph-radosgw-14.2.11-181.el8cp.x86_64 python3-rgw-14.2.11-181.el8cp.x86_64 librgw2-14.2.11-181.el8cp.x86_64 How reproducible: unknown Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I don't think the swiftclient would be at fault here, even if it were involved. But it sure looks to me that Tempest uses Openstack SDK instead. I suspect RGW is in the wrong there. The RFC for HTTP that REST services obey has some weird conditions when 404 overtakes other codes. However, it's never a substitute for 401 or 403. So, perhaps RGW ignores the ID in the AUTH_xxxxx, and is looking for the account in the token first. If there's no token, there's no account, and so 404. Something like that. I'm going to re-assign this to Ceph folks to take a quick look. If they think it's all okay on their side, they can bounce it back to tests. Not the swiftclient though.
*** This bug has been marked as a duplicate of bug 1960306 ***