Description of problem: Swift related tempest tests are failing in RHOSP 17 integration line:- ~~~ {1} tempest.api.object_storage.test_account_services.AccountTest.test_list_no_account_metadata [0.084079s] ... FAILED {1} tempest.api.object_storage.test_account_services.AccountTest.test_update_account_metadata_with_create_and_delete_metadata [0.345026s] ... FAILED {1} tearDownClass (tempest.api.object_storage.test_object_formpost.ObjectFormPostTest) [0.000000s] ... FAILED {1} tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way [0.588196s] ... FAILED {1} tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container [0.537852s] ... FAILED {1} tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself [0.380567s] ... FAILED {1} tempest.api.object_storage.test_object_services.ObjectTest.test_update_object_metadata [0.119344s] ... FAILED {1} tempest.api.object_storage.test_object_services.ObjectTest.test_update_object_metadata_with_x_object_manifest [1.322319s] ... FAILED {0} tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time [0.175609s] ... FAILED ~~~ Most tests are failing with below error:- ~~~ {2} tempest.api.object_storage.test_account_bulk.BulkTest.test_extract_archive [0.389800s] ... FAILED Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tempest/api/object_storage/test_account_bulk.py", line 34, in tearDown self.object_client) File "/usr/lib/python3.6/site-packages/tempest/api/object_storage/base.py", line 50, in delete_containers object_client.delete_object(cont, obj['name']) File "/usr/lib/python3.6/site-packages/tempest/lib/services/object_storage/object_client.py", line 60, in delete_object resp, body = self.delete(url, headers={}) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 703, in request self._error_checker(resp, resp_body) File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 824, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: b'<html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>' ~~~ No log file found under /var/log/containers/swift/ ~~ controller-2 swift]# pwd /var/log/containers/swift controller-2 swift]# ls -al total 0 drwxr-x---. 2 root root 6 Sep 22 04:51 . drwxr-x---. 19 root root 255 Sep 22 04:51 .. ~~~ Journal log have the following error:- ~~~ 2021-09-21T23:24:52+0000 overcloud-controller-0 container-server[100271]: ERROR __call__ error with PUT /d1/217/AUTH_c4167395a2f341b88695f0cd5e25f7b1/tempest-TestContainerSync-182384912 : #012Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/swift/container/server.py", line 893, in __call__#012 res = getattr(self, req.method)(req)#012 File "/usr/lib/python3.6/site-packages/swift/common/utils.py", line 2045, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python3.6/site-packages/swift/container/server.py", line 507, in PUT#012 self.realms_conf)#012 File "/usr/lib/python3.6/site-packages/swift/common/utils.py", line 3634, in validate_sync_to#012 p = urlparse(value)#012 File "/usr/lib64/python3.6/urllib/parse.py", line 369, in urlparse#012 splitresult = urlsplit(url, scheme, allow_fragments)#012 File "/usr/lib64/python3.6/urllib/parse.py", line 461, in urlsplit#012 raise ValueError("Invalid IPv6 URL")#012ValueError: Invalid IPv6 URL (txn: tx99f47b2f4eee4e64936f7-00614a69c3 ~~~ Version-Release number of selected component (if applicable): 17 How reproducible: Everytime Steps to Reproduce: 1. Deploy 17 with ipv6 2. Run the above tempest tests Actual results: Tempest tests are failing Expected results: Tempest tests should pass Additional info: Attaching logs in private.
I think the 202 return codes reported in tempest logs should be OK, given that the tempest suit is executed with low memory footprint, thus having all OS WSGI workers set to processes=1. IIUC, that makes the API queuing requests expected: "02 Accepted response status code indicates that the request has been accepted for processing, but the processing has not been completed; in fact, processing may not have started yet"
Looks to me like this is an error in Tempest when using IPv6. Let's check the headers sent by Tempest in https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-17/c8dd5c8/logs/undercloud/var/log/tempest/testrepository.subunit.gz 2021-09-21 19:24:52,372 413895 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Container-Sync-Key': 'sync_key', 'X-Container-Sync-To': 'https://127.0.0.1:db8:fd00:1000::5]:13808/v1/AUTH_3511b6067581482181d91157d0282f57/tempest-TestContainerSync-1122109270', 'X-Auth-Token': '<omitted>'} Note the X-Container-Sync-To - that URL is wrong. If one uses that URL and does an urlparse in Python it will raise the error seen in this BZ: >>> urlparse("https://127.0.0.1:db8:fd00:1000::5]:13808/v1/AUTH_3511b6067581482181d91157d0282f57/tempest-TestContainerSync-1122109270") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.6/urllib/parse.py", line 369, in urlparse splitresult = urlsplit(url, scheme, allow_fragments) File "/usr/lib64/python3.6/urllib/parse.py", line 461, in urlsplit raise ValueError("Invalid IPv6 URL") ValueError: Invalid IPv6 URL The 127.0.0.1 is wrong; the original IPv6 address is replaced within Tempest with the hardcoded IPv4. https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_container_sync.py#L60 https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_container_sync.py#L140-L141