Tempest is the upstream test tool for OpenStack. RGW should pass all the tests for its Swift API tests.
*** This bug has been marked as a duplicate of bug 1250713 ***
Reopening this bug. It is different from bug 1250713. bug 1250713 deals with tempest configuration when RGW is part of openstack solution. It is discovered by Tempest and then tempest configuration fails. So that BZ is a temporary fix to ignore swift and not configure tempest for swift testing if ceph is the object store. This bug is talking about fixing it correctly. That is Ceph swift API respond correctly to tempest configurator calls. This BZ is more closely related to https://bugzilla.redhat.com/show_bug.cgi?id=1246162.
Let's use this ticket to track authoritatively what tests are not passing.
I've taken on Neil's request to update list of non-conformances.
Yehuda->Matt.
The new proposal from Arkady is to reduce the test criteria down to just those needed to pass DefCore RefStack: https://refstack.openstack.org/#/capabilities We should get a list of failures for both this and the full Tempest suite and then perhaps attack this in two phases i) RefStack and ii) full Tempest.
To be precise we want two things: 1. we want all swift tempest test that Ceph can support to pass. 1a. Understand why others do not pass 2. tempest tests of current 2016.1 version of refstack (https://refstack.openstack.org/#/capabilities) are passed with Swift API on Ceph (in our joint solution configuration) These are: tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_across_containers tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself tempest.api.object_storage.test_object_services.ObjectTest.test_create_object tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time tempest.api.object_storage.test_object_services.ObjectTest.test_get_object tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_if_different tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url Longer term, every time refstack releases new version somebody need to review what tests are not part of object storage "core" and ensure they pass. We can file new BZ when that happens. But these release come roughly every quarter. Since we need current refstack object tests to pass we potentially need bug fixes "backported" into the version of Ceph that is part of our joint solution we are releasing.
We think the current builds pass RefCore.
We are targeting this at RHCS 2.1 -
*** Bug 1413164 has been marked as a duplicate of this bug. ***
Tried with ceph-10.2.5-29. config_tempest.py now fails with this error: # python tools/config_tempest.py --debug identity.uri http://10.8.128.113:5000/v2.0 identity.admin_password da426a86fd7b4b4c --create 2017-02-22 09:45:41.179 4469 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf 2017-02-22 09:45:41.411 4469 INFO __main__ [-] Reading defaults from file '/root/tempest/etc/default-overrides.conf' 2017-02-22 09:45:41.412 4469 DEBUG __main__ [-] Setting [identity] uri = http://10.8.128.113:5000/v2.0 set tools/config_tempest.py:461 2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] admin_password = da426a86fd7b4b4c set tools/config_tempest.py:461 2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] uri_v3 = http://10.8.128.113:5000/v3 set tools/config_tempest.py:461 2017-02-22 09:45:41.532 4469 INFO tempest.lib.common.rest_client [req-5cf3fed9-b52b-49a1-a33d-6b941a3b012c ] Request (main): 200 POST http://10.8.128.113:5000/v2.0/tokens 2017-02-22 09:45:41.553 4469 INFO tempest.lib.common.rest_client [req-327791eb-9277-47d8-a108-70cea4942e25 ] Request (main): 200 GET http://10.8.128.113:35357/v2.0/tenants 0.020s 2017-02-22 09:45:41.553 4469 DEBUG __main__ [-] Setting [identity] admin_tenant_id = 0552cdc8375c4596a38faf32c679d746 set tools/config_tempest.py:461 2017-02-22 09:45:44.231 4469 CRITICAL tempest [-] ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404 2017-02-22 09:45:44.231 4469 ERROR tempest Traceback (most recent call last): 2017-02-22 09:45:44.231 4469 ERROR tempest File "tools/config_tempest.py", line 830, in <module> 2017-02-22 09:45:44.231 4469 ERROR tempest main() 2017-02-22 09:45:44.231 4469 ERROR tempest File "tools/config_tempest.py", line 161, in main 2017-02-22 09:45:44.231 4469 ERROR tempest api_version=api_version) 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 164, in discover 2017-02-22 09:45:44.231 4469 ERROR tempest services[name]['extensions'] = service.get_extensions() 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 112, in get_extensions 2017-02-22 09:45:44.231 4469 ERROR tempest top_level_path="info") 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 54, in do_get 2017-02-22 09:45:44.231 4469 ERROR tempest " with code %d" % (self.name, url, r.status)) 2017-02-22 09:45:44.231 4469 ERROR tempest ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404 I am not able to proceed with verifying the fix. Need this https://bugzilla.redhat.com/show_bug.cgi?id=1250713 to be resolved.
Hi Matt, Upgraded to ceph-10.2.5-34 and re-ran the tests. "tempest.api.object_storage.test_object_services.ObjectTest.test_delete_objecttest" still fails: # ostestr --pdb tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object {0} tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object [0.021684s] ... FAILED ============================== Failed 1 tests - output below: ============================== tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object[id-17738d45-03bd-4d45-9e0b-7b2f58f98687] ---------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "tempest/api/object_storage/test_object_services.py", line 342, in test_delete_object self.assertHeaders(resp, 'Object', 'DELETE') File "tempest/api/object_storage/base.py", line 95, in assertHeaders target, method)) File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: content-length header does not exist Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2017-02-27 09:12:45,905 30408 INFO [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 201 PUT http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.010s 2017-02-27 09:12:45,905 30408 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: test Response - Headers: {'status': '201', 'content-length': '0', 'last-modified': 'Mon, 27 Feb 2017 09:12:41 GMT', 'connection': 'close', 'etag': '098f6bcd4621d373cade4e832627b4f6', 'x-trans-id': 'tx000000000000000000002-0058b3ed89-2f660-us-east', 'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'content-type': 'application/json; charset=utf-8'} Body: 2017-02-27 09:12:45,914 30408 INFO [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 204 DELETE http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.008s 2017-02-27 09:12:45,914 30408 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'status': '204', 'connection': 'close', 'content-type': 'text/plain; charset=utf-8', 'x-trans-id': 'tx000000000000000000003-0058b3ed89-2f660-us-east'} Body:
Need to document that "rgw print prohibited content length = true" has to be set in ceph.conf.
Changing back to eng bug and adding this to doc tracker.
Was it tested with latest available refstack version? OpenStack Powered Platform 2016.08
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0514.html
(In reply to arkady kanevsky from comment #63) > Was it tested with latest available refstack version? > OpenStack Powered Platform 2016.08 Red Hat Ceph 2.2 passed the swift tests in RefStack version 10.0.0-3.
*** Bug 1448076 has been marked as a duplicate of this bug. ***
I have insured that rgw_print_prohibited_content_length = true (which looks like an invalid attribute according to latest ceph docs) and rgw_content_length_compat = true is set in ceph.conf and restarted the RGW service on all three controllers and I am still seeing the following failure for all object store tests. Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix self.assertHeaders(resp, 'Account', 'GET') File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/base.py", line 145, in assertHeaders target, method, self.policies)) File "/home/osp_admin/refstack-client/.tempest/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: accept-ranges header does not exist Ceph version is: 12.2.5-59.el7cp (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)
accept-ranges header missing causing all tests in test_account_services.py to fail. tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_prefix[id-365e6fc7-1cfe-463b-a37c-8bd08d47b6aa] -------------------------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix self.assertHeaders(resp, 'Account', 'GET') File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 145, in assertHeaders target, method, self.policies)) File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: accept-ranges header does not exist Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2019-08-30 16:46:30,212 990 INFO [tempest.lib.common.rest_client] Request (AccountTest:test_list_containers_with_prefix): 200 GET http://100.67.154.160:8080/swift/v1/?prefix=tempest-a 0.007s 2019-08-30 16:46:30,213 990 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'status': '200', u'content-length': '20', u'x-account-storage-policy-default-placement-object-count': '0', u'x-account-storage-policy-default-placement-bytes-used-actual': '0', u'x-trans-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', u'date': 'Fri, 30 Aug 2019 16:46:30 GMT', u'x-account-bytes-used': '0', u'x-account-container-count': '6', u'x-openstack-request-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', 'content-location': 'http://100.67.154.160:8080/swift/v1/?prefix=tempest-a', u'x-account-storage-policy-default-placement-bytes-used': '0', u'x-account-object-count': '0', u'connection': 'close', u'x-timestamp': '1567183590.21069', u'x-account-bytes-used-actual': '0', u'x-account-storage-policy-default-placement-container-count': '6', u'content-type': 'text/plain; charset=utf-8'} Body: tempest-a--456291905 All ceph libraries are are at version 2:12.2.4-42.el7, see below: ceph-common.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed libcephfs2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-cephfs.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed librgw2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rgw.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed librados2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed libradosstriper1.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rados.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days