Bug 1252600
| Summary: | Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Neil Levine <nlevine> | |
| Component: | RGW | Assignee: | Matt Benjamin (redhat) <mbenjamin> | |
| Status: | CLOSED ERRATA | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> | |
| Severity: | medium | Docs Contact: | Bara Ancincova <bancinco> | |
| Priority: | high | |||
| Version: | 1.3.0 | CC: | aperotti, arkady_kanevsky, cbodley, cdevine, ceph-eng-bugs, christopher_dearborn, david_paterson, dkranz, flucifre, goneri, hnallurv, icolle, kbader, kdreyer, mbenjamin, mburns, morazi, mwatts, nbarcet, nlevine, owasserm, randy_perryman, rsussman, rwheeler, skatlapa, smanjara, sreichar, sweil, thiago, tserlin, uboppana, vumrao | |
| Target Milestone: | rc | |||
| Target Release: | 2.2 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | RHEL: ceph-10.2.5-27.el7cp Ubuntu: ceph_10.2.5-19redhat1 | Doc Type: | Bug Fix | |
| Doc Text: |
.The Ceph Object Gateway now passes all Swift tests in the RefSTack Tempest test suite version 10.0.0-3
Previously, the Ceph Object Gateway failed certain RefSTack Tempest tests, such as the TempURL and object versioning tests. With this update, the underlying source code has been modified, and the Ceph Object Gateway now correctly passes all tests.
In addition, to pass the "(0) content-length header after object deletion present" test, set the `rgw print prohibited content length` setting in the Ceph configuration file to `true`.
If the Ceph Object Gateway is configured for Object Store and not for Swift, perform the following steps to pass the tests:
. During the Tempest configuration, set the following parameters in the Ceph configuration file:
+
----
rgw_swift_url_prefix = "/"
rgw_enable_apis=swift, swift_auth, admin
----
. Once the configuration is complete, comment the parameters out:
+
----
# rgw_swift_url_prefix
# rgw_enable_apis
----
See the https://access.redhat.com/solutions/2953751[config_tempest.py breaks if Rados Gateway is configured for object-store] solution for details.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1436386 (view as bug list) | Environment: | ||
| Last Closed: | 2017-03-14 15:42:42 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1250713 | |||
| Bug Blocks: | 1246162, 1261979, 1273812, 1305654, 1383917, 1412948 | |||
|
Description
Neil Levine
2015-08-11 20:59:31 UTC
*** This bug has been marked as a duplicate of bug 1250713 *** Reopening this bug. It is different from bug 1250713. bug 1250713 deals with tempest configuration when RGW is part of openstack solution. It is discovered by Tempest and then tempest configuration fails. So that BZ is a temporary fix to ignore swift and not configure tempest for swift testing if ceph is the object store. This bug is talking about fixing it correctly. That is Ceph swift API respond correctly to tempest configurator calls. This BZ is more closely related to https://bugzilla.redhat.com/show_bug.cgi?id=1246162. Let's use this ticket to track authoritatively what tests are not passing. I've taken on Neil's request to update list of non-conformances. Yehuda->Matt. The new proposal from Arkady is to reduce the test criteria down to just those needed to pass DefCore RefStack: https://refstack.openstack.org/#/capabilities We should get a list of failures for both this and the full Tempest suite and then perhaps attack this in two phases i) RefStack and ii) full Tempest. To be precise we want two things: 1. we want all swift tempest test that Ceph can support to pass. 1a. Understand why others do not pass 2. tempest tests of current 2016.1 version of refstack (https://refstack.openstack.org/#/capabilities) are passed with Swift API on Ceph (in our joint solution configuration) These are: tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_across_containers tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself tempest.api.object_storage.test_object_services.ObjectTest.test_create_object tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time tempest.api.object_storage.test_object_services.ObjectTest.test_get_object tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_if_different tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url Longer term, every time refstack releases new version somebody need to review what tests are not part of object storage "core" and ensure they pass. We can file new BZ when that happens. But these release come roughly every quarter. Since we need current refstack object tests to pass we potentially need bug fixes "backported" into the version of Ceph that is part of our joint solution we are releasing. We think the current builds pass RefCore. We are targeting this at RHCS 2.1 - *** Bug 1413164 has been marked as a duplicate of this bug. *** Tried with ceph-10.2.5-29. config_tempest.py now fails with this error: # python tools/config_tempest.py --debug identity.uri http://10.8.128.113:5000/v2.0 identity.admin_password da426a86fd7b4b4c --create 2017-02-22 09:45:41.179 4469 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf 2017-02-22 09:45:41.411 4469 INFO __main__ [-] Reading defaults from file '/root/tempest/etc/default-overrides.conf' 2017-02-22 09:45:41.412 4469 DEBUG __main__ [-] Setting [identity] uri = http://10.8.128.113:5000/v2.0 set tools/config_tempest.py:461 2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] admin_password = da426a86fd7b4b4c set tools/config_tempest.py:461 2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] uri_v3 = http://10.8.128.113:5000/v3 set tools/config_tempest.py:461 2017-02-22 09:45:41.532 4469 INFO tempest.lib.common.rest_client [req-5cf3fed9-b52b-49a1-a33d-6b941a3b012c ] Request (main): 200 POST http://10.8.128.113:5000/v2.0/tokens 2017-02-22 09:45:41.553 4469 INFO tempest.lib.common.rest_client [req-327791eb-9277-47d8-a108-70cea4942e25 ] Request (main): 200 GET http://10.8.128.113:35357/v2.0/tenants 0.020s 2017-02-22 09:45:41.553 4469 DEBUG __main__ [-] Setting [identity] admin_tenant_id = 0552cdc8375c4596a38faf32c679d746 set tools/config_tempest.py:461 2017-02-22 09:45:44.231 4469 CRITICAL tempest [-] ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404 2017-02-22 09:45:44.231 4469 ERROR tempest Traceback (most recent call last): 2017-02-22 09:45:44.231 4469 ERROR tempest File "tools/config_tempest.py", line 830, in <module> 2017-02-22 09:45:44.231 4469 ERROR tempest main() 2017-02-22 09:45:44.231 4469 ERROR tempest File "tools/config_tempest.py", line 161, in main 2017-02-22 09:45:44.231 4469 ERROR tempest api_version=api_version) 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 164, in discover 2017-02-22 09:45:44.231 4469 ERROR tempest services[name]['extensions'] = service.get_extensions() 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 112, in get_extensions 2017-02-22 09:45:44.231 4469 ERROR tempest top_level_path="info") 2017-02-22 09:45:44.231 4469 ERROR tempest File "/root/tempest/tempest/common/api_discovery.py", line 54, in do_get 2017-02-22 09:45:44.231 4469 ERROR tempest " with code %d" % (self.name, url, r.status)) 2017-02-22 09:45:44.231 4469 ERROR tempest ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404 I am not able to proceed with verifying the fix. Need this https://bugzilla.redhat.com/show_bug.cgi?id=1250713 to be resolved. Hi Matt,
Upgraded to ceph-10.2.5-34 and re-ran the tests.
"tempest.api.object_storage.test_object_services.ObjectTest.test_delete_objecttest" still fails:
# ostestr --pdb tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object
{0} tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object [0.021684s] ... FAILED
==============================
Failed 1 tests - output below:
==============================
tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object[id-17738d45-03bd-4d45-9e0b-7b2f58f98687]
----------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/api/object_storage/test_object_services.py", line 342, in test_delete_object
self.assertHeaders(resp, 'Object', 'DELETE')
File "tempest/api/object_storage/base.py", line 95, in assertHeaders
target, method))
File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: content-length header does not exist
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2017-02-27 09:12:45,905 30408 INFO [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 201 PUT http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.010s
2017-02-27 09:12:45,905 30408 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: test
Response - Headers: {'status': '201', 'content-length': '0', 'last-modified': 'Mon, 27 Feb 2017 09:12:41 GMT', 'connection': 'close', 'etag': '098f6bcd4621d373cade4e832627b4f6', 'x-trans-id': 'tx000000000000000000002-0058b3ed89-2f660-us-east', 'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'content-type': 'application/json; charset=utf-8'}
Body:
2017-02-27 09:12:45,914 30408 INFO [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 204 DELETE http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.008s
2017-02-27 09:12:45,914 30408 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'status': '204', 'connection': 'close', 'content-type': 'text/plain; charset=utf-8', 'x-trans-id': 'tx000000000000000000003-0058b3ed89-2f660-us-east'}
Body:
Need to document that "rgw print prohibited content length = true" has to be set in ceph.conf. Changing back to eng bug and adding this to doc tracker. Was it tested with latest available refstack version? OpenStack Powered Platform 2016.08 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2017-0514.html (In reply to arkady kanevsky from comment #63) > Was it tested with latest available refstack version? > OpenStack Powered Platform 2016.08 Red Hat Ceph 2.2 passed the swift tests in RefStack version 10.0.0-3. *** Bug 1448076 has been marked as a duplicate of this bug. *** I have insured that rgw_print_prohibited_content_length = true (which looks like an invalid attribute according to latest ceph docs) and rgw_content_length_compat = true is set in ceph.conf and restarted the RGW service on all three controllers and I am still seeing the following failure for all object store tests.
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
self.assertHeaders(resp, 'Account', 'GET')
File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/base.py", line 145, in assertHeaders
target, method, self.policies))
File "/home/osp_admin/refstack-client/.tempest/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: accept-ranges header does not exist
Ceph version is: 12.2.5-59.el7cp (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)
I have insured that rgw_print_prohibited_content_length = true (which looks like an invalid attribute according to latest ceph docs) and rgw_content_length_compat = true is set in ceph.conf and restarted the RGW service on all three controllers and I am still seeing the following failure for all object store tests.
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
self.assertHeaders(resp, 'Account', 'GET')
File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/base.py", line 145, in assertHeaders
target, method, self.policies))
File "/home/osp_admin/refstack-client/.tempest/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: accept-ranges header does not exist
Ceph version is: 12.2.5-59.el7cp (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)
accept-ranges header missing causing all tests in test_account_services.py to fail.
tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_prefix[id-365e6fc7-1cfe-463b-a37c-8bd08d47b6aa]
--------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
self.assertHeaders(resp, 'Account', 'GET')
File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 145, in assertHeaders
target, method, self.policies))
File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: accept-ranges header does not exist
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2019-08-30 16:46:30,212 990 INFO [tempest.lib.common.rest_client] Request (AccountTest:test_list_containers_with_prefix): 200 GET http://100.67.154.160:8080/swift/v1/?prefix=tempest-a 0.007s
2019-08-30 16:46:30,213 990 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '200', u'content-length': '20', u'x-account-storage-policy-default-placement-object-count': '0', u'x-account-storage-policy-default-placement-bytes-used-actual': '0', u'x-trans-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', u'date': 'Fri, 30 Aug 2019 16:46:30 GMT', u'x-account-bytes-used': '0', u'x-account-container-count': '6', u'x-openstack-request-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', 'content-location': 'http://100.67.154.160:8080/swift/v1/?prefix=tempest-a', u'x-account-storage-policy-default-placement-bytes-used': '0', u'x-account-object-count': '0', u'connection': 'close', u'x-timestamp': '1567183590.21069', u'x-account-bytes-used-actual': '0', u'x-account-storage-policy-default-placement-container-count': '6', u'content-type': 'text/plain; charset=utf-8'}
Body: tempest-a--456291905
All ceph libraries are are at version 2:12.2.4-42.el7, see below:
ceph-common.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
libcephfs2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
python-cephfs.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
librgw2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
python-rgw.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
librados2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
libradosstriper1.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
python-rados.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |