Bug 1252600 - Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset [NEEDINFO]
Summary: Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 1.3.0
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: 2.2
Assignee: Matt Benjamin (redhat)
QA Contact: ceph-qe-bugs
Bara Ancincova
URL:
Whiteboard:
: 1413164 1448076 (view as bug list)
Depends On: 1250713
Blocks: 1246162 1261979 1273812 1305654 1383917 1412948
TreeView+ depends on / blocked
 
Reported: 2015-08-11 20:59 UTC by Neil Levine
Modified: 2019-08-30 16:56 UTC (History)
32 users (show)

Fixed In Version: RHEL: ceph-10.2.5-27.el7cp Ubuntu: ceph_10.2.5-19redhat1
Doc Type: Bug Fix
Doc Text:
.The Ceph Object Gateway now passes all Swift tests in the RefSTack Tempest test suite version 10.0.0-3 Previously, the Ceph Object Gateway failed certain RefSTack Tempest tests, such as the TempURL and object versioning tests. With this update, the underlying source code has been modified, and the Ceph Object Gateway now correctly passes all tests. In addition, to pass the "(0) content-length header after object deletion present" test, set the `rgw print prohibited content length` setting in the Ceph configuration file to `true`. If the Ceph Object Gateway is configured for Object Store and not for Swift, perform the following steps to pass the tests: . During the Tempest configuration, set the following parameters in the Ceph configuration file: + ---- rgw_swift_url_prefix = "/" rgw_enable_apis=swift, swift_auth, admin ---- . Once the configuration is complete, comment the parameters out: + ---- # rgw_swift_url_prefix # rgw_enable_apis ---- See the https://access.redhat.com/solutions/2953751[config_tempest.py breaks if Rados Gateway is configured for object-store] solution for details.
Clone Of:
: 1436386 (view as bug list)
Environment:
Last Closed: 2017-03-14 15:42:42 UTC
david_paterson: needinfo? (smanjara)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0514 normal SHIPPED_LIVE Red Hat Ceph Storage 2.2 bug fix and enhancement update 2017-03-21 07:24:26 UTC
Ceph Project Bug Tracker 16602 None None None 2017-02-17 22:27:29 UTC
Red Hat Bugzilla 1250713 None CLOSED config_tempest.py breaks if Rados Gateway is configured for object-store 2019-10-07 19:03:25 UTC
Red Hat Bugzilla 1436386 None CLOSED Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset (Refstack 15.0) 2019-10-07 19:03:25 UTC
Red Hat Knowledge Base (Solution) 2858981 None None None 2017-07-30 15:42:52 UTC

Internal Links: 1250713 1436386

Description Neil Levine 2015-08-11 20:59:31 UTC
Tempest is the upstream test tool for OpenStack.

RGW should pass all the tests for its Swift API tests.

Comment 2 Neil Levine 2015-08-11 21:11:59 UTC

*** This bug has been marked as a duplicate of bug 1250713 ***

Comment 3 arkady kanevsky 2015-08-11 21:34:40 UTC
Reopening this bug.
It is different from bug 1250713.
bug 1250713 deals with tempest configuration when RGW is part of openstack solution. It is discovered by Tempest and then tempest configuration fails.
So that BZ is a temporary fix to ignore swift and not configure tempest for swift testing if ceph is the object store.

This bug is talking about fixing it correctly. That is Ceph swift API respond correctly to tempest configurator calls.

This BZ is more closely related to https://bugzilla.redhat.com/show_bug.cgi?id=1246162.

Comment 6 Federico Lucifredi 2015-12-11 20:02:54 UTC
Let's use this ticket to track authoritatively what tests are not passing.

Comment 8 Matt Benjamin (redhat) 2015-12-11 20:06:41 UTC
I've taken on Neil's request to update list of non-conformances.

Comment 9 Federico Lucifredi 2015-12-11 20:09:33 UTC
Yehuda->Matt.

Comment 10 Neil Levine 2016-02-17 23:53:29 UTC
The new proposal from Arkady is to reduce the test criteria down to just those needed to pass DefCore RefStack:

https://refstack.openstack.org/#/capabilities

We should get a list of failures for both this and the full Tempest suite and then perhaps attack this in two phases i) RefStack and ii) full Tempest.

Comment 11 arkady kanevsky 2016-02-18 01:51:50 UTC
To be precise we want two things:
1. we want all swift tempest test that Ceph can support to pass.
1a. Understand why others do not pass
2. tempest tests of current 2016.1 version of refstack (https://refstack.openstack.org/#/capabilities) are passed with Swift API on Ceph (in our joint solution configuration)
These are:
tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_across_containers
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself
 tempest.api.object_storage.test_object_services.ObjectTest.test_create_object
 tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object
 tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time
 tempest.api.object_storage.test_object_services.ObjectTest.test_get_object
 tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_if_different
tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url
 tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments
 tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
 tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url

Longer term, every time refstack releases new version somebody need to review what tests are not part of object storage "core" and ensure they pass.
We can file new BZ when that happens. But these release come roughly every quarter. Since we need current refstack object tests to pass we potentially need bug fixes "backported" into the version of Ceph that is part of our joint solution we are releasing.

Comment 16 Matt Benjamin (redhat) 2016-10-03 17:39:00 UTC
We think the current builds pass RefCore.

Comment 17 Federico Lucifredi 2016-10-07 23:12:37 UTC
We are targeting this at RHCS 2.1 -

Comment 26 Andrea Perotti 2017-01-13 20:08:34 UTC
*** Bug 1413164 has been marked as a duplicate of this bug. ***

Comment 43 shilpa 2017-02-22 12:20:36 UTC
Tried with  ceph-10.2.5-29. 

config_tempest.py now fails with this error:

# python tools/config_tempest.py --debug identity.uri http://10.8.128.113:5000/v2.0 identity.admin_password da426a86fd7b4b4c --create
2017-02-22 09:45:41.179 4469 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2017-02-22 09:45:41.411 4469 INFO __main__ [-] Reading defaults from file '/root/tempest/etc/default-overrides.conf'
2017-02-22 09:45:41.412 4469 DEBUG __main__ [-] Setting [identity] uri = http://10.8.128.113:5000/v2.0 set tools/config_tempest.py:461
2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] admin_password = da426a86fd7b4b4c set tools/config_tempest.py:461
2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] uri_v3 = http://10.8.128.113:5000/v3 set tools/config_tempest.py:461
2017-02-22 09:45:41.532 4469 INFO tempest.lib.common.rest_client [req-5cf3fed9-b52b-49a1-a33d-6b941a3b012c ] Request (main): 200 POST http://10.8.128.113:5000/v2.0/tokens
2017-02-22 09:45:41.553 4469 INFO tempest.lib.common.rest_client [req-327791eb-9277-47d8-a108-70cea4942e25 ] Request (main): 200 GET http://10.8.128.113:35357/v2.0/tenants 0.020s
2017-02-22 09:45:41.553 4469 DEBUG __main__ [-] Setting [identity] admin_tenant_id = 0552cdc8375c4596a38faf32c679d746 set tools/config_tempest.py:461
2017-02-22 09:45:44.231 4469 CRITICAL tempest [-] ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404
2017-02-22 09:45:44.231 4469 ERROR tempest Traceback (most recent call last):
2017-02-22 09:45:44.231 4469 ERROR tempest   File "tools/config_tempest.py", line 830, in <module>
2017-02-22 09:45:44.231 4469 ERROR tempest     main()
2017-02-22 09:45:44.231 4469 ERROR tempest   File "tools/config_tempest.py", line 161, in main
2017-02-22 09:45:44.231 4469 ERROR tempest     api_version=api_version)
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 164, in discover
2017-02-22 09:45:44.231 4469 ERROR tempest     services[name]['extensions'] = service.get_extensions()
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 112, in get_extensions
2017-02-22 09:45:44.231 4469 ERROR tempest     top_level_path="info")
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 54, in do_get
2017-02-22 09:45:44.231 4469 ERROR tempest     " with code %d" % (self.name, url, r.status))
2017-02-22 09:45:44.231 4469 ERROR tempest ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404


I am not able to proceed with verifying the fix. Need this https://bugzilla.redhat.com/show_bug.cgi?id=1250713 to be resolved.

Comment 53 shilpa 2017-02-27 09:16:05 UTC
Hi Matt,

Upgraded to ceph-10.2.5-34 and re-ran the tests.

"tempest.api.object_storage.test_object_services.ObjectTest.test_delete_objecttest" still fails:

# ostestr --pdb tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object
{0} tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object [0.021684s] ... FAILED

==============================
Failed 1 tests - output below:
==============================

tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object[id-17738d45-03bd-4d45-9e0b-7b2f58f98687]
----------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/object_storage/test_object_services.py", line 342, in test_delete_object
        self.assertHeaders(resp, 'Object', 'DELETE')
      File "tempest/api/object_storage/base.py", line 95, in assertHeaders
        target, method))
      File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: content-length header does not exist
    

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2017-02-27 09:12:45,905 30408 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 201 PUT http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.010s
    2017-02-27 09:12:45,905 30408 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: test
        Response - Headers: {'status': '201', 'content-length': '0', 'last-modified': 'Mon, 27 Feb 2017 09:12:41 GMT', 'connection': 'close', 'etag': '098f6bcd4621d373cade4e832627b4f6', 'x-trans-id': 'tx000000000000000000002-0058b3ed89-2f660-us-east', 'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'content-type': 'application/json; charset=utf-8'}
            Body: 
    2017-02-27 09:12:45,914 30408 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 204 DELETE http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.008s
    2017-02-27 09:12:45,914 30408 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'status': '204', 'connection': 'close', 'content-type': 'text/plain; charset=utf-8', 'x-trans-id': 'tx000000000000000000003-0058b3ed89-2f660-us-east'}
            Body:

Comment 57 shilpa 2017-02-27 16:08:55 UTC
Need to document that "rgw print prohibited content length = true" has to be set in ceph.conf.

Comment 60 shilpa 2017-02-27 16:23:55 UTC
Changing back to eng bug and adding this to doc tracker.

Comment 63 arkady kanevsky 2017-02-27 23:19:02 UTC
Was it tested with latest available refstack version?
OpenStack Powered Platform 2016.08

Comment 77 errata-xmlrpc 2017-03-14 15:42:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0514.html

Comment 78 Uday Boppana 2017-03-14 18:52:48 UTC
(In reply to arkady kanevsky from comment #63)
> Was it tested with latest available refstack version?
> OpenStack Powered Platform 2016.08

Red Hat Ceph  2.2 passed the swift tests in RefStack version 10.0.0-3.

Comment 79 Gonéri Le Bouder 2017-05-04 15:10:51 UTC
*** Bug 1448076 has been marked as a duplicate of this bug. ***

Comment 80 David Paterson 2018-12-07 16:45:13 UTC
I have insured that rgw_print_prohibited_content_length = true (which looks like an invalid attribute according to latest ceph docs) and rgw_content_length_compat = true is set in ceph.conf and restarted the RGW service on all three controllers and I am still seeing the following failure for all object store tests.


Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
        self.assertHeaders(resp, 'Account', 'GET')
      File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/base.py", line 145, in assertHeaders
        target, method, self.policies))
      File "/home/osp_admin/refstack-client/.tempest/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: accept-ranges header does not exist


Ceph version is: 12.2.5-59.el7cp (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)

Comment 81 David Paterson 2018-12-07 16:45:40 UTC
I have insured that rgw_print_prohibited_content_length = true (which looks like an invalid attribute according to latest ceph docs) and rgw_content_length_compat = true is set in ceph.conf and restarted the RGW service on all three controllers and I am still seeing the following failure for all object store tests.


Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
        self.assertHeaders(resp, 'Account', 'GET')
      File "/home/osp_admin/refstack-client/.tempest/tempest/api/object_storage/base.py", line 145, in assertHeaders
        target, method, self.policies))
      File "/home/osp_admin/refstack-client/.tempest/.venv/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: accept-ranges header does not exist


Ceph version is: 12.2.5-59.el7cp (d4b9f17b56b3348566926849313084dd6efc2ca2) luminous (stable)

Comment 84 David Paterson 2019-08-30 16:56:23 UTC
accept-ranges header missing causing all tests in test_account_services.py to fail.

tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_prefix[id-365e6fc7-1cfe-463b-a37c-8bd08d47b6aa]
--------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_services.py", line 248, in test_list_containers_with_prefix
        self.assertHeaders(resp, 'Account', 'GET')
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 145, in assertHeaders
        target, method, self.policies))
      File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: accept-ranges header does not exist


Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-08-30 16:46:30,212 990 INFO     [tempest.lib.common.rest_client] Request (AccountTest:test_list_containers_with_prefix): 200 GET http://100.67.154.160:8080/swift/v1/?prefix=tempest-a 0.007s
    2019-08-30 16:46:30,213 990 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', u'content-length': '20', u'x-account-storage-policy-default-placement-object-count': '0', u'x-account-storage-policy-default-placement-bytes-used-actual': '0', u'x-trans-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', u'date': 'Fri, 30 Aug 2019 16:46:30 GMT', u'x-account-bytes-used': '0', u'x-account-container-count': '6', u'x-openstack-request-id': 'tx00000000000000000004d-005d6952e6-1adaa-default', 'content-location': 'http://100.67.154.160:8080/swift/v1/?prefix=tempest-a', u'x-account-storage-policy-default-placement-bytes-used': '0', u'x-account-object-count': '0', u'connection': 'close', u'x-timestamp': '1567183590.21069', u'x-account-bytes-used-actual': '0', u'x-account-storage-policy-default-placement-container-count': '6', u'content-type': 'text/plain; charset=utf-8'}
            Body: tempest-a--456291905

All ceph libraries are are at version 2:12.2.4-42.el7, see below:

ceph-common.x86_64                  2:12.2.4-42.el7    @rhos-13.0-signed
libcephfs2.x86_64                   2:12.2.4-42.el7    @rhos-13.0-signed
python-cephfs.x86_64                2:12.2.4-42.el7    @rhos-13.0-signed
librgw2.x86_64                      2:12.2.4-42.el7    @rhos-13.0-signed
python-rgw.x86_64                   2:12.2.4-42.el7    @rhos-13.0-signed
librados2.x86_64                    2:12.2.4-42.el7    @rhos-13.0-signed
libradosstriper1.x86_64             2:12.2.4-42.el7    @rhos-13.0-signed
python-rados.x86_64                 2:12.2.4-42.el7    @rhos-13.0-signed


Note You need to log in before you can comment on or make changes to this bug.