Bug 1252600 - Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset
Ensure RGW passes all upstream Swift Tempest tests in RefStack toolset
Status: CLOSED ERRATA
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
1.3.0
Unspecified Unspecified
high Severity unspecified
: rc
: 2.2
Assigned To: Matt Benjamin (redhat)
ceph-qe-bugs
Bara Ancincova
:
: 1413164 1448076 (view as bug list)
Depends On: 1250713
Blocks: 1246162 1261979 1273812 1305654 1383917 1412948
  Show dependency treegraph
 
Reported: 2015-08-11 16:59 EDT by Neil Levine
Modified: 2017-07-30 11:57 EDT (History)
29 users (show)

See Also:
Fixed In Version: RHEL: ceph-10.2.5-27.el7cp Ubuntu: ceph_10.2.5-19redhat1
Doc Type: Bug Fix
Doc Text:
.The Ceph Object Gateway now passes all Swift tests in the RefSTack Tempest test suite version 10.0.0-3 Previously, the Ceph Object Gateway failed certain RefSTack Tempest tests, such as the TempURL and object versioning tests. With this update, the underlying source code has been modified, and the Ceph Object Gateway now correctly passes all tests. In addition, to pass the "(0) content-length header after object deletion present" test, set the `rgw print prohibited content length` setting in the Ceph configuration file to `true`. If the Ceph Object Gateway is configured for Object Store and not for Swift, perform the following steps to pass the tests: . During the Tempest configuration, set the following parameters in the Ceph configuration file: + ---- rgw_swift_url_prefix = "/" rgw_enable_apis=swift, swift_auth, admin ---- . Once the configuration is complete, comment the parameters out: + ---- # rgw_swift_url_prefix # rgw_enable_apis ---- See the https://access.redhat.com/solutions/2953751[config_tempest.py breaks if Rados Gateway is configured for object-store] solution for details.
Story Points: ---
Clone Of:
: 1436386 (view as bug list)
Environment:
Last Closed: 2017-03-14 11:42:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2858981 None None None 2017-07-30 11:42 EDT
Ceph Project Bug Tracker 16602 None None None 2017-02-17 17:27 EST

  None (edit)
Description Neil Levine 2015-08-11 16:59:31 EDT
Tempest is the upstream test tool for OpenStack.

RGW should pass all the tests for its Swift API tests.
Comment 2 Neil Levine 2015-08-11 17:11:59 EDT

*** This bug has been marked as a duplicate of bug 1250713 ***
Comment 3 arkady kanevsky 2015-08-11 17:34:40 EDT
Reopening this bug.
It is different from bug 1250713.
bug 1250713 deals with tempest configuration when RGW is part of openstack solution. It is discovered by Tempest and then tempest configuration fails.
So that BZ is a temporary fix to ignore swift and not configure tempest for swift testing if ceph is the object store.

This bug is talking about fixing it correctly. That is Ceph swift API respond correctly to tempest configurator calls.

This BZ is more closely related to https://bugzilla.redhat.com/show_bug.cgi?id=1246162.
Comment 6 Federico Lucifredi 2015-12-11 15:02:54 EST
Let's use this ticket to track authoritatively what tests are not passing.
Comment 8 Matt Benjamin (redhat) 2015-12-11 15:06:41 EST
I've taken on Neil's request to update list of non-conformances.
Comment 9 Federico Lucifredi 2015-12-11 15:09:33 EST
Yehuda->Matt.
Comment 10 Neil Levine 2016-02-17 18:53:29 EST
The new proposal from Arkady is to reduce the test criteria down to just those needed to pass DefCore RefStack:

https://refstack.openstack.org/#/capabilities

We should get a list of failures for both this and the full Tempest suite and then perhaps attack this in two phases i) RefStack and ii) full Tempest.
Comment 11 arkady kanevsky 2016-02-17 20:51:50 EST
To be precise we want two things:
1. we want all swift tempest test that Ceph can support to pass.
1a. Understand why others do not pass
2. tempest tests of current 2016.1 version of refstack (https://refstack.openstack.org/#/capabilities) are passed with Swift API on Ceph (in our joint solution configuration)
These are:
tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_2d_way
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_across_containers
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_in_same_container
 tempest.api.object_storage.test_object_services.ObjectTest.test_copy_object_to_itself
 tempest.api.object_storage.test_object_services.ObjectTest.test_create_object
 tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object
 tempest.api.object_storage.test_object_expiry.ObjectExpiryTest.test_get_object_after_expiry_time
 tempest.api.object_storage.test_object_services.ObjectTest.test_get_object
 tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_if_different
tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url
 tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments
 tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
 tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_get_object_using_temp_url

Longer term, every time refstack releases new version somebody need to review what tests are not part of object storage "core" and ensure they pass.
We can file new BZ when that happens. But these release come roughly every quarter. Since we need current refstack object tests to pass we potentially need bug fixes "backported" into the version of Ceph that is part of our joint solution we are releasing.
Comment 16 Matt Benjamin (redhat) 2016-10-03 13:39:00 EDT
We think the current builds pass RefCore.
Comment 17 Federico Lucifredi 2016-10-07 19:12:37 EDT
We are targeting this at RHCS 2.1 -
Comment 26 Andrea Perotti 2017-01-13 15:08:34 EST
*** Bug 1413164 has been marked as a duplicate of this bug. ***
Comment 43 shilpa 2017-02-22 07:20:36 EST
Tried with  ceph-10.2.5-29. 

config_tempest.py now fails with this error:

# python tools/config_tempest.py --debug identity.uri http://10.8.128.113:5000/v2.0 identity.admin_password da426a86fd7b4b4c --create
2017-02-22 09:45:41.179 4469 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2017-02-22 09:45:41.411 4469 INFO __main__ [-] Reading defaults from file '/root/tempest/etc/default-overrides.conf'
2017-02-22 09:45:41.412 4469 DEBUG __main__ [-] Setting [identity] uri = http://10.8.128.113:5000/v2.0 set tools/config_tempest.py:461
2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] admin_password = da426a86fd7b4b4c set tools/config_tempest.py:461
2017-02-22 09:45:41.413 4469 DEBUG __main__ [-] Setting [identity] uri_v3 = http://10.8.128.113:5000/v3 set tools/config_tempest.py:461
2017-02-22 09:45:41.532 4469 INFO tempest.lib.common.rest_client [req-5cf3fed9-b52b-49a1-a33d-6b941a3b012c ] Request (main): 200 POST http://10.8.128.113:5000/v2.0/tokens
2017-02-22 09:45:41.553 4469 INFO tempest.lib.common.rest_client [req-327791eb-9277-47d8-a108-70cea4942e25 ] Request (main): 200 GET http://10.8.128.113:35357/v2.0/tenants 0.020s
2017-02-22 09:45:41.553 4469 DEBUG __main__ [-] Setting [identity] admin_tenant_id = 0552cdc8375c4596a38faf32c679d746 set tools/config_tempest.py:461
2017-02-22 09:45:44.231 4469 CRITICAL tempest [-] ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404
2017-02-22 09:45:44.231 4469 ERROR tempest Traceback (most recent call last):
2017-02-22 09:45:44.231 4469 ERROR tempest   File "tools/config_tempest.py", line 830, in <module>
2017-02-22 09:45:44.231 4469 ERROR tempest     main()
2017-02-22 09:45:44.231 4469 ERROR tempest   File "tools/config_tempest.py", line 161, in main
2017-02-22 09:45:44.231 4469 ERROR tempest     api_version=api_version)
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 164, in discover
2017-02-22 09:45:44.231 4469 ERROR tempest     services[name]['extensions'] = service.get_extensions()
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 112, in get_extensions
2017-02-22 09:45:44.231 4469 ERROR tempest     top_level_path="info")
2017-02-22 09:45:44.231 4469 ERROR tempest   File "/root/tempest/tempest/common/api_discovery.py", line 54, in do_get
2017-02-22 09:45:44.231 4469 ERROR tempest     " with code %d" % (self.name, url, r.status))
2017-02-22 09:45:44.231 4469 ERROR tempest ServiceError: Request on service 'object-store' with url 'http://magna039:8080/info' failed with code 404


I am not able to proceed with verifying the fix. Need this https://bugzilla.redhat.com/show_bug.cgi?id=1250713 to be resolved.
Comment 53 shilpa 2017-02-27 04:16:05 EST
Hi Matt,

Upgraded to ceph-10.2.5-34 and re-ran the tests.

"tempest.api.object_storage.test_object_services.ObjectTest.test_delete_objecttest" still fails:

# ostestr --pdb tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object
{0} tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object [0.021684s] ... FAILED

==============================
Failed 1 tests - output below:
==============================

tempest.api.object_storage.test_object_services.ObjectTest.test_delete_object[id-17738d45-03bd-4d45-9e0b-7b2f58f98687]
----------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/object_storage/test_object_services.py", line 342, in test_delete_object
        self.assertHeaders(resp, 'Object', 'DELETE')
      File "tempest/api/object_storage/base.py", line 95, in assertHeaders
        target, method))
      File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: content-length header does not exist
    

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2017-02-27 09:12:45,905 30408 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 201 PUT http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.010s
    2017-02-27 09:12:45,905 30408 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: test
        Response - Headers: {'status': '201', 'content-length': '0', 'last-modified': 'Mon, 27 Feb 2017 09:12:41 GMT', 'connection': 'close', 'etag': '098f6bcd4621d373cade4e832627b4f6', 'x-trans-id': 'tx000000000000000000002-0058b3ed89-2f660-us-east', 'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'content-type': 'application/json; charset=utf-8'}
            Body: 
    2017-02-27 09:12:45,914 30408 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_delete_object): 204 DELETE http://magna039:8080/swift/v1/tempest-TestContainer-186450910/tempest-TestObject-1143299553 0.008s
    2017-02-27 09:12:45,914 30408 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'date': 'Mon, 27 Feb 2017 09:12:41 GMT', 'status': '204', 'connection': 'close', 'content-type': 'text/plain; charset=utf-8', 'x-trans-id': 'tx000000000000000000003-0058b3ed89-2f660-us-east'}
            Body:
Comment 57 shilpa 2017-02-27 11:08:55 EST
Need to document that "rgw print prohibited content length = true" has to be set in ceph.conf.
Comment 60 shilpa 2017-02-27 11:23:55 EST
Changing back to eng bug and adding this to doc tracker.
Comment 63 arkady kanevsky 2017-02-27 18:19:02 EST
Was it tested with latest available refstack version?
OpenStack Powered Platform 2016.08
Comment 77 errata-xmlrpc 2017-03-14 11:42:42 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0514.html
Comment 78 Uday Boppana 2017-03-14 14:52:48 EDT
(In reply to arkady kanevsky from comment #63)
> Was it tested with latest available refstack version?
> OpenStack Powered Platform 2016.08

Red Hat Ceph  2.2 passed the swift tests in RefStack version 10.0.0-3.
Comment 79 Gonéri Le Bouder 2017-05-04 11:10:51 EDT
*** Bug 1448076 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.