Bug 1670217
| Summary: | RGW - Tempest test: ObjectTestACLs.test_read_object_with_rights fails with 404 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Paterson <david_paterson> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Giulio Fidente <gfidente> |
| Status: | CLOSED ERRATA | QA Contact: | Eliad Cohen <elicohen> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 13.0 (Queens) | CC: | arkady_kanevsky, cdevine, chadd, christopher_dearborn, david_paterson, dcain, elicohen, gael_rehault, gfidente, jdurgin, kurt_hey, lhh, mbenjamin, mburns, morazi, mwatts, nweinber, prsrivas, rajini.karthik, sclewis, tserlin |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:19:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
David Paterson
2019-01-28 23:10:34 UTC
Hi David, Can you please re-run the test with the following additional configs: In ceph.conf: rgw swift account in url = true in ceph.conf and while setting up endpoints for ceph swift, you need to use the following command: openstack endpoint create --region RegionOne <service-id> public 'http://<hostname>:<portnum>/v1/AUTH_$(project_id)s' ------(Note AUTH_$(project_id)s In tempest conf file, set discoverability to True: [object-storage-feature-enabled] discoverability = True The cross tenant tests passed for me with the above mentioned configs in place. Thanks, Pritha I'll need to see your tempest.conf as when I set
[object-storage-feature-enabled]
discoverability = True
I get:
tempest.lib.exceptions.NotFound: Object not found
Details: {u'HostId': u'19603-default-default', u'Code': u'NoSuchBucket', u'BucketName': u'info', u'RequestId': u'tx0000000000000000001f3-005cb
[DEFAULT] debug = true use_stderr = false log_file = tempest.log [auth] tempest_roles = member,Member admin_username = admin admin_project_name = admin admin_domain_name = Default use_dynamic_credentials = true admin_password = nova admin_project_id = 6e73ac1d59204036b195f1883cc7ff4d [identity] uri_v3 = http://192.168.122.1/identity/v3/auth/tokens username=demo password=nova project_name = demo domain_name = Default auth_version = v3 endpoint_type=publicURL catalog_type=identity disable_ssl_certificate_validation = true region = regionOne [identity-feature-enabled] api_v2 = false api_v3 = true [object-storage-feature-enabled] discoverability = True [network] floating_network_name = public [object-storage] operator_role = SwiftOperator reseller_admin_role = ResellerAdmin [oslo_concurrency] lock_path = /tmp/tempest [service_available] cinder = false neutron = false glance = false swift = true nova = false heat = false [validation] run_validation = true connect_method = floating auth_method = keypair Still having same issue, must be something different in ceph.conf. Can you please paste your controller's ceph.conf and I can find the delta. Thanks, David rgw keystone url = http://192.168.122.1/identity/ rgw keystone admin project = admin rgw keystone admin tenant = admin rgw keystone admin domain = default rgw keystone api version = 3 rgw keystone implicit tenants = true rgw keystone admin password = nova rgw keystone admin user = admin rgw keystone accepted roles = SwiftOperator,admin,_member_,ResellerAdmin rgw keystone accepted admin roles = ResellerAdmin rgw keystone token cache size = 500 rgw keystone revocation interval = 500 rgw s3 auth use keystone = true rgw nss db path = /var/ceph/nss rgw keystone verify ssl = false rgw swift enforce content length = true rgw swift versioning enabled = true rgw swift url prefix = "/" rgw enable apis = swift, swift_auth, admin rgw swift account in url = true Hi David, Do you have the following config set in ceph.conf? rgw swift account in url = true This is needed for the test to pass. Thanks, Pritha Yes I do have that set, the problem is when I set: [object-storage-feature-enabled] discoverability = True Tempest attempts a call to <rgw_url>/info, which throws an error. Think we may have disparate version of ceph rgw, do your package versions match the ones in my initial bug report? Hi David, In order to get /info working, you need to have the following two configs set in ceph.conf. The first one puts swift apis at root (/) and the second one explicitly enables swift and swift related apis (leaving out s3 related apis). rgw swift url prefix = "/" rgw enable apis = swift, swift_auth, admin Please try with these options and let me know. Thanks, Pritha Will test your suggestion and verify, are these config attributes noted anywhere in RHOSP docs? If not it would be useful to add. Thanks Pritha. I've tried running the tempest config script with object store discoverability turned on and all three controllers having the attributes you mention. The config script runs but no api discovery is done. I just end up with the following in tempest.conf [object-storage-feature-enabled] discoverability = true discoverable_apis = I have tried all the recommended work arounds and I am still receiving 404 erorrs (BucketNotFound) for many object tests:
tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_write_object_with_rights[id-aa58bfa5-40d9-4bc3-82b4-d07f4a9e392a]
-----------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_container_acl.py", line 88, in test_write_object_with_rights
object_name, 'data', headers={})
File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/object_client.py", line 41, in create_object
resp, body = self.put(url, data, headers, chunked=chunked)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
return self.request('PUT', url, extra_headers, headers, body, chunked)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
self._error_checker(resp, resp_body)
File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 774, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: NoSuchBucket
Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
2019-08-30 17:40:20,295 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 POST http://100.67.154.160:5000/v3/auth/tokens
2019-08-30 17:40:20,295 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
Body: <omitted>
Response - Headers: {'status': '201', u'content-length': '9712', 'content-location': 'http://100.67.154.160:5000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Fri, 30 Aug 2019 17:40:19 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-66672378-7a35-4272-927b-aaf08905a3ac'}
Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "7cc1ddc9d55c4fae847f9508bdaf1bfd", "name": "ResellerAdmin"}, {"id": "57505ea313ad43d280103b88cf8f4a19", "name": "Member"}], "expires_at": "2019-08-30T18:40:20.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "36e9c313427e49748af5e6e237406f88", "name": "tempest-ObjectTestACLs-1148683088"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "admin", "id": "59053e38dc1b42ca999348f04f64b79b"}, {"region_id": "regionOne", "url": "http://100.67.154.160:9292", "region": "regionOne", "interface": "public", "id": "729de2bb8a7b4df8bb0e5688e8973836"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "internal", "id": "c95f6e29793e411095e9c9ebdfe3bbd0"}], "type": "image", "id": "08cde5f69e914bf497ea5bad3ff7f5bd", "name": "glance"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.67.154.160:8042", "region": "regionOne", "interface": "public", "id": "3c38df4d3a0d43e79f398f859589c1cb"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8042", "region": "regionOne", "interface": "internal", "id": "94be2b8f8ee34e9c813680e4d5236b42"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8042", "region": "regionOne", "interface": "admin", "id": "d341a3fb000947b393c597ed791bca34"}], "type": "alarming", "id": "10253f25166a4a9587beb4f3b56e82b0", "name": "aodh"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "internal", "id": "0702dd9100f34fd1945d9ea43b17ae81"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "public", "id": "681e036e70da4619b655bf794bc953ea"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "admin", "id": "abd858865fe14d7aaaab816a731bcd03"}], "type": "orchestration", "id": "11b37d21d85b46e8a41e5d701eea1b93", "name": "heat"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "admin", "id": "47d6896f039147519acdd323f207dc4f"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "internal", "id": "cbe7a31bd5f4495eb167d4bdcd0f6cf6"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "public", "id": "f706bde2fa824e7eb2eb590caa3f55a7"}], "type": "volumev3", "id": "156385ed278c4bfeb5c864fa84ed4419", "name": "cinderv3"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.67.154.160:8774/v2.1", "region": "regionOne", "interface": "public", "id": "5666f0a00eaa42f3b96e747aa08c8058"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "92f59ee5c38c43df8c577294976ea584"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "d4c37580b471438695f3246ca7cfa9c0"}], "type": "compute", "id": "1ecd325cd71d462e80e9a006fb3cfd06", "name": "nova"}, {"endpoints": [], "type": "tempest-type-1214509154", "id": "247d3dfceb6f472496b7c4f3c9702b1b", "name": "tempest-service-1389917785"}, {"endpoints": [], "type": "tempest-type-839988184", "id": "3085e4103d0a4ce486b3e0b4fb2d56d1", "name": "tempest-service-644891461"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8041", "region": "regionOne", "interface": "internal", "id": "0b2717c8444d4df59c10b0d199a63a8c"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8041", "region": "regionOne", "interface": "public", "id": "a18f79b662994a9894209e03fc6d4328"}, {"region_id": "r
2019-08-30 17:40:20,490 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 PUT http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.194s
2019-08-30 17:40:20,490 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '201', u'content-length': '0', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx0000000000000000013d7-005d695f84-13689-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013d7-005d695f84-13689-default'}
Body:
2019-08-30 17:40:20,669 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_write_object_with_rights): 204 POST http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.178s
2019-08-30 17:40:20,669 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Container-Write': u'tempest-ObjectTestACLs-1966151171:tempest-ObjectTestACLs-1966151171', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '204', u'connection': 'close', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'x-trans-id': 'tx0000000000000000013a9-005d695f84-13b5e-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013a9-005d695f84-13b5e-default'}
Body:
2019-08-30 17:40:20,847 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_write_object_with_rights): 404 PUT http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920/tempest-Object-291679424 0.176s
2019-08-30 17:40:20,847 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
Body: data
Response - Headers: {'status': '404', u'content-length': '12', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920/tempest-Object-291679424', u'accept-ranges': 'bytes', u'last-modified': 'Thu, 01 Jan 1970 00:00:00 GMT', u'connection': 'close', u'x-trans-id': 'tx000000000000000001375-005d695f84-1adaa-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000001375-005d695f84-1adaa-default'}
Body: NoSuchBucket
2019-08-30 17:40:20,855 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 200 GET http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920?&limit=9999&format=json 0.004s
2019-08-30 17:40:20,856 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '200', u'content-length': '2', u'x-container-object-count': '0', u'x-container-write': 'tempest-ObjectTestACLs-1966151171:tempest-ObjectTestACLs-1966151171', u'accept-ranges': 'bytes', u'x-storage-policy': 'default-placement', u'x-container-bytes-used-actual': '0', u'x-container-bytes-used': '0', u'connection': 'close', u'x-timestamp': '1567186820.48106', u'x-trans-id': 'tx0000000000000000013d8-005d695f84-13689-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013d8-005d695f84-13689-default', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920?&limit=9999&format=json'}
Body: []
2019-08-30 17:40:22,874 7442 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 204 DELETE http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.015s
2019-08-30 17:40:22,875 7442 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
Body: None
Response - Headers: {'status': '204', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx0000000000000000013ad-005d695f86-13b5e-default', u'date': 'Fri, 30 Aug 2019 17:40:22 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013ad-005d695f86-13b5e-default'}
Body:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |