Bug 1670217 - RGW - Tempest test: ObjectTestACLs.test_read_object_with_rights fails with 404 [NEEDINFO]
Summary: RGW - Tempest test: ObjectTestACLs.test_read_object_with_rights fails with 404
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 15.0 (Stein)
Assignee: Giulio Fidente
QA Contact: Eliad Cohen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-28 23:10 UTC by David Paterson
Modified: 2019-09-26 10:47 UTC (History)
21 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:19:59 UTC
Target Upstream Version:
david_paterson: needinfo? (prsrivas)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1826894 0 None None None 2019-04-29 15:53:12 UTC
OpenStack gerrit 656361 0 'None' MERGED Enable Ceph RGW 'account in url' option 2021-01-27 16:39:21 UTC
OpenStack gerrit 666652 0 'None' MERGED Enable Ceph RGW 'account in url' option 2021-01-27 16:39:21 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:20:21 UTC

Description David Paterson 2019-01-28 23:10:34 UTC
Description of problem:


Version-Release number of selected component (if applicable):
RefStack current object store policy: wget "https://refstack.openstack.org/api/v1/guidelines/2018.11/tests?target=object&type=required&alias=true&flag=false" -O 2018.11-test-list.txt

RGW packages on controller
librgw2.x86_64                      2:12.2.4-42.el7    @rhos-13.0-signed
python-rgw.x86_64                   2:12.2.4-42.el7    @rhos-13.0-signed
librados2.x86_64                    2:12.2.4-42.el7    @rhos-13.0-signed
libradosstriper1.x86_64             2:12.2.4-42.el7    @rhos-13.0-signed
python-rados.x86_64                 2:12.2.4-42.el7    @rhos-13.0-signed



How reproducible:

Every time
Steps to Reproduce:
1. Run ostestr --regex tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_read_object_with_rights
2.
3.

Actual results:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_container_acl.py", line 62, in test_read_object_with_rights
        self.container_name, object_name)
      File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/object_client.py", line 85, in get_object
        resp, body = self.get(url, headers=headers)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get
        return self.request('GET', url, extra_headers, headers)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 776, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: NoSuchBucket


Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-01-28 23:06:33,450 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 POST http://100.82.36.190:5000/v3/auth/tokens
    2019-01-28 23:06:33,451 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
            Body: <omitted>
        Response - Headers: {'status': '201', u'content-length': '9429', 'content-location': 'http://100.82.36.190:5000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Mon, 28 Jan 2019 23:06:33 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-633430ee-095f-459e-87a2-1848aee54293'}
            Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5fe6a7b1273a413c9dfa1655870fe40d", "name": "Member"}, {"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "c268566a511a4281bf1c2de2e9501e5a", "name": "ResellerAdmin"}], "expires_at": "2019-01-29T00:06:33.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cf2202f05e8949fd8c9d2a055b38bb62", "name": "tempest-ObjectTestACLs-806390082"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "http://192.168.170.251:8080/swift/v1", "region": "regionOne", "interface": "admin", "id": "1a3912c8c597443f973bc8ac5bff0f45"}, {"region_id": "regionOne", "url": "http://192.168.170.251:8080/swift/v1", "region": "regionOne", "interface": "internal", "id": "1c51111e9da5433ebeb28589eab41d95"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8080/swift/v1", "region": "regionOne", "interface": "public", "id": "ac2de181ee9e43eb85490680d1b37cce"}], "type": "object-store", "id": "015abd8e050743f8b5575c8c0cf2be1d", "name": "swift"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:8776/v3/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "internal", "id": "a024460483ef4b268bff389570f9fd81"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8776/v3/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "public", "id": "e925b62bddf74e09a8c3881bab1645ee"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v3/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "admin", "id": "e9957aefc2784d8b815bcb16df978fd2"}], "type": "volumev3", "id": "0448310d75fe4313b748e295f4d4eda9", "name": "cinderv3"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.82.36.190:9292", "region": "regionOne", "interface": "public", "id": "35b49532f7604d73b7847b32f6ef024b"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "internal", "id": "7ea8645c6c1b4fd6a4ef06a12c54820d"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "admin", "id": "896a29a6442b4759b76c0083f53bd19f"}], "type": "image", "id": "0baf8ffd467345b38fb798784b05c8b7", "name": "glance"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:9696", "region": "regionOne", "interface": "internal", "id": "1d6782d296964b69942bf725dca7c99c"}, {"region_id": "regionOne", "url": "http://100.82.36.190:9696", "region": "regionOne", "interface": "public", "id": "30c0262501244369af7422d606bfed10"}, {"region_id": "regionOne", "url": "http://192.168.140.251:9696", "region": "regionOne", "interface": "admin", "id": "33b2e1df3b1148b6a419190f9386b5ac"}], "type": "network", "id": "24cd4bd5384449b183e44edc48ae93a0", "name": "neutron"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "2774048e10614ea7bafd5eef8494ce9b"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8774/v2.1", "region": "regionOne", "interface": "public", "id": "4272634957b348c88cd21404fcfc0b8e"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "7cbeece3b6fb425b871866ca032462b4"}], "type": "compute", "id": "556e411d989545feb396de183b608f93", "name": "nova"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.82.36.190:8776/v2/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "public", "id": "1bf9d9cd666040a791a0c70a1a55430f"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v2/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "admin", "id": "5155f74571864d158359bc1292cd66de"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v2/cf2202f05e8949fd8c9d2a055b38bb62", "region": "regionOne", "interface": "internal", "id": "6b53208fd40e409690812451c0cbc5e8"}], "type": "volumev2", "id": "5fa5997beb5a4621910eca7be8654798", "name": "cinderv2"}, {"endpoints": [{"region
    2019-01-28 23:06:33,662 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 PUT http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426 0.211s
    2019-01-28 23:06:33,663 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '201', u'content-length': '0', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx00000000000000000395f-005c4f8af9-46c45-default', u'date': 'Mon, 28 Jan 2019 23:06:33 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx00000000000000000395f-005c4f8af9-46c45-default'}
            Body:
    2019-01-28 23:06:33,866 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_read_object_with_rights): 204 POST http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426 0.203s
    2019-01-28 23:06:33,866 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'X-Container-Read': u'4499769742d24a33bc9f8f141d4c1834:b7f72372e38741ceaa9d2b53a4c09e5b'}
            Body: None
        Response - Headers: {'status': '204', u'connection': 'close', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426', u'x-trans-id': 'tx000000000000000049746-005c4f8af9-2ec00-default', u'date': 'Mon, 28 Jan 2019 23:06:33 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000049746-005c4f8af9-2ec00-default'}
            Body:
    2019-01-28 23:06:34,066 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_read_object_with_rights): 201 PUT http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786 0.200s
    2019-01-28 23:06:34,067 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: data
        Response - Headers: {'status': '201', u'content-length': '0', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786', u'last-modified': 'Mon, 28 Jan 2019 23:06:34 GMT', u'connection': 'close', u'etag': '8d777f385d3dfec8815d20f7496026dc', u'x-trans-id': 'tx000000000000000049765-005c4f8af9-2ebf7-default', u'date': 'Mon, 28 Jan 2019 23:06:34 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000049765-005c4f8af9-2ebf7-default'}
            Body:
    2019-01-28 23:06:34,280 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_read_object_with_rights): 404 GET http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786 0.213s
    2019-01-28 23:06:34,280 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '404', u'content-length': '12', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000003960-005c4f8afa-46c45-default', u'date': 'Mon, 28 Jan 2019 23:06:34 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000003960-005c4f8afa-46c45-default'}
            Body: NoSuchBucket
    2019-01-28 23:06:34,285 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 200 GET http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426?&limit=9999&format=json 0.004s
    2019-01-28 23:06:34,285 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', u'content-length': '165', u'x-container-object-count': '1', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426?&limit=9999&format=json', u'accept-ranges': 'bytes', u'x-container-read': '4499769742d24a33bc9f8f141d4c1834:b7f72372e38741ceaa9d2b53a4c09e5b', u'x-storage-policy': 'default-placement', u'x-container-bytes-used-actual': '4096', u'x-container-bytes-used': '4', u'connection': 'close', u'x-timestamp': '1548716793.64254', u'x-trans-id': 'tx000000000000000049747-005c4f8afa-2ec00-default', u'date': 'Mon, 28 Jan 2019 23:06:34 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000049747-005c4f8afa-2ec00-default'}
            Body: [{"name":"tempest-Object-13806786","hash":"8d777f385d3dfec8815d20f7496026dc","bytes":4,"content_type":"application/json","last_modified":"2019-01-28T23:06:34.122Z"}]
    2019-01-28 23:06:34,291 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 204 DELETE http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786 0.005s
    2019-01-28 23:06:34,291 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', u'connection': 'close', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426/tempest-Object-13806786', u'x-trans-id': 'tx000000000000000049766-005c4f8afa-2ebf7-default', u'date': 'Mon, 28 Jan 2019 23:06:34 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000049766-005c4f8afa-2ebf7-default'}
            Body:
    2019-01-28 23:06:36,307 27593 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 204 DELETE http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426 0.013s
    2019-01-28 23:06:36,307 27593 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestContainer-2138236426', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000003964-005c4f8afc-46c45-default', u'date': 'Mon, 28 Jan 2019 23:06:36 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000003964-005c4f8afc-46c45-default'}
            Body:

Expected results:

Test should pass
Additional info:
Tempest.conf
[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,tag-ext,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,trunk-details

[auth]
tempest_roles = _member_,Member, ResellerAdmin
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = mKgZzguD8aj38E99FnmjkxVpc
admin_project_id = c9fb570856934e5ca84d1f3d1cd2b526

[scenario]
img_dir = etc
img_file = cirros-0.3.5-x86_64-disk.img

[object-storage]
reseller_admin_role = ResellerAdmin
region = regionOne

[oslo-concurrency]
lock_path = /tmp

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
console_output = false
resize = True
attach_encrypted_volume = False
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[identity]
username = demo
password = secrete
project_name = demo
alt_username = alt_demo
alt_password = secrete
alt_project_name = alt_demo
disable_ssl_certificate_validation = true
region = regionOne
uri = http://100.82.36.190:5000//v3
auth_version = v3
uri_v3 = http://100.82.36.190:5000/v3

[image]
image_path = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
region = regionOne
http_image = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img

[compute]
region = regionOne
flavor_ref = 5ca18338-ac66-4cbd-91cf-89f870951de6
flavor_ref_alt = f0c66dfa-e435-4df1-b09c-4688eb5c00c9
image_ref = 5e6bb6f3-d212-4423-8da2-8566abda7cb8
image_ref_alt = b3128550-717b-4f7a-b8dd-749c458dfd7a

[network]
region = regionOne
public_network_id = 008b9378-c481-4c57-8199-858aa67a105d
floating_network_name = public

[orchestration]
stack_owner_role = swiftoperator
region = regionOne

[volume]
backend1_name = tripleo_iscsi
region = regionOne
min_microversion = 3.0
max_microversion = 3.50

[volume-feature-enabled]
bootable = true
backup = False
api_v2 = False
api_v3 = True
api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,os-volume-actions,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats

[object-storage-feature-enabled]
discoverability = False
discoverable_apis = container_quotas
# discoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb

[validation]
image_ssh_user = cirros

[service_available]
ceilometer = True
horizon = True
cinder = True
nova = True
neutron = True
trove = False
glance = True
manila = False
panko = True
ironic = False
mistral = False
heat = True
zaqar = False
swift = True
sahara = False
gnocchi = True
octavia = False
aodh = True
aodh_plugin = True

[dashboard]
dashboard_url = http://100.82.36.190/dashboard/
login_url = http://100.82.36.190/dashboard/auth/login/

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = False
api_v3 = True
api_extensions = s3tokens,OS-EP-FILTER,OS-REVOKE,OS-FEDERATION,OS-INHERIT,OS-SIMPLE-CERT,OS-TRUST,OS-PKI,OS-ENDPOINT-POLICY,OS-OAUTH1,OS-EC2



--------------------------
ceph.conf on controller

--------------------------

[client.rgw.mr-14g-controller-0]
host = mr-14g-controller-0
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-0/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-0.log
rgw frontends = civetweb port=192.168.170.12:8080 num_threads=100

[client.rgw.mr-14g-controller-1]
host = mr-14g-controller-1
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-1/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-1.log
rgw frontends = civetweb port=192.168.170.13:8080 num_threads=100

[client.rgw.mr-14g-controller-2]
host = mr-14g-controller-2
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-2/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-2.log
rgw frontends = civetweb port=192.168.170.14:8080 num_threads=100

# Please do not change this file directly since it is managed by Ansible and will be overwritten
[global]
cluster network = 192.168.180.0/24
fsid = eb28c9a4-1b45-11e9-b81c-5254001e8ca3
journal_collocation = False
journal_size = 10000
# log file = /dev/null
log file = /var/log/ceph/ceph.log
mon cluster log file = /dev/null
mon host = 192.168.170.12,192.168.170.13,192.168.170.14
mon initial members = mr-14g-controller-0,mr-14g-controller-1,mr-14g-controller-2
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128
osd_pool_default_size = 3
public network = 192.168.170.0/24
raw_multi_journal = True
rgw_keystone_admin_domain = default
rgw_keystone_admin_password = FGfWyB4q6xfkM3DtG9RXteRHW
rgw_keystone_admin_project = service
rgw_keystone_admin_user = swift
rgw_keystone_api_version = 3
rgw_keystone_implicit_tenants = true
rgw_keystone_revocation_interval = 0
rgw_keystone_url = http://192.168.140.251:5000
rgw_s3_auth_use_keystone = true
rgw_keystone_accepted_roles = Member, admin, _member_, ResellerAdmin
rgw_swift_enforce_content_length = true
rgw_log_nonexistent_bucket = true
rgw_enable_ops_log = true
debug ms = 1
debug rgw = 20
# Preluminous_compat entry added - Start
mon_health_preluminous_compat=true
# Preluminous_compat entry added - End

Comment 21 Pritha Srivastava 2019-04-02 06:15:20 UTC
Hi David,

Can you please re-run the test with the following additional configs:

In ceph.conf:

rgw swift account in url = true in ceph.conf

and while setting up endpoints for ceph swift, you need to use the following command:
openstack endpoint create --region RegionOne <service-id> public 'http://<hostname>:<portnum>/v1/AUTH_$(project_id)s'   ------(Note AUTH_$(project_id)s

In tempest conf file, set discoverability to True:


[object-storage-feature-enabled]
discoverability = True

The cross tenant tests passed for me with the above mentioned configs in place.

Thanks,
Pritha

Comment 25 David Paterson 2019-04-15 19:31:45 UTC
I'll need to see your tempest.conf as when I set 
[object-storage-feature-enabled]
discoverability = True

I get:
tempest.lib.exceptions.NotFound: Object not found
    Details: {u'HostId': u'19603-default-default', u'Code': u'NoSuchBucket', u'BucketName': u'info', u'RequestId': u'tx0000000000000000001f3-005cb

Comment 26 Pritha Srivastava 2019-04-16 04:09:17 UTC
[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[auth]
tempest_roles = member,Member
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = nova
admin_project_id = 6e73ac1d59204036b195f1883cc7ff4d

[identity]
uri_v3 = http://192.168.122.1/identity/v3/auth/tokens
username=demo
password=nova
project_name = demo
domain_name = Default
auth_version = v3
endpoint_type=publicURL
catalog_type=identity
disable_ssl_certificate_validation = true
region = regionOne

[identity-feature-enabled]
api_v2 = false
api_v3 = true

[object-storage-feature-enabled]
discoverability = True

[network]
floating_network_name = public

[object-storage]
operator_role = SwiftOperator
reseller_admin_role = ResellerAdmin

[oslo_concurrency]
lock_path = /tmp/tempest

[service_available]
cinder = false
neutron = false
glance = false
swift = true
nova = false
heat = false

[validation]
run_validation = true
connect_method = floating
auth_method = keypair

Comment 27 David Paterson 2019-04-16 13:01:03 UTC
Still having same issue, must be something different in ceph.conf. Can you please paste your controller's ceph.conf and I can find the delta.

Thanks,
David

Comment 28 Pritha Srivastava 2019-04-16 15:05:32 UTC
rgw keystone url = http://192.168.122.1/identity/
rgw keystone admin project = admin
rgw keystone admin tenant = admin
rgw keystone admin domain = default
rgw keystone api version = 3
rgw keystone implicit tenants = true
rgw keystone admin password = nova
rgw keystone admin user = admin
rgw keystone accepted roles = SwiftOperator,admin,_member_,ResellerAdmin
rgw keystone accepted admin roles = ResellerAdmin
rgw keystone token cache size = 500
rgw keystone revocation interval = 500
rgw s3 auth use keystone = true
rgw nss db path = /var/ceph/nss
rgw keystone verify ssl = false
rgw swift enforce content length = true
rgw swift versioning enabled = true
rgw swift url prefix = "/"
rgw enable apis = swift, swift_auth, admin
rgw swift account in url = true

Comment 29 Pritha Srivastava 2019-04-16 15:11:35 UTC
Hi David,

Do you have the following config set in ceph.conf?

rgw swift account in url = true

This is needed for the test to pass.

Thanks,
Pritha

Comment 30 David Paterson 2019-04-16 17:35:18 UTC
Yes I do have that set, the problem is when I set:
[object-storage-feature-enabled]
discoverability = True

Tempest attempts a call to <rgw_url>/info, which throws an error.  Think we may have disparate version of ceph rgw,  do your package versions match the ones in my initial bug report?

Comment 31 Pritha Srivastava 2019-04-17 03:43:12 UTC
Hi David,

In order to get /info working, you need to have the following two configs set in ceph.conf. The first one puts swift apis at root (/) and the second one explicitly enables swift and swift related apis (leaving out s3 related apis).

rgw swift url prefix = "/"
rgw enable apis = swift, swift_auth, admin

Please try with these options and let me know.

Thanks,
Pritha

Comment 32 David Paterson 2019-04-17 13:16:39 UTC
Will test your suggestion and verify,  are these config attributes noted anywhere in RHOSP docs? If not it would be useful to add.  Thanks Pritha.

Comment 37 David Paterson 2019-08-30 16:25:04 UTC
I've tried running the tempest config script with object store discoverability turned on and all three controllers having the attributes you mention.  The config script runs but no api discovery is done. I just end up with the following in tempest.conf

[object-storage-feature-enabled]
discoverability = true
discoverable_apis =

Comment 38 David Paterson 2019-08-30 17:57:42 UTC
I have tried all the recommended work arounds and I am still receiving 404 erorrs (BucketNotFound) for many object tests:

tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_write_object_with_rights[id-aa58bfa5-40d9-4bc3-82b4-d07f4a9e392a]
-----------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_container_acl.py", line 88, in test_write_object_with_rights
        object_name, 'data', headers={})
      File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/object_client.py", line 41, in create_object
        resp, body = self.put(url, data, headers, chunked=chunked)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
        return self.request('PUT', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 774, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: NoSuchBucket


Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-08-30 17:40:20,295 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 POST http://100.67.154.160:5000/v3/auth/tokens
    2019-08-30 17:40:20,295 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
            Body: <omitted>
        Response - Headers: {'status': '201', u'content-length': '9712', 'content-location': 'http://100.67.154.160:5000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Fri, 30 Aug 2019 17:40:19 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-66672378-7a35-4272-927b-aaf08905a3ac'}
            Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "7cc1ddc9d55c4fae847f9508bdaf1bfd", "name": "ResellerAdmin"}, {"id": "57505ea313ad43d280103b88cf8f4a19", "name": "Member"}], "expires_at": "2019-08-30T18:40:20.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "36e9c313427e49748af5e6e237406f88", "name": "tempest-ObjectTestACLs-1148683088"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "admin", "id": "59053e38dc1b42ca999348f04f64b79b"}, {"region_id": "regionOne", "url": "http://100.67.154.160:9292", "region": "regionOne", "interface": "public", "id": "729de2bb8a7b4df8bb0e5688e8973836"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "internal", "id": "c95f6e29793e411095e9c9ebdfe3bbd0"}], "type": "image", "id": "08cde5f69e914bf497ea5bad3ff7f5bd", "name": "glance"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.67.154.160:8042", "region": "regionOne", "interface": "public", "id": "3c38df4d3a0d43e79f398f859589c1cb"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8042", "region": "regionOne", "interface": "internal", "id": "94be2b8f8ee34e9c813680e4d5236b42"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8042", "region": "regionOne", "interface": "admin", "id": "d341a3fb000947b393c597ed791bca34"}], "type": "alarming", "id": "10253f25166a4a9587beb4f3b56e82b0", "name": "aodh"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "internal", "id": "0702dd9100f34fd1945d9ea43b17ae81"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "public", "id": "681e036e70da4619b655bf794bc953ea"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8004/v1/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "admin", "id": "abd858865fe14d7aaaab816a731bcd03"}], "type": "orchestration", "id": "11b37d21d85b46e8a41e5d701eea1b93", "name": "heat"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "admin", "id": "47d6896f039147519acdd323f207dc4f"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "internal", "id": "cbe7a31bd5f4495eb167d4bdcd0f6cf6"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8776/v3/36e9c313427e49748af5e6e237406f88", "region": "regionOne", "interface": "public", "id": "f706bde2fa824e7eb2eb590caa3f55a7"}], "type": "volumev3", "id": "156385ed278c4bfeb5c864fa84ed4419", "name": "cinderv3"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.67.154.160:8774/v2.1", "region": "regionOne", "interface": "public", "id": "5666f0a00eaa42f3b96e747aa08c8058"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "92f59ee5c38c43df8c577294976ea584"}, {"region_id": "regionOne", "url": "http://192.168.140.252:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "d4c37580b471438695f3246ca7cfa9c0"}], "type": "compute", "id": "1ecd325cd71d462e80e9a006fb3cfd06", "name": "nova"}, {"endpoints": [], "type": "tempest-type-1214509154", "id": "247d3dfceb6f472496b7c4f3c9702b1b", "name": "tempest-service-1389917785"}, {"endpoints": [], "type": "tempest-type-839988184", "id": "3085e4103d0a4ce486b3e0b4fb2d56d1", "name": "tempest-service-644891461"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.252:8041", "region": "regionOne", "interface": "internal", "id": "0b2717c8444d4df59c10b0d199a63a8c"}, {"region_id": "regionOne", "url": "http://100.67.154.160:8041", "region": "regionOne", "interface": "public", "id": "a18f79b662994a9894209e03fc6d4328"}, {"region_id": "r
    2019-08-30 17:40:20,490 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:setUp): 201 PUT http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.194s
    2019-08-30 17:40:20,490 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '201', u'content-length': '0', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx0000000000000000013d7-005d695f84-13689-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013d7-005d695f84-13689-default'}
            Body:
    2019-08-30 17:40:20,669 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_write_object_with_rights): 204 POST http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.178s
    2019-08-30 17:40:20,669 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Container-Write': u'tempest-ObjectTestACLs-1966151171:tempest-ObjectTestACLs-1966151171', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', u'connection': 'close', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'x-trans-id': 'tx0000000000000000013a9-005d695f84-13b5e-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013a9-005d695f84-13b5e-default'}
            Body:
    2019-08-30 17:40:20,847 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:test_write_object_with_rights): 404 PUT http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920/tempest-Object-291679424 0.176s
    2019-08-30 17:40:20,847 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: data
        Response - Headers: {'status': '404', u'content-length': '12', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920/tempest-Object-291679424', u'accept-ranges': 'bytes', u'last-modified': 'Thu, 01 Jan 1970 00:00:00 GMT', u'connection': 'close', u'x-trans-id': 'tx000000000000000001375-005d695f84-1adaa-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000001375-005d695f84-1adaa-default'}
            Body: NoSuchBucket
    2019-08-30 17:40:20,855 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 200 GET http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920?&limit=9999&format=json 0.004s
    2019-08-30 17:40:20,856 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', u'content-length': '2', u'x-container-object-count': '0', u'x-container-write': 'tempest-ObjectTestACLs-1966151171:tempest-ObjectTestACLs-1966151171', u'accept-ranges': 'bytes', u'x-storage-policy': 'default-placement', u'x-container-bytes-used-actual': '0', u'x-container-bytes-used': '0', u'connection': 'close', u'x-timestamp': '1567186820.48106', u'x-trans-id': 'tx0000000000000000013d8-005d695f84-13689-default', u'date': 'Fri, 30 Aug 2019 17:40:20 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013d8-005d695f84-13689-default', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920?&limit=9999&format=json'}
            Body: []
    2019-08-30 17:40:22,874 7442 INFO     [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 204 DELETE http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920 0.015s
    2019-08-30 17:40:22,875 7442 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', 'content-location': 'http://100.67.154.160:8080/swift/v1/tempest-TestContainer-1876810920', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx0000000000000000013ad-005d695f86-13b5e-default', u'date': 'Fri, 30 Aug 2019 17:40:22 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx0000000000000000013ad-005d695f86-13b5e-default'}
            Body:

Comment 46 errata-xmlrpc 2019-09-21 11:19:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.