Bug 1670227

Summary: RGW - Tempest test: test_object_version.ContainerTest.test_versioned_container fails with 412 PreconditionFailed
Product: Red Hat OpenStack Reporter: David Paterson <david_paterson>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Eliad Cohen <elicohen>
Severity: low Docs Contact:
Priority: low    
Version: 13.0 (Queens)CC: arkady_kanevsky, cdevine, chadd, christopher_dearborn, cschwede, dang, david_paterson, dcain, elicohen, gael_rehault, gfidente, jdurgin, kurt_hey, lhh, mbenjamin, mburns, morazi, nweinber, rajini.karthik
Target Milestone: betaKeywords: Reopened, Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1807772 (view as bug list) Environment:
Last Closed: 2019-09-21 11:19:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1807772    

Description David Paterson 2019-01-29 00:06:15 UTC
Description of problem:
When trying to run RefStack object store certification tests against RGW I am seeing the following failure.

Version-Release number of selected component (if applicable):
RefStack current object store policy: wget "https://refstack.openstack.org/api/v1/guidelines/2018.11/tests?target=object&type=required&alias=true&flag=false" -O 2018.11-test-list.txt

RGW packages on controller
librgw2.x86_64                      2:12.2.4-42.el7    @rhos-13.0-signed
python-rgw.x86_64                   2:12.2.4-42.el7    @rhos-13.0-signed
librados2.x86_64                    2:12.2.4-42.el7    @rhos-13.0-signed
libradosstriper1.x86_64             2:12.2.4-42.el7    @rhos-13.0-signed
python-rados.x86_64                 2:12.2.4-42.el7    @rhos-13.0-signed

How reproducible:
Every time

Steps to Reproduce:
1.run
ostests --regex tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
2.
3.

Actual results:
Test fails, output from tempest:


tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container[id-a151e158-dcbf-4a1f-a1e7-46cd65895a6f]
------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_object_version.py", line 56, in test_versioned_container
        **headers)
      File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/container_client.py", line 37, in update_container
        resp, body = self.put(url, body=None, headers=headers)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
        return self.request('PUT', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 796, in _error_checker
        raise exceptions.PreconditionFailed(resp_body, resp=resp)
    tempest.lib.exceptions.PreconditionFailed: Precondition Failed
    Details: PreconditionFailed


Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-01-29 00:03:29,963 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 POST http://100.82.36.190:5000/v3/auth/tokens
    2019-01-29 00:03:29,963 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
            Body: <omitted>
        Response - Headers: {'status': '201', u'content-length': '9429', 'content-location': 'http://100.82.36.190:5000/v3/auth/tokens', u'x-subject-token': '<omitted>', u'vary': 'X-Auth-Token', u'server': 'Apache', u'connection': 'close', u'date': 'Tue, 29 Jan 2019 00:03:29 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-ac407ed3-5a30-405f-b73d-e6e7a39c0edb'}
            Body: {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5fe6a7b1273a413c9dfa1655870fe40d", "name": "Member"}, {"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "c268566a511a4281bf1c2de2e9501e5a", "name": "ResellerAdmin"}], "expires_at": "2019-01-29T01:03:29.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "c4df108a65a44eb387b2dc5a6d858733", "name": "tempest-ContainerTest-1846919605"}, "catalog": [{"endpoints": [{"region_id": "regionOne", "url": "http://192.168.170.251:8080/swift/v1", "region": "regionOne", "interface": "admin", "id": "1a3912c8c597443f973bc8ac5bff0f45"}, {"region_id": "regionOne", "url": "http://192.168.170.251:8080/swift/v1", "region": "regionOne", "interface": "internal", "id": "1c51111e9da5433ebeb28589eab41d95"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8080/swift/v1", "region": "regionOne", "interface": "public", "id": "ac2de181ee9e43eb85490680d1b37cce"}], "type": "object-store", "id": "015abd8e050743f8b5575c8c0cf2be1d", "name": "swift"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:8776/v3/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "internal", "id": "a024460483ef4b268bff389570f9fd81"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8776/v3/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "public", "id": "e925b62bddf74e09a8c3881bab1645ee"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v3/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "admin", "id": "e9957aefc2784d8b815bcb16df978fd2"}], "type": "volumev3", "id": "0448310d75fe4313b748e295f4d4eda9", "name": "cinderv3"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.82.36.190:9292", "region": "regionOne", "interface": "public", "id": "35b49532f7604d73b7847b32f6ef024b"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "internal", "id": "7ea8645c6c1b4fd6a4ef06a12c54820d"}, {"region_id": "regionOne", "url": "http://192.168.170.251:9292", "region": "regionOne", "interface": "admin", "id": "896a29a6442b4759b76c0083f53bd19f"}], "type": "image", "id": "0baf8ffd467345b38fb798784b05c8b7", "name": "glance"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:9696", "region": "regionOne", "interface": "internal", "id": "1d6782d296964b69942bf725dca7c99c"}, {"region_id": "regionOne", "url": "http://100.82.36.190:9696", "region": "regionOne", "interface": "public", "id": "30c0262501244369af7422d606bfed10"}, {"region_id": "regionOne", "url": "http://192.168.140.251:9696", "region": "regionOne", "interface": "admin", "id": "33b2e1df3b1148b6a419190f9386b5ac"}], "type": "network", "id": "24cd4bd5384449b183e44edc48ae93a0", "name": "neutron"}, {"endpoints": [{"region_id": "regionOne", "url": "http://192.168.140.251:8774/v2.1", "region": "regionOne", "interface": "admin", "id": "2774048e10614ea7bafd5eef8494ce9b"}, {"region_id": "regionOne", "url": "http://100.82.36.190:8774/v2.1", "region": "regionOne", "interface": "public", "id": "4272634957b348c88cd21404fcfc0b8e"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8774/v2.1", "region": "regionOne", "interface": "internal", "id": "7cbeece3b6fb425b871866ca032462b4"}], "type": "compute", "id": "556e411d989545feb396de183b608f93", "name": "nova"}, {"endpoints": [{"region_id": "regionOne", "url": "http://100.82.36.190:8776/v2/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "public", "id": "1bf9d9cd666040a791a0c70a1a55430f"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v2/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "admin", "id": "5155f74571864d158359bc1292cd66de"}, {"region_id": "regionOne", "url": "http://192.168.140.251:8776/v2/c4df108a65a44eb387b2dc5a6d858733", "region": "regionOne", "interface": "internal", "id": "6b53208fd40e409690812451c0cbc5e8"}], "type": "volumev2", "id": "5fa5997beb5a4621910eca7be8654798", "name": "cinderv2"}, {"endpoints": [{"region
    2019-01-29 00:03:30,179 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 PUT http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957 0.215s
    2019-01-29 00:03:30,179 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '201', u'content-length': '0', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx00000000000000004ab72-005c4f9852-2ebf7-default', u'date': 'Tue, 29 Jan 2019 00:03:30 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx00000000000000004ab72-005c4f9852-2ebf7-default'}
            Body:
    2019-01-29 00:03:30,361 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 204 HEAD http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957 0.182s
    2019-01-29 00:03:30,362 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', u'x-container-object-count': '0', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957', u'accept-ranges': 'bytes', u'x-storage-policy': 'default-placement', u'x-container-bytes-used-actual': '0', u'x-container-bytes-used': '0', u'connection': 'close', u'x-timestamp': '1548720210.21554', u'x-trans-id': 'tx000000000000000004d6d-005c4f9852-46c45-default', u'date': 'Tue, 29 Jan 2019 00:03:30 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000004d6d-005c4f9852-46c45-default'}
            Body:
    2019-01-29 00:03:30,598 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 412 PUT http://100.82.36.190:8080/swift/v1/tempest-TestBaseContainer-702802669 0.236s
    2019-01-29 00:03:30,598 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-versions-Location': 'tempest-TestVersionContainer-1847305957', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '412', u'content-length': '18', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestBaseContainer-702802669', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx00000000000000004ab54-005c4f9852-2ec00-default', u'date': 'Tue, 29 Jan 2019 00:03:30 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx00000000000000004ab54-005c4f9852-2ec00-default'}
            Body: PreconditionFailed
    2019-01-29 00:03:30,602 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:_run_cleanups): 200 GET http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957?&limit=9999&format=json 0.003s
    2019-01-29 00:03:30,602 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '200', u'content-length': '2', u'x-container-object-count': '0', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957?&limit=9999&format=json', u'accept-ranges': 'bytes', u'x-storage-policy': 'default-placement', u'x-container-bytes-used-actual': '0', u'x-container-bytes-used': '0', u'connection': 'close', u'x-timestamp': '1548720210.21554', u'x-trans-id': 'tx00000000000000004ab73-005c4f9852-2ebf7-default', u'date': 'Tue, 29 Jan 2019 00:03:30 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx00000000000000004ab73-005c4f9852-2ebf7-default'}
            Body: []
    2019-01-29 00:03:32,618 31300 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:_run_cleanups): 204 DELETE http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957 0.013s
    2019-01-29 00:03:32,619 31300 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: None
        Response - Headers: {'status': '204', 'content-location': 'http://100.82.36.190:8080/swift/v1/tempest-TestVersionContainer-1847305957', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000004d71-005c4f9854-46c45-default', u'date': 'Tue, 29 Jan 2019 00:03:32 GMT', u'content-type': 'application/json; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000004d71-005c4f9854-46c45-default'}
            Body:


Expected results:
Test should pass

Additional info:
Tempest.conf
[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,tag-ext,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,trunk-details

[auth]
tempest_roles = _member_,Member, ResellerAdmin
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = mKgZzguD8aj38E99FnmjkxVpc
admin_project_id = c9fb570856934e5ca84d1f3d1cd2b526

[scenario]
img_dir = etc
img_file = cirros-0.3.5-x86_64-disk.img

[object-storage]
reseller_admin_role = ResellerAdmin
region = regionOne

[oslo-concurrency]
lock_path = /tmp

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
console_output = false
resize = True
attach_encrypted_volume = False
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[identity]
username = demo
password = secrete
project_name = demo
alt_username = alt_demo
alt_password = secrete
alt_project_name = alt_demo
disable_ssl_certificate_validation = true
region = regionOne
uri = http://100.82.36.190:5000//v3
auth_version = v3
uri_v3 = http://100.82.36.190:5000/v3

[image]
image_path = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
region = regionOne
http_image = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img

[compute]
region = regionOne
flavor_ref = 5ca18338-ac66-4cbd-91cf-89f870951de6
flavor_ref_alt = f0c66dfa-e435-4df1-b09c-4688eb5c00c9
image_ref = 5e6bb6f3-d212-4423-8da2-8566abda7cb8
image_ref_alt = b3128550-717b-4f7a-b8dd-749c458dfd7a

[network]
region = regionOne
public_network_id = 008b9378-c481-4c57-8199-858aa67a105d
floating_network_name = public

[orchestration]
stack_owner_role = swiftoperator
region = regionOne

[volume]
backend1_name = tripleo_iscsi
region = regionOne
min_microversion = 3.0
max_microversion = 3.50

[volume-feature-enabled]
bootable = true
backup = False
api_v2 = False
api_v3 = True
api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,os-volume-actions,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats

[object-storage-feature-enabled]
discoverability = False
discoverable_apis = container_quotas
# discoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb

[validation]
image_ssh_user = cirros

[service_available]
ceilometer = True
horizon = True
cinder = True
nova = True
neutron = True
trove = False
glance = True
manila = False
panko = True
ironic = False
mistral = False
heat = True
zaqar = False
swift = True
sahara = False
gnocchi = True
octavia = False
aodh = True
aodh_plugin = True

[dashboard]
dashboard_url = http://100.82.36.190/dashboard/
login_url = http://100.82.36.190/dashboard/auth/login/

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = False
api_v3 = True
api_extensions = s3tokens,OS-EP-FILTER,OS-REVOKE,OS-FEDERATION,OS-INHERIT,OS-SIMPLE-CERT,OS-TRUST,OS-PKI,OS-ENDPOINT-POLICY,OS-OAUTH1,OS-EC2



--------------------------
ceph.conf on controller

--------------------------

[client.rgw.mr-14g-controller-0]
host = mr-14g-controller-0
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-0/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-0.log
rgw frontends = civetweb port=192.168.170.12:8080 num_threads=100

[client.rgw.mr-14g-controller-1]
host = mr-14g-controller-1
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-1/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-1.log
rgw frontends = civetweb port=192.168.170.13:8080 num_threads=100

[client.rgw.mr-14g-controller-2]
host = mr-14g-controller-2
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-2/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-2.log
rgw frontends = civetweb port=192.168.170.14:8080 num_threads=100

# Please do not change this file directly since it is managed by Ansible and will be overwritten
[global]
cluster network = 192.168.180.0/24
fsid = eb28c9a4-1b45-11e9-b81c-5254001e8ca3
journal_collocation = False
journal_size = 10000
# log file = /dev/null
log file = /var/log/ceph/ceph.log
mon cluster log file = /dev/null
mon host = 192.168.170.12,192.168.170.13,192.168.170.14
mon initial members = mr-14g-controller-0,mr-14g-controller-1,mr-14g-controller-2
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128
osd_pool_default_size = 3
public network = 192.168.170.0/24
raw_multi_journal = True
rgw_keystone_admin_domain = default
rgw_keystone_admin_password = FGfWyB4q6xfkM3DtG9RXteRHW
rgw_keystone_admin_project = service
rgw_keystone_admin_user = swift
rgw_keystone_api_version = 3
rgw_keystone_implicit_tenants = true
rgw_keystone_revocation_interval = 0
rgw_keystone_url = http://192.168.140.251:5000
rgw_s3_auth_use_keystone = true
rgw_keystone_accepted_roles = Member, admin, _member_, ResellerAdmin
rgw_swift_enforce_content_length = true
rgw_log_nonexistent_bucket = true
rgw_enable_ops_log = true
debug ms = 1
debug rgw = 20
# Preluminous_compat entry added - Start
mon_health_preluminous_compat=true
# Preluminous_compat entry added - End

Comment 2 Pritha Srivastava 2019-02-11 07:13:22 UTC
From the logs, it looks like fetching an auth token from the Identity Service is failing with error 201:

201 POST http://100.82.36.190:5000/v3/auth/tokens

Can you please check if Keystone is responding to auth token requests, using the username and password that is there in the identity section of tempest config using:

curl -i -H "Content-Type: application/json" -d '{ "auth": {"identity": {"methods": ["password"],"password": {"user": {"name": "demo","domain": { "id": "demo" },"password": "secrete"}}}}}' "http://100.82.36.190:5000/v3/auth/tokens" ; echo

Comment 6 Pritha Srivastava 2019-02-11 11:06:10 UTC
So I figured out that since this is a object versioning test, we need to have that enabled in rgw using the following config:

rgw swift versioning enabled = true

Else, the test fails with the same error for me.

Comment 7 Matt Benjamin (redhat) 2019-02-11 13:23:13 UTC
(In reply to Pritha Srivastava from comment #6)
> So I figured out that since this is a object versioning test, we need to
> have that enabled in rgw using the following config:
> 
> rgw swift versioning enabled = true
> 
> Else, the test fails with the same error for me.

Great work, Pritha.

It sounds as if, with confirmation, this issue is NotABug.  At the same time, might it make sense to make rgw_swift_versioning_enabled default to true in the configuration?

Matt

Comment 10 Daniel Gryniewicz 2019-02-11 16:37:53 UTC
Pritha is correct, having versioning enabled dosen't do anything unless X-Versions-Location is provided.  However, if that *is* provided, and versioning is disabled, we return ERR_PRECONDITION_FAILED, so clients may use that to probe for version support.  If so, then defaulting it to on will have those clients suddenly start using it.  I don't have a strong opinion either way, but it generally seems like a useful feature.  On the other hand, it seems (it's hard to tell from the docs) to be off by default in OpenStack, so maybe it should be off by default in RGW as well.

Comment 11 Matt Benjamin (redhat) 2019-02-11 18:14:04 UTC
Off by default in OpenStack seems to argue for leaving it alone?

@david, can you post an update here when you've had a chance to retest, and with your preference for rgw default enablement of versioning (currently off)?

thanks!

Matt

Comment 12 David Paterson 2019-02-11 21:33:57 UTC
Yup that seems to work, rgw_swift_versioning_enabled appears to be false by default.

From: http://docs.ceph.com/docs/jewel/radosgw/config-ref/
rgw swift versioning enabled

Description:	Enables the Object Versioning of OpenStack Object Storage API. This allows clients to put the X-Versions-Location attribute on containers that should be versioned. The attribute specifies the name of container storing archived versions. It must be owned by the same user that the versioned container due to access control verification - ACLs are NOT taken into consideration. Those containers cannot be versioned by the S3 object versioning mechanism.
Type:	Boolean
Default:	false

--------------------------------------
I changed it to true on each controller in /etc/ceph/ceph.conf and restarted the rgw service and the test now passes.

ceph.conf
---------
rgw_swift_versioning_enabled = true

restart service
---------------
sudo systemctl restart ceph-radosgw.service

Comment 20 errata-xmlrpc 2019-09-21 11:19:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811