Bug 1669214 - RGW - Tempest test: AccountQuotasNegativeTest.test_user_modify_quota fails with 403
Summary: RGW - Tempest test: AccountQuotasNegativeTest.test_user_modify_quota fails wi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: beta
: 15.0 (Stein)
Assignee: Giulio Fidente
QA Contact: Eliad Cohen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-24 15:57 UTC by David Paterson
Modified: 2019-09-26 10:47 UTC (History)
19 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:19:59 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 637815 0 None MERGED Make Ceph RGW accept ResellerAdmin role for administration 2020-12-17 12:58:40 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:20:21 UTC

Description David Paterson 2019-01-24 15:57:00 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
ostestr --regex tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota

Actual results:
tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota[id-d1dc5076-555e-4e6d-9697-28f1fe976324,negative]
----------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_quotas_negative.py", line 56, in setUp
        "POST", url="", headers=headers, body="")
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 769, in _error_checker
        raise exceptions.Forbidden(resp_body, resp=resp)
    tempest.lib.exceptions.Forbidden: Forbidden
    Details: AccessDenied


Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-01-09 23:12:26,639 365 INFO     [tempest.lib.common.rest_client] Request (AccountQuotasNegativeTest:setUp): 403 POST http://100.82.36.190:8080/swift/v1 0.181s
    2019-01-09 23:12:26,639 365 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'X-Account-Meta-Quota-Bytes': '20', 'X-Auth-Token': '<omitted>'}
            Body:
        Response - Headers: {'status': '403', u'content-length': '12', 'content-location': 'http://100.82.36.190:8080/swift/v1', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000000d9b-005c367fda-15789-default', u'date': 'Wed, 09 Jan 2019 23:12:26 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000000d9b-005c367fda-15789-default'}
            Body: AccessDenied

----------------------------------------
civetweb logging in /var/log/messages, unsure if related.

Jan 23 20:08:33 mr-14g-controller-0 journal: 2019-01-23 20:08:33.945361 7f46d00b1700  0 NOTICE: couldn't map swift user 4eba560a393945c3a53460aac8afa515

Expected results:
Tempest test should pass

Additional info:

Tempest.conf
------------------

[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,tag-ext,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,trunk-details

[auth]
tempest_roles = _member_,Member, ResellerAdmin
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = xxxxxxxxxxxxxxxxxx
admin_project_id = c9fb570856934e5ca84d1f3d1cd2b526

[scenario]
img_dir = etc
img_file = cirros-0.3.5-x86_64-disk.img

[object-storage]
reseller_admin_role = ResellerAdmin
region = regionOne

[oslo-concurrency]
lock_path = /tmp

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
console_output = false
resize = True
attach_encrypted_volume = False
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[identity]
username = demo
password = secrete
project_name = demo
alt_username = alt_demo
alt_password = secrete
alt_project_name = alt_demo
disable_ssl_certificate_validation = true
region = regionOne
uri = http://100.82.36.190:5000//v3
auth_version = v3
uri_v3 = http://100.82.36.190:5000/v3

[image]
image_path = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
region = regionOne
http_image = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img

[compute]
region = regionOne
flavor_ref = 5ca18338-ac66-4cbd-91cf-89f870951de6
flavor_ref_alt = f0c66dfa-e435-4df1-b09c-4688eb5c00c9
image_ref = 5e6bb6f3-d212-4423-8da2-8566abda7cb8
image_ref_alt = b3128550-717b-4f7a-b8dd-749c458dfd7a

[network]
region = regionOne
public_network_id = 008b9378-c481-4c57-8199-858aa67a105d
floating_network_name = public

[orchestration]
stack_owner_role = swiftoperator
region = regionOne

[volume]
backend1_name = tripleo_iscsi
region = regionOne
min_microversion = 3.0
max_microversion = 3.50

[volume-feature-enabled]
bootable = true
backup = False
api_v2 = False
api_v3 = True
api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,os-volume-actions,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats

[object-storage-feature-enabled]
discoverability = False
discoverable_apis =

[validation]
image_ssh_user = cirros

[service_available]
ceilometer = True
horizon = True
cinder = True
nova = True
neutron = True
trove = False
glance = True
manila = False
panko = True
ironic = False
mistral = False
heat = True
zaqar = False
swift = True
sahara = False
gnocchi = True
octavia = False
aodh = True
aodh_plugin = True

[dashboard]
dashboard_url = http://100.82.36.190/dashboard/
login_url = http://100.82.36.190/dashboard/auth/login/

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = False
api_v3 = True
api_extensions = s3tokens,OS-EP-FILTER,OS-REVOKE,OS-FEDERATION,OS-INHERIT,OS-SIMPLE-CERT,OS-TRUST,OS-PKI,OS-ENDPOINT-POLICY,OS-OAUTH1,OS-EC2
------------------------------

ceph.conf on controllers

[client.rgw.mr-14g-controller-0]
host = mr-14g-controller-0
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-0/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-0.log
rgw frontends = civetweb port=192.168.170.12:8080 num_threads=100

[client.rgw.mr-14g-controller-1]
host = mr-14g-controller-1
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-1/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-1.log
rgw frontends = civetweb port=192.168.170.13:8080 num_threads=100

[client.rgw.mr-14g-controller-2]
host = mr-14g-controller-2
keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-2/keyring
log file = /var/log/ceph/ceph-rgw-mr-14g-controller-2.log
rgw frontends = civetweb port=192.168.170.14:8080 num_threads=100

# Please do not change this file directly since it is managed by Ansible and will be overwritten
[global]
cluster network = 192.168.180.0/24
fsid = eb28c9a4-1b45-11e9-b81c-5254001e8ca3
journal_collocation = False
journal_size = 10000
log file = /dev/null
# log file = /var/log/ceph/ceph.log
mon cluster log file = /dev/null
mon host = 192.168.170.12,192.168.170.13,192.168.170.14
mon initial members = mr-14g-controller-0,mr-14g-controller-1,mr-14g-controller-2
osd_pool_default_pg_num = 128
osd_pool_default_pgp_num = 128
osd_pool_default_size = 3
public network = 192.168.170.0/24
raw_multi_journal = True
rgw_keystone_admin_domain = default
rgw_keystone_admin_password = FGfWyB4q6xfkM3DtG9RXteRHW
rgw_keystone_admin_project = service
rgw_keystone_admin_user = swift
rgw_keystone_api_version = 3
rgw_keystone_implicit_tenants = true
rgw_keystone_revocation_interval = 0
rgw_keystone_url = http://192.168.140.251:5000
rgw_s3_auth_use_keystone = true
rgw_keystone_accepted_roles = Member, admin, _member_, ResellerAdmin
rgw_swift_enforce_content_length = true
rgw_log_nonexistent_bucket = true
rgw_enable_ops_log = true
debug ms = 1
debug rgw = 20
# Preluminous_compat entry added - Start
mon_health_preluminous_compat=true
# Preluminous_compat entry added - End

Also: it is very difficult to know where to look for the actual exception stacktrace to find the root cause.

Comment 1 David Paterson 2019-01-25 20:49:50 UTC
Reproduced error with swift client as well, the user has Member role and created a container prior to attempting to set quota, see below:

swift --debug post -H "X-Account-Meta-Quota-Bytes: 20"

INFO:swiftclient:REQ: curl -i http://100.82.36.190:8080/swift/v1 -X POST -H "X-Account-Meta-Quota-Bytes: 20" -H "X-Auth-Token: gAAAAABcS3IWIRv1Z8q_F0wBKh9Ep98Cr2RdlW57gU6y0TDVFuAqSrX9WCPAopoovpY2XE6nvoQ-EsKuogmJnK6ARgukXvC_T3gcqiGNMVxg9BVP7q3z-pTwY6usuQzC4eC-9g_mDtMt-JAFfzSMR-8hWa5_T-24YFVDsaX4THItYxoFbLMjFVE"
INFO:swiftclient:RESP STATUS: 403 Forbidden
INFO:swiftclient:RESP HEADERS: {u'Content-Length': u'12', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default', u'Date': u'Fri, 25 Jan 2019 20:31:18 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default'}
INFO:swiftclient:RESP BODY: AccessDenied
ERROR:swiftclient.service:Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden   AccessDenied
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 685, in post
    get_future_result(post)
  File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 230, in get_future_result
    res = f.result(timeout=timeout)
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/multithreading.py", line 187, in conn_fn
    return fn(*conn_args, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 813, in _post_account_job
    return conn.post_account(headers=headers, response_dict=result)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1749, in post_account
    response_dict=response_dict)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 861, in post_account
    raise ClientException.from_response(resp, 'Account POST failed', body)
ClientException: Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden   AccessDenied
Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden   AccessDenied
Failed Transaction ID: tx000000000000000006a59-005c4b7216-2eaed-default

Comment 2 David Paterson 2019-01-28 23:11:57 UTC
RefStack and installed package versions:

RefStack current object store policy: wget "https://refstack.openstack.org/api/v1/guidelines/2018.11/tests?target=object&type=required&alias=true&flag=false" -O 2018.11-test-list.txt

RGW packages on controller
librgw2.x86_64                      2:12.2.4-42.el7    @rhos-13.0-signed
python-rgw.x86_64                   2:12.2.4-42.el7    @rhos-13.0-signed
librados2.x86_64                    2:12.2.4-42.el7    @rhos-13.0-signed
libradosstriper1.x86_64             2:12.2.4-42.el7    @rhos-13.0-signed
python-rados.x86_64                 2:12.2.4-42.el7    @rhos-13.0-signed

Comment 3 Giulio Fidente 2019-01-30 14:34:14 UTC
Can you confirm the user also had ResellerAdmin role set?

Comment 4 David Paterson 2019-01-30 15:35:47 UTC
Yes tempest.conf has ResellerAdmin role assignment for all users created. 
tempest_roles = _member_,Member, ResellerAdmin

Comment 13 Pritha Srivastava 2019-02-12 15:33:34 UTC
So the final changes in the conf files are:

In rgw conf file:

rgw keystone accepted admin roles = ResellerAdmin

And in tempest.conf file:

[auth]
tempest_roles = member,Member

[object-storage]
reseller_admin_role = ResellerAdmin

@Matt - can we ask him to re-test with these changes?

Comment 14 Matt Benjamin (redhat) 2019-02-14 12:41:11 UTC
Hi David, could you review the suggestion in comment #13?

thanks,

Matt

Comment 15 David Paterson 2019-02-14 14:16:58 UTC
Yes that worked with one additional change in tempest.conf you must have [object-storage-feature-enabled].discoverable_apis at least include "account_quotas" or test is skipped.

In summary:
tempest.conf

[auth]
tempest_roles = member,Member

[object-storage]
reseller_admin_role = ResellerAdmin

[object-storage-feature-enabled]
discoverable_apis = account_quotas

ceph.conf on controller(s)
rgw_keystone_accepted_admin_roles = ResellerAdmin

Thanks!

Comment 23 errata-xmlrpc 2019-09-21 11:19:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.