Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: ostestr --regex tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota Actual results: tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota[id-d1dc5076-555e-4e6d-9697-28f1fe976324,negative] ---------------------------------------------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_quotas_negative.py", line 56, in setUp "POST", url="", headers=headers, body="") File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request self._error_checker(resp, resp_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 769, in _error_checker raise exceptions.Forbidden(resp_body, resp=resp) tempest.lib.exceptions.Forbidden: Forbidden Details: AccessDenied Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2019-01-09 23:12:26,639 365 INFO [tempest.lib.common.rest_client] Request (AccountQuotasNegativeTest:setUp): 403 POST http://100.82.36.190:8080/swift/v1 0.181s 2019-01-09 23:12:26,639 365 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Account-Meta-Quota-Bytes': '20', 'X-Auth-Token': '<omitted>'} Body: Response - Headers: {'status': '403', u'content-length': '12', 'content-location': 'http://100.82.36.190:8080/swift/v1', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000000d9b-005c367fda-15789-default', u'date': 'Wed, 09 Jan 2019 23:12:26 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000000d9b-005c367fda-15789-default'} Body: AccessDenied ---------------------------------------- civetweb logging in /var/log/messages, unsure if related. Jan 23 20:08:33 mr-14g-controller-0 journal: 2019-01-23 20:08:33.945361 7f46d00b1700 0 NOTICE: couldn't map swift user 4eba560a393945c3a53460aac8afa515 Expected results: Tempest test should pass Additional info: Tempest.conf ------------------ [DEFAULT] debug = true use_stderr = false log_file = tempest.log [network-feature-enabled] ipv6_subnet_attributes = true api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,tag-ext,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,trunk-details [auth] tempest_roles = _member_,Member, ResellerAdmin admin_username = admin admin_project_name = admin admin_domain_name = Default use_dynamic_credentials = true admin_password = xxxxxxxxxxxxxxxxxx admin_project_id = c9fb570856934e5ca84d1f3d1cd2b526 [scenario] img_dir = etc img_file = cirros-0.3.5-x86_64-disk.img [object-storage] reseller_admin_role = ResellerAdmin region = regionOne [oslo-concurrency] lock_path = /tmp [compute-feature-enabled] live_migration = false live_migrate_paused_instances = true preserve_ports = true console_output = false resize = True attach_encrypted_volume = False api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes [identity] username = demo password = secrete project_name = demo alt_username = alt_demo alt_password = secrete alt_project_name = alt_demo disable_ssl_certificate_validation = true region = regionOne uri = http://100.82.36.190:5000//v3 auth_version = v3 uri_v3 = http://100.82.36.190:5000/v3 [image] image_path = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img region = regionOne http_image = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img [compute] region = regionOne flavor_ref = 5ca18338-ac66-4cbd-91cf-89f870951de6 flavor_ref_alt = f0c66dfa-e435-4df1-b09c-4688eb5c00c9 image_ref = 5e6bb6f3-d212-4423-8da2-8566abda7cb8 image_ref_alt = b3128550-717b-4f7a-b8dd-749c458dfd7a [network] region = regionOne public_network_id = 008b9378-c481-4c57-8199-858aa67a105d floating_network_name = public [orchestration] stack_owner_role = swiftoperator region = regionOne [volume] backend1_name = tripleo_iscsi region = regionOne min_microversion = 3.0 max_microversion = 3.50 [volume-feature-enabled] bootable = true backup = False api_v2 = False api_v3 = True api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,os-volume-actions,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats [object-storage-feature-enabled] discoverability = False discoverable_apis = [validation] image_ssh_user = cirros [service_available] ceilometer = True horizon = True cinder = True nova = True neutron = True trove = False glance = True manila = False panko = True ironic = False mistral = False heat = True zaqar = False swift = True sahara = False gnocchi = True octavia = False aodh = True aodh_plugin = True [dashboard] dashboard_url = http://100.82.36.190/dashboard/ login_url = http://100.82.36.190/dashboard/auth/login/ [image-feature-enabled] api_v1 = False api_v2 = True [identity-feature-enabled] api_v2 = False api_v3 = True api_extensions = s3tokens,OS-EP-FILTER,OS-REVOKE,OS-FEDERATION,OS-INHERIT,OS-SIMPLE-CERT,OS-TRUST,OS-PKI,OS-ENDPOINT-POLICY,OS-OAUTH1,OS-EC2 ------------------------------ ceph.conf on controllers [client.rgw.mr-14g-controller-0] host = mr-14g-controller-0 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-0/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-0.log rgw frontends = civetweb port=192.168.170.12:8080 num_threads=100 [client.rgw.mr-14g-controller-1] host = mr-14g-controller-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-1/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-1.log rgw frontends = civetweb port=192.168.170.13:8080 num_threads=100 [client.rgw.mr-14g-controller-2] host = mr-14g-controller-2 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-2/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-2.log rgw frontends = civetweb port=192.168.170.14:8080 num_threads=100 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.180.0/24 fsid = eb28c9a4-1b45-11e9-b81c-5254001e8ca3 journal_collocation = False journal_size = 10000 log file = /dev/null # log file = /var/log/ceph/ceph.log mon cluster log file = /dev/null mon host = 192.168.170.12,192.168.170.13,192.168.170.14 mon initial members = mr-14g-controller-0,mr-14g-controller-1,mr-14g-controller-2 osd_pool_default_pg_num = 128 osd_pool_default_pgp_num = 128 osd_pool_default_size = 3 public network = 192.168.170.0/24 raw_multi_journal = True rgw_keystone_admin_domain = default rgw_keystone_admin_password = FGfWyB4q6xfkM3DtG9RXteRHW rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_api_version = 3 rgw_keystone_implicit_tenants = true rgw_keystone_revocation_interval = 0 rgw_keystone_url = http://192.168.140.251:5000 rgw_s3_auth_use_keystone = true rgw_keystone_accepted_roles = Member, admin, _member_, ResellerAdmin rgw_swift_enforce_content_length = true rgw_log_nonexistent_bucket = true rgw_enable_ops_log = true debug ms = 1 debug rgw = 20 # Preluminous_compat entry added - Start mon_health_preluminous_compat=true # Preluminous_compat entry added - End Also: it is very difficult to know where to look for the actual exception stacktrace to find the root cause.
Reproduced error with swift client as well, the user has Member role and created a container prior to attempting to set quota, see below: swift --debug post -H "X-Account-Meta-Quota-Bytes: 20" INFO:swiftclient:REQ: curl -i http://100.82.36.190:8080/swift/v1 -X POST -H "X-Account-Meta-Quota-Bytes: 20" -H "X-Auth-Token: gAAAAABcS3IWIRv1Z8q_F0wBKh9Ep98Cr2RdlW57gU6y0TDVFuAqSrX9WCPAopoovpY2XE6nvoQ-EsKuogmJnK6ARgukXvC_T3gcqiGNMVxg9BVP7q3z-pTwY6usuQzC4eC-9g_mDtMt-JAFfzSMR-8hWa5_T-24YFVDsaX4THItYxoFbLMjFVE" INFO:swiftclient:RESP STATUS: 403 Forbidden INFO:swiftclient:RESP HEADERS: {u'Content-Length': u'12', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default', u'Date': u'Fri, 25 Jan 2019 20:31:18 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default'} INFO:swiftclient:RESP BODY: AccessDenied ERROR:swiftclient.service:Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 685, in post get_future_result(post) File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 230, in get_future_result res = f.result(timeout=timeout) File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/multithreading.py", line 187, in conn_fn return fn(*conn_args, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 813, in _post_account_job return conn.post_account(headers=headers, response_dict=result) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1749, in post_account response_dict=response_dict) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 861, in post_account raise ClientException.from_response(resp, 'Account POST failed', body) ClientException: Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Failed Transaction ID: tx000000000000000006a59-005c4b7216-2eaed-default
RefStack and installed package versions: RefStack current object store policy: wget "https://refstack.openstack.org/api/v1/guidelines/2018.11/tests?target=object&type=required&alias=true&flag=false" -O 2018.11-test-list.txt RGW packages on controller librgw2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rgw.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed librados2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed libradosstriper1.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rados.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed
Can you confirm the user also had ResellerAdmin role set?
Yes tempest.conf has ResellerAdmin role assignment for all users created. tempest_roles = _member_,Member, ResellerAdmin
So the final changes in the conf files are: In rgw conf file: rgw keystone accepted admin roles = ResellerAdmin And in tempest.conf file: [auth] tempest_roles = member,Member [object-storage] reseller_admin_role = ResellerAdmin @Matt - can we ask him to re-test with these changes?
Hi David, could you review the suggestion in comment #13? thanks, Matt
Yes that worked with one additional change in tempest.conf you must have [object-storage-feature-enabled].discoverable_apis at least include "account_quotas" or test is skipped. In summary: tempest.conf [auth] tempest_roles = member,Member [object-storage] reseller_admin_role = ResellerAdmin [object-storage-feature-enabled] discoverable_apis = account_quotas ceph.conf on controller(s) rgw_keystone_accepted_admin_roles = ResellerAdmin Thanks!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811