Bug 1669214
| Summary: | RGW - Tempest test: AccountQuotasNegativeTest.test_user_modify_quota fails with 403 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Paterson <david_paterson> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Giulio Fidente <gfidente> |
| Status: | CLOSED ERRATA | QA Contact: | Eliad Cohen <elicohen> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 13.0 (Queens) | CC: | arkady_kanevsky, cdevine, chadd, christopher_dearborn, david_paterson, dcain, elicohen, gael_rehault, gfidente, jdurgin, kurt_hey, lhh, mbenjamin, mburns, morazi, nweinber, prsrivas, rajini.karthik, smanjara |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-10.5.1-0.20190701110422.889d4d4.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:19:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Reproduced error with swift client as well, the user has Member role and created a container prior to attempting to set quota, see below: swift --debug post -H "X-Account-Meta-Quota-Bytes: 20" INFO:swiftclient:REQ: curl -i http://100.82.36.190:8080/swift/v1 -X POST -H "X-Account-Meta-Quota-Bytes: 20" -H "X-Auth-Token: gAAAAABcS3IWIRv1Z8q_F0wBKh9Ep98Cr2RdlW57gU6y0TDVFuAqSrX9WCPAopoovpY2XE6nvoQ-EsKuogmJnK6ARgukXvC_T3gcqiGNMVxg9BVP7q3z-pTwY6usuQzC4eC-9g_mDtMt-JAFfzSMR-8hWa5_T-24YFVDsaX4THItYxoFbLMjFVE" INFO:swiftclient:RESP STATUS: 403 Forbidden INFO:swiftclient:RESP HEADERS: {u'Content-Length': u'12', u'Accept-Ranges': u'bytes', u'X-Trans-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default', u'Date': u'Fri, 25 Jan 2019 20:31:18 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000006a59-005c4b7216-2eaed-default'} INFO:swiftclient:RESP BODY: AccessDenied ERROR:swiftclient.service:Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 685, in post get_future_result(post) File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 230, in get_future_result res = f.result(timeout=timeout) File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/multithreading.py", line 187, in conn_fn return fn(*conn_args, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 813, in _post_account_job return conn.post_account(headers=headers, response_dict=result) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1749, in post_account response_dict=response_dict) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 861, in post_account raise ClientException.from_response(resp, 'Account POST failed', body) ClientException: Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Account POST failed: http://100.82.36.190:8080/swift/v1 403 Forbidden AccessDenied Failed Transaction ID: tx000000000000000006a59-005c4b7216-2eaed-default RefStack and installed package versions: RefStack current object store policy: wget "https://refstack.openstack.org/api/v1/guidelines/2018.11/tests?target=object&type=required&alias=true&flag=false" -O 2018.11-test-list.txt RGW packages on controller librgw2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rgw.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed librados2.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed libradosstriper1.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed python-rados.x86_64 2:12.2.4-42.el7 @rhos-13.0-signed Can you confirm the user also had ResellerAdmin role set? Yes tempest.conf has ResellerAdmin role assignment for all users created. tempest_roles = _member_,Member, ResellerAdmin So the final changes in the conf files are: In rgw conf file: rgw keystone accepted admin roles = ResellerAdmin And in tempest.conf file: [auth] tempest_roles = member,Member [object-storage] reseller_admin_role = ResellerAdmin @Matt - can we ask him to re-test with these changes? Hi David, could you review the suggestion in comment #13? thanks, Matt Yes that worked with one additional change in tempest.conf you must have [object-storage-feature-enabled].discoverable_apis at least include "account_quotas" or test is skipped. In summary: tempest.conf [auth] tempest_roles = member,Member [object-storage] reseller_admin_role = ResellerAdmin [object-storage-feature-enabled] discoverable_apis = account_quotas ceph.conf on controller(s) rgw_keystone_accepted_admin_roles = ResellerAdmin Thanks! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 |
Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: ostestr --regex tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota Actual results: tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest.test_user_modify_quota[id-d1dc5076-555e-4e6d-9697-28f1fe976324,negative] ---------------------------------------------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_quotas_negative.py", line 56, in setUp "POST", url="", headers=headers, body="") File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request self._error_checker(resp, resp_body) File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 769, in _error_checker raise exceptions.Forbidden(resp_body, resp=resp) tempest.lib.exceptions.Forbidden: Forbidden Details: AccessDenied Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2019-01-09 23:12:26,639 365 INFO [tempest.lib.common.rest_client] Request (AccountQuotasNegativeTest:setUp): 403 POST http://100.82.36.190:8080/swift/v1 0.181s 2019-01-09 23:12:26,639 365 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Account-Meta-Quota-Bytes': '20', 'X-Auth-Token': '<omitted>'} Body: Response - Headers: {'status': '403', u'content-length': '12', 'content-location': 'http://100.82.36.190:8080/swift/v1', u'accept-ranges': 'bytes', u'connection': 'close', u'x-trans-id': 'tx000000000000000000d9b-005c367fda-15789-default', u'date': 'Wed, 09 Jan 2019 23:12:26 GMT', u'content-type': 'text/plain; charset=utf-8', u'x-openstack-request-id': 'tx000000000000000000d9b-005c367fda-15789-default'} Body: AccessDenied ---------------------------------------- civetweb logging in /var/log/messages, unsure if related. Jan 23 20:08:33 mr-14g-controller-0 journal: 2019-01-23 20:08:33.945361 7f46d00b1700 0 NOTICE: couldn't map swift user 4eba560a393945c3a53460aac8afa515 Expected results: Tempest test should pass Additional info: Tempest.conf ------------------ [DEFAULT] debug = true use_stderr = false log_file = tempest.log [network-feature-enabled] ipv6_subnet_attributes = true api_extensions = default-subnetpools,qos,availability_zone,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,address-scope,external-net,standard-attr-tag,flavors,segment,net-mtu,network-ip-availability,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,l2_adjacency,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,project-id,qos-bw-limit-direction,tag-ext,standard-attr-description,ip-substring-filtering,router,allowed-address-pairs,ip_allocation,qos-fip,trunk-details [auth] tempest_roles = _member_,Member, ResellerAdmin admin_username = admin admin_project_name = admin admin_domain_name = Default use_dynamic_credentials = true admin_password = xxxxxxxxxxxxxxxxxx admin_project_id = c9fb570856934e5ca84d1f3d1cd2b526 [scenario] img_dir = etc img_file = cirros-0.3.5-x86_64-disk.img [object-storage] reseller_admin_role = ResellerAdmin region = regionOne [oslo-concurrency] lock_path = /tmp [compute-feature-enabled] live_migration = false live_migrate_paused_instances = true preserve_ports = true console_output = false resize = True attach_encrypted_volume = False api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes [identity] username = demo password = secrete project_name = demo alt_username = alt_demo alt_password = secrete alt_project_name = alt_demo disable_ssl_certificate_validation = true region = regionOne uri = http://100.82.36.190:5000//v3 auth_version = v3 uri_v3 = http://100.82.36.190:5000/v3 [image] image_path = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img region = regionOne http_image = http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img [compute] region = regionOne flavor_ref = 5ca18338-ac66-4cbd-91cf-89f870951de6 flavor_ref_alt = f0c66dfa-e435-4df1-b09c-4688eb5c00c9 image_ref = 5e6bb6f3-d212-4423-8da2-8566abda7cb8 image_ref_alt = b3128550-717b-4f7a-b8dd-749c458dfd7a [network] region = regionOne public_network_id = 008b9378-c481-4c57-8199-858aa67a105d floating_network_name = public [orchestration] stack_owner_role = swiftoperator region = regionOne [volume] backend1_name = tripleo_iscsi region = regionOne min_microversion = 3.0 max_microversion = 3.50 [volume-feature-enabled] bootable = true backup = False api_v2 = False api_v3 = True api_extensions = OS-SCH-HNT,os-hosts,os-vol-tenant-attr,os-quota-sets,os-types-manage,os-volume-encryption-metadata,os-snapshot-actions,backups,cgsnapshots,os-used-limits,os-volume-type-access,consistencygroups,os-vol-host-attr,encryption,os-availability-zone,capabilities,os-volume-actions,os-types-extra-specs,os-snapshot-manage,os-vol-mig-status-attr,os-volume-unmanage,os-volume-manage,os-image-create,os-extended-services,os-extended-snapshot-attributes,os-snapshot-unmanage,qos-specs,os-quota-class-sets,os-volume-transfer,os-vol-image-meta,os-admin-actions,os-services,scheduler-stats [object-storage-feature-enabled] discoverability = False discoverable_apis = [validation] image_ssh_user = cirros [service_available] ceilometer = True horizon = True cinder = True nova = True neutron = True trove = False glance = True manila = False panko = True ironic = False mistral = False heat = True zaqar = False swift = True sahara = False gnocchi = True octavia = False aodh = True aodh_plugin = True [dashboard] dashboard_url = http://100.82.36.190/dashboard/ login_url = http://100.82.36.190/dashboard/auth/login/ [image-feature-enabled] api_v1 = False api_v2 = True [identity-feature-enabled] api_v2 = False api_v3 = True api_extensions = s3tokens,OS-EP-FILTER,OS-REVOKE,OS-FEDERATION,OS-INHERIT,OS-SIMPLE-CERT,OS-TRUST,OS-PKI,OS-ENDPOINT-POLICY,OS-OAUTH1,OS-EC2 ------------------------------ ceph.conf on controllers [client.rgw.mr-14g-controller-0] host = mr-14g-controller-0 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-0/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-0.log rgw frontends = civetweb port=192.168.170.12:8080 num_threads=100 [client.rgw.mr-14g-controller-1] host = mr-14g-controller-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-1/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-1.log rgw frontends = civetweb port=192.168.170.13:8080 num_threads=100 [client.rgw.mr-14g-controller-2] host = mr-14g-controller-2 keyring = /var/lib/ceph/radosgw/ceph-rgw.mr-14g-controller-2/keyring log file = /var/log/ceph/ceph-rgw-mr-14g-controller-2.log rgw frontends = civetweb port=192.168.170.14:8080 num_threads=100 # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] cluster network = 192.168.180.0/24 fsid = eb28c9a4-1b45-11e9-b81c-5254001e8ca3 journal_collocation = False journal_size = 10000 log file = /dev/null # log file = /var/log/ceph/ceph.log mon cluster log file = /dev/null mon host = 192.168.170.12,192.168.170.13,192.168.170.14 mon initial members = mr-14g-controller-0,mr-14g-controller-1,mr-14g-controller-2 osd_pool_default_pg_num = 128 osd_pool_default_pgp_num = 128 osd_pool_default_size = 3 public network = 192.168.170.0/24 raw_multi_journal = True rgw_keystone_admin_domain = default rgw_keystone_admin_password = FGfWyB4q6xfkM3DtG9RXteRHW rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_api_version = 3 rgw_keystone_implicit_tenants = true rgw_keystone_revocation_interval = 0 rgw_keystone_url = http://192.168.140.251:5000 rgw_s3_auth_use_keystone = true rgw_keystone_accepted_roles = Member, admin, _member_, ResellerAdmin rgw_swift_enforce_content_length = true rgw_log_nonexistent_bucket = true rgw_enable_ops_log = true debug ms = 1 debug rgw = 20 # Preluminous_compat entry added - Start mon_health_preluminous_compat=true # Preluminous_compat entry added - End Also: it is very difficult to know where to look for the actual exception stacktrace to find the root cause.