With OSP17 RGW is deployed by default and the we run a set of tests (previously run against swift) to make sure everything works as expected. As per the config [1], RGW is able to process and validate users authorized by keystone and let them to stat/download/upload/etc objects within a logical swift container mapped to rgw. However, sounds like ACL(s) set using the swift cli are not respected, and this makes several tests (including the related tempest suit) fail. For instance, if I make the following: openstack project create acl_test openstack user create --project acl_test --password user1 user1 openstack role add --user user1 member and run the stat command against an existing container (created by admin): [stack@undercloud-0 ~]$ swift stat admincontainer Unauthorized. Check username/id, password, tenant name/id and user/tenant domain name/id. but, adding the user to the admin project: [stack@undercloud-0 ~]$ openstack role add --user user1 --project admin member [stack@undercloud-0 ~]$ swift stat admincontainer Account: AUTH_3c8251b886cc4c03b64e275f68e4ee66 Container: admincontainer Objects: 7 Bytes: 704428 Read ACL: Write ACL: Sync To: Sync Key: X-Timestamp: 1647003643.11096 X-Container-Bytes-Used-Actual: 724992 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Fri, 11 Mar 2022 14:34:25 GMT X-Trans-Id: tx00000eedc2e1bb7412d97-00622b6070-5e8b-default X-Openstack-Request-Id: tx00000eedc2e1bb7412d97-00622b6070-5e8b-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 Now, trying to restrict the write acl to an existing user2 (but it works also passing an empty string): [stack@undercloud-0 ~]$ swift --debug post admincontainer --read-acl "*:*" --write-acl "admin:user2" DEBUG:swiftclient:REQ: curl -i http://10.0.0.122:8080/swift/v1/AUTH_3c8251b886cc4c03b64e275f68e4ee66/admincontainer -X POST -H "X-Auth-Token: gAAAAABiK2GltzxPvknOB6OTJsRfT3MaBeueRFHJju0vJQYw3ZF9HH-um3EM8YFbK-ej4BxITtYunN0IUGkv4HLvBALFLuC_Rl_Udhu1Ufoy6A4VEQ706U3CUm-3EOV-JttAjHgjHJTBRDiv0hGXBqnsagBiMQmbbPUneM-wegyF7515IMpDXXw" -H "X-Container-Read: *:*" -H "X-Container-Write: admin:user2" -H "Content-Length: 0" DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: {'X-Trans-Id': 'tx0000061d1f0d8b4f014a9-00622b61a5-5e9d-default', 'X-Openstack-Request-Id': 'tx0000061d1f0d8b4f014a9-00622b61a5-5e9d-default', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Fri, 11 Mar 2022 14:50:13 GMT'} I'm still able to stat (and download) using user1: [stack@undercloud-0 ~]$ swift stat admincontainer Account: AUTH_3c8251b886cc4c03b64e275f68e4ee66 Container: admincontainer Objects: 7 Bytes: 704428 Read ACL: *:* Write ACL: admin:user2 Sync To: Sync Key: X-Timestamp: 1647003643.11096 X-Container-Bytes-Used-Actual: 724992 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Fri, 11 Mar 2022 14:34:25 GMT X-Trans-Id: tx00000eedc2e1bb7412d97-00622b6070-5e8b-default X-Openstack-Request-Id: tx00000eedc2e1bb7412d97-00622b6070-5e8b-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 But again, using user1, I'm still able to put objects, and I shouldn't be able to do that: [stack@undercloud-0 ~]$ swift upload admincontainer /tmp/foo Warning: failed to create container 'admincontainer': 409 Conflict: BucketAlreadyExists tmp/collect-container-logs.sh [stack@undercloud-0 ~]$ swift list admincontainer tmp/foo The same thing happens with tempest, where a test is supposed to change the ACL (and deny the writes) but at the end of the process we see the object being created [2]: Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2022-03-11 10:51:55,616 867248 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:setUp): 201 PUT http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779 0.160s 2022-03-11 10:51:55,617 867248 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'content-length': '0', 'x-trans-id': 'tx000006d492d0289310290-00622b29cb-5e8d-default', 'x-openstack-request-id': 'tx000006d492d0289310290-00622b29cb-5e8d-default', 'accept-ranges': 'bytes', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 11 Mar 2022 10:51:55 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779'} Body: b'' 2022-03-11 10:51:55,736 867248 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:test_write_object_without_write_rights): 204 POST http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779 0.118s 2022-03-11 10:51:55,737 867248 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Container-Read': 'tempest-ObjectACLsNegativeTest-489040919:tempest-ObjectACLsNegativeTest-489040919-project-admin', 'X-Container-Write': '', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'x-trans-id': 'tx000005a6dacf2433578f3-00622b29cb-5e8b-default', 'x-openstack-request-id': 'tx000005a6dacf2433578f3-00622b29cb-5e8b-default', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 11 Mar 2022 10:51:55 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779'} Body: b'' 2022-03-11 10:51:55,875 867248 INFO [tempest.lib.common.rest_client] Request (ObjectACLsNegativeTest:test_write_object_without_write_rights): 201 PUT http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779/tempest-Object-505497810 0.137s 2022-03-11 10:51:55,875 867248 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'} Body: data Response - Headers: {'etag': '8d777f385d3dfec8815d20f7496026dc', 'last-modified': 'Fri, 11 Mar 2022 10:51:55 GMT', 'x-trans-id': 'tx0000072399c037f7ea0eb-00622b29cb-5e9d-default', 'x-openstack-request-id': 'tx0000072399c037f7ea0eb-00622b29cb-5e9d-default', 'content-type': 'text/plain; charset=utf-8', 'content-length': '0', 'date': 'Fri, 11 Mar 2022 10:51:55 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.122:8080/swift/v1/AUTH_6e238e67122e454eb1bbcc3789fd7b1e/tempest-TestContainer-1141385779/tempest-Object-505497810'} Body: b'' [ceph: root@controller-0 /]# rados -p default.rgw.buckets.data ls | grep 810 cc452055-f71a-4286-80e7-e2b4673ca5a8.24217.119_tempest-Object-505497810 [1] rgw config within the ceph mgr: global advanced rgw_enforce_swift_acls true global advanced rgw_keystone_accepted_admin_roles ResellerAdmin, swiftoperator global advanced rgw_keystone_accepted_roles member, Member, admin global advanced rgw_keystone_admin_domain default global advanced rgw_keystone_admin_password IzWGbPcwmhlFGzhcjmhLXOmpd global advanced rgw_keystone_admin_project service global advanced rgw_keystone_admin_user swift global advanced rgw_keystone_api_version 3 global advanced rgw_keystone_implicit_tenants true global basic rgw_keystone_url http://172.17.1.19:5000 global advanced rgw_max_attr_name_len 128 global advanced rgw_max_attr_size 256 global advanced rgw_max_attrs_num_in_req 90 global advanced rgw_s3_auth_use_keystone true global advanced rgw_swift_account_in_url true global advanced rgw_swift_enforce_content_length true global advanced rgw_swift_versioning_enabled true global advanced rgw_trust_forwarded_https true [2] https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_container_acl_negative.py#L184
I had issues with rhos17 and tempest failing tests. Managed to run tempest successfull with radosgw . tempest.conf : (Valid based on tripleo defaults) [auth] tempest_roles = member,creator [object-storage] reseller_admin_role = ResellerAdmin operator_role = admin Default from radosgw which may be related : rgw_keystone_accepted_roles: 'member, Member, admin' rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator tempest use the operator_role as the os_operator and in radowgw they set the swiftoperator as rgw admin role ... When i tried to run operator_role = swiftoperator , i got tempest failures. operator_role = admin works ! I think that its related to the defaults from tripleo ... Hope it helped, Benny
(In reply to bkopilov from comment #6) > I had issues with rhos17 and tempest failing tests. > Managed to run tempest successfull with radosgw . > > tempest.conf : (Valid based on tripleo defaults) > [auth] > tempest_roles = member,creator > > [object-storage] > reseller_admin_role = ResellerAdmin > operator_role = admin > > > Default from radosgw which may be related : > rgw_keystone_accepted_roles: 'member, Member, admin' > rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator > > tempest use the operator_role as the os_operator and in radowgw they set the > swiftoperator as rgw admin role ... > When i tried to run operator_role = swiftoperator , i got tempest failures. > operator_role = admin works ! For the record, the failing configuration mentioned in the description does not use operator_role = admin. It uses [auth] tempest_roles = swiftoperator,creator [object-storage] reseller_admin_role = ResellerAdmin operator_role = admin Also, the description lists a set of manual steps (not tempest) which used to work previously and now fails.
(In reply to Luigi Toscano from comment #7) > (In reply to bkopilov from comment #6) > > I had issues with rhos17 and tempest failing tests. > > Managed to run tempest successfull with radosgw . > > > > tempest.conf : (Valid based on tripleo defaults) > > [auth] > > tempest_roles = member,creator > > > > [object-storage] > > reseller_admin_role = ResellerAdmin > > operator_role = admin > > > > > > Default from radosgw which may be related : > > rgw_keystone_accepted_roles: 'member, Member, admin' > > rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator > > > > tempest use the operator_role as the os_operator and in radowgw they set the > > swiftoperator as rgw admin role ... > > When i tried to run operator_role = swiftoperator , i got tempest failures. > > operator_role = admin works ! > > For the record, the failing configuration mentioned in the description does > not use operator_role = admin. Of course that should read as "the failing configuration *does* use operator_role = admin" > It uses > > > [auth] > tempest_roles = swiftoperator,creator > > [object-storage] > reseller_admin_role = ResellerAdmin > operator_role = admin > > > Also, the description lists a set of manual steps (not tempest) which used > to work previously and now fails.
Bumping this to urgent as it is blocking the Phase 2 testing gate for the OSP 17.0 compose.
(In reply to bkopilov from comment #6) > I had issues with rhos17 and tempest failing tests. > Managed to run tempest successfull with radosgw . > > tempest.conf : (Valid based on tripleo defaults) > [auth] > tempest_roles = member,creator > > [object-storage] > reseller_admin_role = ResellerAdmin > operator_role = admin "operator_role" should be set to "member" or left unset (defaults to "member") > Default from radosgw which may be related : > rgw_keystone_accepted_roles: 'member, Member, admin' > rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator this looks correct, it is how tripleo configures rgw; do I understand correctly that by using the above configs we do see the tempest test passing?
(In reply to Giulio Fidente from comment #13) > (In reply to bkopilov from comment #6) > > I had issues with rhos17 and tempest failing tests. > > Managed to run tempest successfull with radosgw . > > > > tempest.conf : (Valid based on tripleo defaults) > > [auth] > > tempest_roles = member,creator > > > > [object-storage] > > reseller_admin_role = ResellerAdmin > > operator_role = admin > > "operator_role" should be set to "member" or left unset (defaults to > "member") > > > Default from radosgw which may be related : > > rgw_keystone_accepted_roles: 'member, Member, admin' > > rgw_keystone_accepted_admin_roles: ResellerAdmin, swiftoperator > > this looks correct, it is how tripleo configures rgw; do I understand > correctly that by using the above configs we do see the tempest test passing? Yes, in case tempest.conf configured with : [auth] tempest_roles = member,creator [object-storage] reseller_admin_role = ResellerAdmin operator_role = member # default tempest swift(object) tests passed successfully. Thanks, Benny
Disabling the older code/workaround for bug #1484419 which was setting auth.tempest_roles = swiftoperator, seems resolved the issue, leaving it just on tempest/tempest-conf defaults (member) seems working with osp17+ceph+radosgw. https://review.gerrithub.io/c/rhos-infra/cloud-config/+/539662 So closing as NOTABUG since defaults seem to work as expected now, mistake was on bad override which is not needed anymore.