Bug 2293036

Summary: [rgw-ms][assume-role]:After a successful assume role api call on the secondary site, s3 bucket create fails with http_status 403
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Hemanth Sai <hmaheswa>
Component: RGW-MultisiteAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED DUPLICATE QA Contact: Madhavi Kasturi <mkasturi>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0CC: akraj, ceph-eng-bugs, cephqe-warriors, mbenjamin, prsrivas, smanjara, tserlin, vimishra
Target Milestone: ---Keywords: Automation
Target Release: 7.1z3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2271399 Environment:
Last Closed: 2025-01-15 14:14:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2271399    
Bug Blocks: 2267614, 2271595, 2273547, 2298578, 2298579    

Description Hemanth Sai 2024-06-19 06:55:14 UTC
+++ This bug was initially created as a clone of Bug #2271399 +++

Description of problem:


In a multisite environment, the creation of a role, attachment of a role policy on the primary site, and assume role API calls on the 'secondary site' are successful. 

However, when attempting s3 operations such as bucket creation using the session token generated post the assume role API call on the secondary site, the operation fails with http_status = 403.



Version-Release number of selected component (if applicable):

ceph version 18.2.1-77.el9cp

How reproducible:

2/2

Steps to Reproduce:


1. Set up a multisite environment, and perform the below steps on the primary site

- create 2 users 'lynna.271' and 'annief.469' 
- add the role capability to the user 'lynna.271'
- create a role 'S3RoleOf.lynna.271' and attach a role policy 

(venv) [root@pluto003 rgw]# radosgw-admin role get --role-name S3RoleOf.lynna.271
{
    "RoleId": "fe7ddcc3-e2d8-41dc-a80c-2f4cecfb4313",
    "RoleName": "S3RoleOf.lynna.271",
    "Path": "/",
    "Arn": "arn:aws:iam:::role/S3RoleOf.lynna.271",
    "CreateDate": "2024-03-25T03:53:45.404Z",
    "MaxSessionDuration": 3600,
    "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/annief.469\"]},\"Action\":[\"sts:AssumeRole\"]}]}",
    "PermissionPolicies": [
        {
            "PolicyName": "policy.lynna.271",
            "PolicyValue": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}"
        }
    ]
}

(venv) [root@pluto003 rgw]# radosgw-admin role-policy get --role-name S3RoleOf.lynna.271 --policy-name policy.lynna.271
{
    "Permission policy": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}"
}

2. Ensure the role and the role policy are synced to the secondary site.

3. Perform the below steps on the secondary site

- Perform the assume role API call,
- Obtain the session token generated from the assume role API call.
- Attempt to perform s3 operations such as creating a bucket using the obtained session token on the secondary site.


------------------ Use the below boto script on the secondary site---------- 

import boto3

import json
import logging
from botocore.exceptions import ClientError
logging.basicConfig(filename="boto.log", level=logging.DEBUG)

from botocore.handlers import validate_bucket_name


sts_client = boto3.client('sts',
aws_access_key_id='user_annief.469_access_key',
aws_secret_access_key='user_annief.469_secret_key',
endpoint_url='http://secondary_site_endpoint',
region_name='shared',
)

response = sts_client.assume_role(
RoleArn='arn:aws:iam:::role/S3RoleOf.lynna.271',
RoleSessionName='Bob',
DurationSeconds=3600
)

print(f"print the assume role response {response}")

s3client = boto3.client('s3',
aws_access_key_id = response['Credentials']['AccessKeyId'],
aws_secret_access_key = response['Credentials']['SecretAccessKey'],
aws_session_token = response['Credentials']['SessionToken'],
endpoint_url='http://secondary_site_endpoint',
region_name='shared',)

bucket_name = 'sec-my-bucket'
s3bucket = s3client.create_bucket(Bucket=bucket_name)
resp = s3client.list_buckets()
print(resp)



Actual results:

Observed the hhtp_status 403 failure for s3 create_bucket, despite a successful assume role API call on the secondary site.


Expected results:

s3 operations should pass based on the role policy on both sites.


Additional info:

On the primary site, the same script (with primary zone endpoint) runs successfully without any issues.