+++ This bug was initially created as a clone of Bug #2271399 +++

Description of problem:


In a multisite environment, the creation of a role, attachment of a role policy on the primary site, and assume role API calls on the 'secondary site' are successful. 

However, when attempting s3 operations such as bucket creation using the session token generated post the assume role API call on the secondary site, the operation fails with http_status = 403.



Version-Release number of selected component (if applicable):

ceph version 18.2.1-77.el9cp

How reproducible:

2/2

Steps to Reproduce:


1. Set up a multisite environment, and perform the below steps on the primary site

- create 2 users 'lynna.271' and 'annief.469' 
- add the role capability to the user 'lynna.271'
- create a role 'S3RoleOf.lynna.271' and attach a role policy 

(venv) [root@pluto003 rgw]# radosgw-admin role get --role-name S3RoleOf.lynna.271
{
    "RoleId": "fe7ddcc3-e2d8-41dc-a80c-2f4cecfb4313",
    "RoleName": "S3RoleOf.lynna.271",
    "Path": "/",
    "Arn": "arn:aws:iam:::role/S3RoleOf.lynna.271",
    "CreateDate": "2024-03-25T03:53:45.404Z",
    "MaxSessionDuration": 3600,
    "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/annief.469\"]},\"Action\":[\"sts:AssumeRole\"]}]}",
    "PermissionPolicies": [
        {
            "PolicyName": "policy.lynna.271",
            "PolicyValue": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}"
        }
    ]
}

(venv) [root@pluto003 rgw]# radosgw-admin role-policy get --role-name S3RoleOf.lynna.271 --policy-name policy.lynna.271
{
    "Permission policy": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}"
}

2. Ensure the role and the role policy are synced to the secondary site.

3. Perform the below steps on the secondary site

- Perform the assume role API call,
- Obtain the session token generated from the assume role API call.
- Attempt to perform s3 operations such as creating a bucket using the obtained session token on the secondary site.


------------------ Use the below boto script on the secondary site---------- 

import boto3

import json
import logging
from botocore.exceptions import ClientError
logging.basicConfig(filename="boto.log", level=logging.DEBUG)

from botocore.handlers import validate_bucket_name


sts_client = boto3.client('sts',
aws_access_key_id='user_annief.469_access_key',
aws_secret_access_key='user_annief.469_secret_key',
endpoint_url='http://secondary_site_endpoint',
region_name='shared',
)

response = sts_client.assume_role(
RoleArn='arn:aws:iam:::role/S3RoleOf.lynna.271',
RoleSessionName='Bob',
DurationSeconds=3600
)

print(f"print the assume role response {response}")

s3client = boto3.client('s3',
aws_access_key_id = response['Credentials']['AccessKeyId'],
aws_secret_access_key = response['Credentials']['SecretAccessKey'],
aws_session_token = response['Credentials']['SessionToken'],
endpoint_url='http://secondary_site_endpoint',
region_name='shared',)

bucket_name = 'sec-my-bucket'
s3bucket = s3client.create_bucket(Bucket=bucket_name)
resp = s3client.list_buckets()
print(resp)



Actual results:

Observed the hhtp_status 403 failure for s3 create_bucket, despite a successful assume role API call on the secondary site.


Expected results:

s3 operations should pass based on the role policy on both sites.


Additional info:

On the primary site, the same script (with primary zone endpoint) runs successfully without any issues.