Description of problem:
put-bucket-notification-configuration/get-bucket-notification-configuration are not allowed from other users even after bucket policy is set to allow them
fail log: http://magna002.ceph.redhat.com/cephci-jenkins/results/openstack/RH/6.1/rhel-9/Weekly/17.2.6-239/rgw/34/tier-2_rgw_regression/test_bucket_policy_with_multiple_statements_0.log


the same issue is seen with AssumeRole user as well.
fail log: http://magna002.ceph.redhat.com/cephci-jenkins/results/openstack/IBM/6.1/rhel-9/Regression/17.2.6-239/rgw/64/tier-1-extn_rgw/STS_test_to_verify_session_policy_allow_actions_0.log


only these two operations are denied, the other op's are allowed as expected

these tests are passing on reef.


Version-Release number of selected component (if applicable):
ceph version 17.2.6-246.el9cp

How reproducible:
always

Steps to Reproduce:
1.create two rgw users and configure awscli for both users
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ radosgw-admin user create --uid t1user1 --display-name t1user1 --access-key abc1 --secret abc1 --debug-rgw 0
{
    "user_id": "t1user1",
    "display_name": "t1user1",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "t1user1",
            "access_key": "abc1",
            "secret_key": "abc1"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ radosgw-admin user create --uid t1user2 --display-name t1user2 --access-key abc2 --secret abc2 --debug-rgw 0
{
    "user_id": "t1user2",
    "display_name": "t1user2",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "t1user2",
            "access_key": "abc2",
            "secret_key": "abc2"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ cat ~/.aws/credentials 
[user1]
aws_access_key_id = abc1
aws_secret_access_key = abc1
[user2]
aws_access_key_id = abc2
aws_secret_access_key = abc2
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$

2.create a bucket
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3 mb s3://bkt1
make_bucket: bkt1
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3 cp obj10MB s3://bkt1
upload: ./obj10MB to s3://bkt1/obj10MB                           
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$

3.put-bucket-policy to allow all users and all s3 actions. access bucket from user2
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ cat bkt_policy.json 
{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": "s3:*",
         "Resource": "arn:aws:s3:::*"
      }
   ]
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3api put-bucket-policy --bucket bkt1 --policy file://bkt_policy.json
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3api get-bucket-policy --bucket bkt1
{
    "Policy": "{\n   \"Statement\": [\n      {\n         \"Effect\": \"Allow\",\n         \"Principal\": \"*\",\n         \"Action\": \"s3:*\",\n         \"Resource\": \"arn:aws:s3:::*\"\n      }\n   ]\n}\n"
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user2 s3 ls s3://bkt1
2024-08-21 13:07:35   10000000 obj10MB
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 

4.create sns topic
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 sns create-topic --name fishtopic
{
    "TopicArn": "arn:aws:sns:default::fishtopic"
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$

5.perform put-bucket-notification-configuration and get-bucket-notification-configuration on the bucket from user2. it fails with AccessDenied. from user1 it succeeds as expected
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ cat notif.json 
{
    "TopicConfigurations": [
        {
            "Id": "notif1",
            "TopicArn": "arn:aws:sns:default::fishtopic",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user2 s3api put-bucket-notification-configuration --bucket bkt1 --notification-configuration file://notif.json

An error occurred (AccessDenied) when calling the PutBucketNotificationConfiguration operation: Unknown
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3api put-bucket-notification-configuration --bucket bkt1 --notification-configuration file://notif.json
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user1 s3api get-bucket-notification-configuration --bucket bkt1
{
    "TopicConfigurations": [
        {
            "Id": "notif1",
            "TopicArn": "arn:aws:sns:default::fishtopic",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user2 s3api get-bucket-notification-configuration --bucket bkt1

An error occurred (AccessDenied) when calling the GetBucketNotificationConfiguration operation: Unknown
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 

6.but other operations like put-bucket-policy are allowed from user2.
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user2 s3api put-bucket-policy --bucket bkt1 --policy file://bkt_policy.json
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ aws --endpoint-url http://10.0.66.194:80 --profile user2 s3api get-bucket-policy --bucket bkt1
{
    "Policy": "{\n   \"Statement\": [\n      {\n         \"Effect\": \"Allow\",\n         \"Principal\": \"*\",\n         \"Action\": \"s3:*\",\n         \"Resource\": \"arn:aws:s3:::*\"\n      }\n   ]\n}\n"
}
[cephuser@ceph-hsm-quincy-kkgz9z-node4 ~]$ 




Actual results:
put/get bucket notifications are not allowed from other_user/sts_user even after bucket_policy/sts_policy permission is set

Expected results:
expected put/get bucket notifications are allowed from other_user/sts_user after bucket_policy/sts_policy permission is set

Additional info:
rgw logs at debug_level 20: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/bz_put_bkt_notif_access_denied_quincy/ceph-client.rgw.rgw.1.ceph-hsm-quincy-kkgz9z-node4.uatgpr.log