Description of problem: with an rgw ssl endpoint, assume-role request is failing with "ERROR: Invalid rgw sts key", but working fine with non-ssl rgw endpoint error in rgw logs at debug_level 20: 2025-04-30T10:22:36.774+0000 7f80f55a2640 2 req 5748945268038894284 0.002999983s sts:assume_role executing 2025-04-30T10:22:36.774+0000 7f80f55a2640 0 req 5748945268038894284 0.002999983s ERROR: Invalid rgw sts key, please ensure it is an alphanumeric key of length 16 2025-04-30T10:22:36.774+0000 7f80f55a2640 2 req 5748945268038894284 0.002999983s sts:assume_role completing 2025-04-30T10:22:36.774+0000 7f80f95aa640 10 req 5748945268038894284 0.002999983s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) 2025-04-30T10:22:36.774+0000 7f80f95aa640 2 req 5748945268038894284 0.002999983s sts:assume_role op status=-22 2025-04-30T10:22:36.774+0000 7f80f95aa640 2 req 5748945268038894284 0.002999983s sts:assume_role http status=400 2025-04-30T10:22:36.774+0000 7f80f95aa640 1 ====== req done req=0x7f80ad2ed4f0 op status=-22 http_status=400 latency=0.002999983s ====== ---------------------------------------------------------------- rgw logs at debug_level 20: ssl rgw debug logs: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/TFA_rgw_ssl_sts_assume_role_fail_with_invalid_sts_key/ceph-client.rgw.rgw.ssl2.ceph-hsm-ssl-8-1-0sjgat-node5.bqlwuu.log non ssl rgw debug logs: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/TFA_rgw_ssl_sts_assume_role_fail_with_invalid_sts_key/ceph-client.rgw.rgw.3.ceph-hsm-ssl-8-1-0sjgat-node5.ygsmat.log ---------------------------------------------------------------- these tests were passing on 8.0 and failing on 8.1: automation fail log on 8.1: http://magna002.ceph.redhat.com/cephci-jenkins/results/openstack/RH/8.1/rhel-9/Regression/19.2.1-147/rgw/102/tier-2_ssl_rgw_regression_test/STS_Tests_to_perform_assume_role_call_with_permissive_session_policies_0.log manual test fail log on 8.1: https://docs.google.com/document/d/1RYDVm2VGaMqEutPqXgdU8GhPYJp27L8Lwzm_eun6f1E/edit?usp=sharing automation pass log on 8.0: http://magna002.ceph.redhat.com/cephci-jenkins/results/openstack/RH/8.0/rhel-9/Regression/19.2.0-124/rgw/96/tier-2_ssl_rgw_regression_test/STS_Tests_to_perform_assume_role_call_with_permissive_session_policies_0.log ----------------------------------------------------------------- log snippet: [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ ceph config dump | grep sts client.rgw.rgw.3.ceph-hsm-ssl-8-1-0sjgat-node5.ygsmat advanced rgw_s3_auth_use_sts true client.rgw.rgw.3.ceph-hsm-ssl-8-1-0sjgat-node5.ygsmat advanced rgw_sts_key abcdefghijklmn12 * client.rgw.rgw.ssl2.ceph-hsm-ssl-8-1-0sjgat-node5.bqlwuu advanced rgw_s3_auth_use_sts true client.rgw.rgw.ssl2.ceph-hsm-ssl-8-1-0sjgat-node5.bqlwuu advanced rgw_sts_key abcdefghijklmn12 * [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 109s ago 5d count:1 crash 5/5 8m ago 5d * grafana ?:3000 1/1 2m ago 5d ceph-hsm-ssl-8-1-0sjgat-node1-installer mgr 2/2 3m ago 5d label:mgr mon 3/3 3m ago 5d label:mon node-exporter ?:9100 5/5 8m ago 5d * osd.all-available-devices 16 8m ago 5d * prometheus ?:9095 1/1 2m ago 5d ceph-hsm-ssl-8-1-0sjgat-node1-installer;count:1 rgw.rgw.3 ?:80 1/1 109s ago 96m ceph-hsm-ssl-8-1-0sjgat-node5;count:1 rgw.rgw.ssl2 ?:443 1/1 109s ago 43m ceph-hsm-ssl-8-1-0sjgat-node5 [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ ceph orch ls --export --service-type rgw --service-name rgw.rgw.ssl2 service_type: rgw service_id: rgw.ssl2 service_name: rgw.rgw.ssl2 placement: hosts: - ceph-hsm-ssl-8-1-0sjgat-node5 spec: rgw_exit_timeout_secs: 120 rgw_frontend_ssl_certificate: '-----BEGIN RSA PRIVATE KEY----- MIIEowIBAAKCAQEAm6FKocG7KBESsdH+Cm4Yi9glclg/TU5C23ywkhsfCl0dvL9y 1JJkbdoyguH3a5cYfMGezdh/dKU1oZPa8vykKDF9oHG/FiaO24s/GgsTyQ6We6Vx 9eJAZinN3gS5PEvJltxQo7ZhR2XSnZQewoeo50KLKMFClvl46OIVupVhCWbSn1k8 l5wNC2cUT9MxCdgIG2QfbXJJUbX0Xu8v5quqVUYvXnJHEa7mkwLQECq7tDWUMqZU d1whvMF2SMZxNwO2upW1TviAn3dmBES+6TuCjh9SHQLxjNGpdJ3bjS2FdIwLa60C ORvFh4DVEwjgqNxPSqE1UclOLXor0jV7ro36bQIDAQABAoIBACHldV3Z1iuZ3FqC hDQ9WndK8oz/D3qE6ybm94Y7Bf253jo38IjAAcOzXIEJOlxiomC5wx7OYqRDP3Ub wPpfcFmYveXuIFJwv3it0WaYtwx5cuknVbTaPHCD9mS+3qF1WEjVX3LozDNCyCau Fi0EYxNLeQe9YAWF/IGsjFMBKgtj1LuX20fEjKkUUuT7UDJcv5r3PDKfl4OBFSKZ iaxee+Z1zwyxN8JXlWCfsq4gmgsmIHyaMqjWOyt+JzOLFq1DcBZUYy+yGu1xnY7w sk0FVJt1X0K1eYIasj0VS88C2WK6T75cYGyNmxWfiRRVhuVV8cVPD0bcIzq/SEgf FEAjgsECgYEA0WfqdI1E079gAQ4XmRAhNOJ4RvoRvZKksC+HQOValqwTL0Voz12+ PoCO+7XMMnZRHorSOx56VaGLJakSRLm19E0yrIpCP8pqF2pNkC2tFGha5LIkH6s2 2qumScXqbbS26LZHLdSUdJDOJnFUUAf3x/7LaFUY4btyooOuzelw/pUCgYEAvkI4 B22qinC0UzBCunzV9ddpwxxlx8WoHyNFIqilCtHzvEuuB0KawRrvKTkhl0VsBEJg pgtaLMeBTjCuyhfmG8g8TdDG2YAjq+x17DxeuwmTBbewgi/ePHEYaOrnjAJ7vZSl l2XkHe9w9PAkD5c6jzNt1zYXOUefY5UzRpZzjnkCgYEAw27kVZpyndyB0PpB88wl 8aFa94Lzg6WUpgn1hQ+ImA5IaVmyE7Y9kz+QMkLdLEtYMdUFl78+FSULJD7Cmflo Y3SY0obGhZp3oRrJAMJgQieSjNXk+Nll/HcuXrOjsndyLYXQsGkxlAiBHUevlHGb Qr/9PoYNDr9OoXWccvhnSHECgYBtIGIusJI+8cIcbPP5Lx0x8ypwhNtEKHTz1zES XhS8Dgx1FhttgqBs3lwLEv/XWjRIhSIWWzCPuqbKCFZZCDgt+z3LjFJh8oODWw+Q Plg/g5BWmTOemIIpvNy5YPMKdWowJRCNtB2RpgLuGsc0QhOOi3sXE6lia1har/tH I4iIUQKBgHK9tjm/3j+jM9BHV9JwgrRMJ8smpSywHvWfwT605hKYPkmLSvnnV6mt NjD+5xU/92MhpIQeER9uRJmnvN8v6bet8+IHyZgfyOcEUweWKoEkD1tslcqpGx60 5iR7JUzo3Ohs4vaoksBld/i8P24ZKBWjMp7xlbMPxofLmER1Pf8w -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIID/TCCAuWgAwIBAgIUO2FNbrRs8f5sHyRx1fCtxcCdavAwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAklOMRIwEAYDVQQIDAlLYXJuYXRha2ExEjAQBgNVBAcM CUJhbmdhbG9yZTEUMBIGA1UECgwLUmVkIEhhdCBJbmMxFTATBgNVBAsMDERhdGEg U3RvcmFnZTEPMA0GA1UEAwwGY2VwaENJMSAwHgYJKoZIhvcNAQkBFhFjZXBoY2lA cmVkaGF0LmNvbTAeFw0yNTA0MjUwNTExMjNaFw0yNTA1MjUwNTExMjNaMIGFMQsw CQYDVQQGEwJJTjESMBAGA1UECAwJS2FybmF0YWthMRIwEAYDVQQHDAlCZW5nYWx1 cnUxFDASBgNVBAoMC1JlZCBIYXQgSW5jMRAwDgYDVQQLDAdTdG9yYWdlMSYwJAYD VQQDDB1jZXBoLWhzbS1zc2wtOC0xLTBzamdhdC1ub2RlNTCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBAJuhSqHBuygRErHR/gpuGIvYJXJYP01OQtt8sJIb HwpdHby/ctSSZG3aMoLh92uXGHzBns3Yf3SlNaGT2vL8pCgxfaBxvxYmjtuLPxoL E8kOlnulcfXiQGYpzd4EuTxLyZbcUKO2YUdl0p2UHsKHqOdCiyjBQpb5eOjiFbqV YQlm0p9ZPJecDQtnFE/TMQnYCBtkH21ySVG19F7vL+arqlVGL15yRxGu5pMC0BAq u7Q1lDKmVHdcIbzBdkjGcTcDtrqVtU74gJ93ZgREvuk7go4fUh0C8YzRqXSd240t hXSMC2utAjkbxYeA1RMI4KjcT0qhNVHJTi16K9I1e66N+m0CAwEAAaNTMFEwTwYD VR0RBEgwRoIfKi5jZXBoLWhzbS1zc2wtOC0xLTBzamdhdC1ub2RlNYIdY2VwaC1o c20tc3NsLTgtMS0wc2pnYXQtbm9kZTWHBAoAQ9YwDQYJKoZIhvcNAQELBQADggEB AGP87U5bU3Yc57RCb5awly2p2Q44FeaApZDxyZ981hXICHBCJq7vWrT/UZbpZ7bl fjiM2adVS80+ZOsnrS3jugZlwq9DlM4fxsnuDKU92qvIVpMHM6sA3hoIESxKzanl d29L/fvb1oQaBxd68HTmTvT0NZepgCtZIGf9CaacBD+L3c8a/ynPkIOJay8iKcvB 64MDRvUN/dUFtP/U5szDuF+6nEo8EYw3I8elKl0mcawMNfLZD01Zw+NBepPoK5OK omXiEJnoiypTv1krPAH91y9dBpum2nLdvQbK69TogVCz98qijRrsn4GyP9Zbvc0W UpDE1YciRjB/DX1SIkHdFhE= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIID/zCCAuegAwIBAgIJAI0xMKJjzvKNMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYD VQQGEwJJTjESMBAGA1UECAwJS2FybmF0YWthMRIwEAYDVQQHDAlCYW5nYWxvcmUx FDASBgNVBAoMC1JlZCBIYXQgSW5jMRUwEwYDVQQLDAxEYXRhIFN0b3JhZ2UxDzAN BgNVBAMMBmNlcGhDSTEgMB4GCSqGSIb3DQEJARYRY2VwaGNpQHJlZGhhdC5jb20w HhcNMjEwODIzMDkxNzQ2WhcNMzEwODIxMDkxNzQ2WjCBlTELMAkGA1UEBhMCSU4x EjAQBgNVBAgMCUthcm5hdGFrYTESMBAGA1UEBwwJQmFuZ2Fsb3JlMRQwEgYDVQQK DAtSZWQgSGF0IEluYzEVMBMGA1UECwwMRGF0YSBTdG9yYWdlMQ8wDQYDVQQDDAZj ZXBoQ0kxIDAeBgkqhkiG9w0BCQEWEWNlcGhjaUByZWRoYXQuY29tMIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwwtZlkbTvHsMjpA7WYf1e6c+XdaOeVN/ QBXCUss2KSDaFYKfX0keOfTSFuE3yYucG4hXAVyjfEHrNMC5wt5omggUlW3cLtXX PEXMUFTgBlerNVqvkO0L6JqaZhVaAhwMWSc/5gx4kA2vquoMSQisb7vwVuo2okSO L07oGXsOBhGjggwuBRb0muY23tIb6Z63kEy8baLaeQJti2to4VJ6btOMV4sYyOEO 6ppd170Gb8MOb43HJjgk7apSj24Aq4koMddKOOFCSCUre6PiiZirDuHsxZmEK+MF 3RKFrix3b/m80/a4aUmwOQezQgtq0pIFOb+rLITby9jr/+aNfP9MkQIDAQABo1Aw TjAdBgNVHQ4EFgQUMgp0QuoVzuVfmR1/nWdhOZtBayowHwYDVR0jBBgwFoAUMgp0 QuoVzuVfmR1/nWdhOZtBayowDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC AQEAUZAaOZDp8ksEfds8w2wgrool8n1e/zds8t9dbJoGiAyttvdcq94fzH9VyKsu eoUYoC2fhsMYwxuYfCiKM5bpiuNGT+LnErKzzykRcnkiGFm8ocBSSkA5+1+8I73m JmhEmy6lPz6dvxswlnOmz6n6SM9GMte0OqkdmsJ/LDJFkjRZYWyI4gumZK1f1uzd H43OdiO1crszx79DIFPK0eucmep+xMSiTLgS3qQnHsOX3WHb4STbtPdxmreg87va vgU3PEe5pI3Kx0top1GQqHXyNPiZaQQZv9stDyn9GRccsqZ80zUN/k/oDlogh2At ZMlXvHmwYFEu/j1lMMWmvTsQ+A== -----END CERTIFICATE----- ' ssl: true [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ radosgw-admin role create --role-name hsm-role1 --assume-role-policy-doc "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/hsm\"]},\"Action\":[\"sts:AssumeRole\"]}]}" { "RoleId": "99bedb42-324c-4335-a4b8-f7bec70e7c83", "RoleName": "hsm-role1", "Path": "/", "Arn": "arn:aws:iam:::role/hsm-role1", "CreateDate": "2025-04-30T09:46:47.412Z", "Description": "", "MaxSessionDuration": 3600, "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/hsm\"]},\"Action\":[\"sts:AssumeRole\"]}]}", "AccountId": "" } [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ date; aws --endpoint-url http://10.0.67.214:80 --no-verify-ssl sts assume-role --role-session-name "hsm_session1" --role-arn "arn:aws:iam:::role/hsm-role1" Wed Apr 30 09:49:24 UTC 2025 { "Credentials": { "AccessKeyId": "vHBKWPU8G5J8J5uRWS1K", "SecretAccessKey": "P71RRPICTZRBOBNGUANQKC0BF732E9P78CLWN532", "SessionToken": "ASNcBSDoiNN0B3/mCdhf5v27sbmcMlO4eiPZzc5Yo1ZP5hzQoOigMslBM2l5qdQhS8jETSnfk59YVTsKwvRolDHe44cjmON4VBaZ4xGgnCVaWwfqVRMDyFjDr9fd1HPBlv4Z06c5kAC803l2f5Tmay9xd1qunBpu2GANytn1qLlrpBEzGO8RSB8SB+mHb85nod5XqoWB+aQjtUn5lo9TVlkXFiWsaJizlgLlZMXsccHvvKkFaBsND/fgcei73AgKhL6AU0YKWEDzbmBKsVFN5IBLv8B+YtW0zRvBJXaX1cTBLbtT1a8PIbPAIQpPZTPsvMMzbcxRsYKJ5xMG9cV/hA==", "Expiration": "2025-04-30T10:49:24.736793610Z" }, "AssumedRoleUser": { "Arn": "arn:aws:sts:::assumed-role/hsm-role1/hsm_session1" }, "PackedPolicySize": 0 } [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ date; aws --endpoint-url https://10.0.67.214:443 --no-verify-ssl sts assume-role --role-session-name "hsm_session1" --role-arn "arn:aws:iam:::role/hsm-role1" Wed Apr 30 10:22:36 UTC 2025 /usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1018: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.67.214'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( An error occurred (InvalidArgument) when calling the AssumeRole operation: None [cephuser@ceph-hsm-ssl-8-1-0sjgat-node5 ~]$ --------------------------------------------------------------------- Version-Release number of selected component (if applicable): ceph version 19.2.1-161.el9cp How reproducible: always Steps to Reproduce: 1.deploy rhcs8.1 cluster with rgw ssl and non-ssl daemons 2.create a user, add roles=* capability, create a role 3.perform assume-role with rgw-ssl endpoint, the request is failing with InvalidArgument. Actual results: assume-role is failing with invalid_sts_key if we use rgw ssl endpoint Expected results: expected assume-role works fine with rgw non-ssl and ssl endpoints Additional info: Environment details: rgw node: 10.0.67.214 creds: root/passwd, cephuser/cephuser
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775