+++ This bug was initially created as a clone of Bug #2237854 +++ Description of problem: STS assume role web identity call is failing with unknown error. debug_rgw 20 log says "sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding" there is an existing upstream tracker for this isssue: https://tracker.ceph.com/issues/54562 workaround followed in automation: Keycloak: realm settings -> keys, edit the rsa-enc-generated provider to have priority 90 rather than 100 and keySize 1024 instead of 2048 Seeing this issue in both pacific and quincy releases. But this issue is intermittent the issue occurs if rsa-enc-generated realm key provider is used for validation instead of rsa-generated sometimes the below error can also be seen in logs: sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:02000077:rsa routines::wrong signature length Version-Release number of selected component (if applicable): ceph version 17.2.6-131.el9cp How reproducible: Steps to Reproduce: 1.deploy rhcs cluster 2.configure keycloak server 3.create openid connect provider 4.create role and put role policy 5.make assume_role_with_web_identity call with above role arn and keycloak web token in the request Actual results: assume_role_with_web_identity request fails intermittently with debug_rgw logs reporting signature validation failed (invalid padding) whenever it uses rsa-enc-generated realm key instead of rsa-generated Expected results: assume_role_with_web_identity request is successful without any errors Additional info: rgw logs and automation failure logs are present at: http://magna002.ceph.redhat.com/ceph-qe-logs/HemanthSai/sts_aswi_realm_key_issue/ rgw node: 10.0.206.78 creds: root/passwd, cephuser/cephuser keycloak server running in podman container: http://10.0.206.101:8180/ --- Additional comment from Matt Benjamin (redhat) on 2024-03-19 18:27:21 UTC --- it seems unlikely we can inspect the original cluster, please restest with 7.1 Matt --- Additional comment from on 2024-03-19 20:06:47 UTC --- Builds are ready for testing. We need a qa_ack+ in order to attach the BZ to the errata advisory and move to ON_QA. --- Additional comment from Madhavi Kasturi on 2024-03-20 04:20:14 UTC --- Provided qa_ack+. QE would retest the issue and update accordingly. --- Additional comment from errata-xmlrpc on 2024-03-20 04:33:57 UTC --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHBA-2024:126567-01 https://errata.engineering.redhat.com/advisory/126567 --- Additional comment from errata-xmlrpc on 2024-03-20 04:34:04 UTC --- This bug has been added to advisory RHBA-2024:126567 by Thomas Serlin (tserlin) --- Additional comment from Hemanth Sai on 2024-04-10 18:57:37 UTC --- retested through automation on ceph version 18.2.1-119.el9cp, AssumeRoleWithWebIdentity call is failing, botocore.exceptions.ClientError: An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown and in the rgw logs invalid padding error is seen 2024-04-10T18:43:12.955+0000 7ff1b3e27640 0 req 7173753159716794147 0.016000105s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding automation fail logs and rgw logs at debug level 20 are present below: http://magna002.ceph.redhat.com/ceph-qe-logs/Hemanth_Sai/sts_aswi_assume_role_invalid_padding/ pass logs with the workaround of decreasing priority of rsa-enc-generated realm key to 90: http://magna002.ceph.redhat.com/cephci-jenkins/test-runs/18.2.1-126/Weekly/rgw/34/tier-2_rgw_sts_aswi/ the same issue is reported in this bz as well: https://bugzilla.redhat.com/show_bug.cgi?id=2242261 moving this bz back to assigned