Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2276931

Summary:	[rgw][sts]: assume_role_with_web_identity call is failing as validation of signature is failing with invalid padding
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Matt Benjamin (redhat) <mbenjamin>
Component:	RGW	Assignee:	Pritha Srivastava <prsrivas>
Status:	CLOSED ERRATA	QA Contact:	Hemanth Sai <hmaheswa>
Severity:	low	Docs Contact:	Rivka Pollack <rpollack>
Priority:	unspecified
Version:	6.1	CC:	ceph-eng-bugs, cephqe-warriors, hmaheswa, kdreyer, mbenjamin, mkasturi, prsrivas, rpollack, tserlin, vereddy
Target Milestone:	---
Target Release:	9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-20.1.0-26	Doc Type:	Bug Fix
Doc Text:	.RGW STS now supports encryption keys larger than 1024 bytes Previously, the RGW STS implementation did not support encryption keys larger than 1024 bytes. Users had to manually adjust Keycloak settings by lowering the priority of the `rsa-enc-generated` provider and reducing the `keySize` to `1024`. With this fix, RGW STS now supports encryption keys larger than 1024 bytes without requiring manual configuration changes in Keycloak. This improves security and simplifies setup.	Story Points:	---
Clone Of:	2237854	Environment:
Last Closed:	2026-01-29 06:48:03 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2237854
Bug Blocks:	2388233, 2267614, 2298578, 2298579

Description Matt Benjamin (redhat) 2024-04-24 16:23:29 UTC

+++ This bug was initially created as a clone of Bug #2237854 +++

Description of problem:
STS assume role web identity call is failing with unknown error.
debug_rgw 20 log says "sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding"

there is an existing upstream tracker for this isssue: https://tracker.ceph.com/issues/54562

workaround followed in automation:
Keycloak: realm settings -> keys, edit the rsa-enc-generated provider to have priority 90 rather than 100 and keySize 1024 instead of 2048

Seeing this issue in both pacific and quincy releases. But this issue is intermittent
the issue occurs if rsa-enc-generated realm key provider is used for validation instead of rsa-generated

sometimes the below error can also be seen in logs:
sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:02000077:rsa routines::wrong signature length

Version-Release number of selected component (if applicable):
ceph version 17.2.6-131.el9cp

How reproducible:


Steps to Reproduce:
1.deploy rhcs cluster
2.configure keycloak server
3.create openid connect provider
4.create role and put role policy
5.make assume_role_with_web_identity call with above role arn and keycloak web token in the request

Actual results:
assume_role_with_web_identity request fails intermittently with debug_rgw logs reporting signature validation failed (invalid padding) whenever it uses rsa-enc-generated realm key instead of rsa-generated

Expected results:
assume_role_with_web_identity request is successful without any errors 

Additional info:
rgw logs and automation failure logs are present at: http://magna002.ceph.redhat.com/ceph-qe-logs/HemanthSai/sts_aswi_realm_key_issue/
rgw node: 10.0.206.78
creds: root/passwd, cephuser/cephuser
keycloak server running in podman container: http://10.0.206.101:8180/

--- Additional comment from Matt Benjamin (redhat) on 2024-03-19 18:27:21 UTC ---

it seems unlikely we can inspect the original cluster, please restest with 7.1

Matt

--- Additional comment from  on 2024-03-19 20:06:47 UTC ---

Builds are ready for testing. We need a qa_ack+ in order to attach the BZ to the errata advisory and move to ON_QA.

--- Additional comment from Madhavi Kasturi on 2024-03-20 04:20:14 UTC ---

Provided qa_ack+.

QE would retest the issue and update accordingly.

--- Additional comment from errata-xmlrpc on 2024-03-20 04:33:57 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:126567-01
https://errata.engineering.redhat.com/advisory/126567

--- Additional comment from errata-xmlrpc on 2024-03-20 04:34:04 UTC ---

This bug has been added to advisory RHBA-2024:126567 by Thomas Serlin (tserlin)

--- Additional comment from Hemanth Sai on 2024-04-10 18:57:37 UTC ---

retested through automation on ceph version 18.2.1-119.el9cp,

AssumeRoleWithWebIdentity call is failing,
botocore.exceptions.ClientError: An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown


and in the rgw logs invalid padding error is seen

2024-04-10T18:43:12.955+0000 7ff1b3e27640  0 req 7173753159716794147 0.016000105s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding


automation fail logs and rgw logs at debug level 20 are present below:
http://magna002.ceph.redhat.com/ceph-qe-logs/Hemanth_Sai/sts_aswi_assume_role_invalid_padding/


pass logs with the workaround of decreasing priority of rsa-enc-generated realm key to 90:
http://magna002.ceph.redhat.com/cephci-jenkins/test-runs/18.2.1-126/Weekly/rgw/34/tier-2_rgw_sts_aswi/


the same issue is reported in this bz as well:
https://bugzilla.redhat.com/show_bug.cgi?id=2242261


moving this bz back to assigned

Comment 17 errata-xmlrpc 2026-01-29 06:48:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2026:1536