Bug 2276931

Summary: [rgw][sts]: assume_role_with_web_identity call is failing as validation of signature is failing with invalid padding
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Matt Benjamin (redhat) <mbenjamin>
Component: RGWAssignee: Pritha Srivastava <prsrivas>
Status: ASSIGNED --- QA Contact: Hemanth Sai <hmaheswa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 6.1CC: ceph-eng-bugs, cephqe-warriors, hmaheswa, kdreyer, mbenjamin, mkasturi, prsrivas, tserlin, vereddy
Target Milestone: ---   
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-18.2.1-151.el9cp Doc Type: Known Issue
Doc Text:
.Current RGW STS implementation does not support encryption keys larger than 1024 bytes The current RGW STS implementation does not support encryption keys larger than 1024 bytes. As a workaround, in `Keycloak: realm settings -> keys`, edit the ‘rsa-enc-generated’ provider to have priority 90 rather than 100 and `keySize` as 1024 instead of 2048.
 Story Points: ---
Clone Of: 2237854 Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2237854    
Bug Blocks: 2267614, 2298578, 2298579    

Description Matt Benjamin (redhat) 2024-04-24 16:23:29 UTC 
+++ This bug was initially created as a clone of Bug #2237854 +++

Description of problem:
STS assume role web identity call is failing with unknown error.
debug_rgw 20 log says "sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding"

there is an existing upstream tracker for this isssue: https://tracker.ceph.com/issues/54562

workaround followed in automation:
Keycloak: realm settings -> keys, edit the rsa-enc-generated provider to have priority 90 rather than 100 and keySize 1024 instead of 2048

Seeing this issue in both pacific and quincy releases. But this issue is intermittent
the issue occurs if rsa-enc-generated realm key provider is used for validation instead of rsa-generated

sometimes the below error can also be seen in logs:
sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:02000077:rsa routines::wrong signature length

Version-Release number of selected component (if applicable):
ceph version 17.2.6-131.el9cp

How reproducible:


Steps to Reproduce:
1.deploy rhcs cluster
2.configure keycloak server
3.create openid connect provider
4.create role and put role policy
5.make assume_role_with_web_identity call with above role arn and keycloak web token in the request

Actual results:
assume_role_with_web_identity request fails intermittently with debug_rgw logs reporting signature validation failed (invalid padding) whenever it uses rsa-enc-generated realm key instead of rsa-generated

Expected results:
assume_role_with_web_identity request is successful without any errors 

Additional info:
rgw logs and automation failure logs are present at: http://magna002.ceph.redhat.com/ceph-qe-logs/HemanthSai/sts_aswi_realm_key_issue/
rgw node: 10.0.206.78
creds: root/passwd, cephuser/cephuser
keycloak server running in podman container: http://10.0.206.101:8180/

--- Additional comment from Matt Benjamin (redhat) on 2024-03-19 18:27:21 UTC ---

it seems unlikely we can inspect the original cluster, please restest with 7.1

Matt

--- Additional comment from  on 2024-03-19 20:06:47 UTC ---

Builds are ready for testing. We need a qa_ack+ in order to attach the BZ to the errata advisory and move to ON_QA.

--- Additional comment from Madhavi Kasturi on 2024-03-20 04:20:14 UTC ---

Provided qa_ack+.

QE would retest the issue and update accordingly.

--- Additional comment from errata-xmlrpc on 2024-03-20 04:33:57 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:126567-01
https://errata.engineering.redhat.com/advisory/126567

--- Additional comment from errata-xmlrpc on 2024-03-20 04:34:04 UTC ---

This bug has been added to advisory RHBA-2024:126567 by Thomas Serlin (tserlin)

--- Additional comment from Hemanth Sai on 2024-04-10 18:57:37 UTC ---

retested through automation on ceph version 18.2.1-119.el9cp,

AssumeRoleWithWebIdentity call is failing,
botocore.exceptions.ClientError: An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown


and in the rgw logs invalid padding error is seen

2024-04-10T18:43:12.955+0000 7ff1b3e27640  0 req 7173753159716794147 0.016000105s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0200008A:rsa routines::invalid padding


automation fail logs and rgw logs at debug level 20 are present below:
http://magna002.ceph.redhat.com/ceph-qe-logs/Hemanth_Sai/sts_aswi_assume_role_invalid_padding/


pass logs with the workaround of decreasing priority of rsa-enc-generated realm key to 90:
http://magna002.ceph.redhat.com/cephci-jenkins/test-runs/18.2.1-126/Weekly/rgw/34/tier-2_rgw_sts_aswi/


the same issue is reported in this bz as well:
https://bugzilla.redhat.com/show_bug.cgi?id=2242261


moving this bz back to assigned