Bug 2319118

Summary: [cephadm][rgw-ms]: ream pull on secondary site is failing with "SSL certificate problem" with rgw ssl endpoint deployed with generate_cert
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Hemanth Sai <hmaheswa>
Component: CephadmAssignee: Adam King <adking>
Status: CLOSED ERRATA QA Contact: Hemanth Sai <hmaheswa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0CC: akraj, cephqe-warriors, tserlin
Target Milestone: ---   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.2.0-51.el9cp Doc Type: Known Issue
Doc Text:
Cause: certificates generated by cephadm with the "generate"cert" option in the RGW spec are self-signed Consequence: You may run into "SSL peer certificate or SSH remote key was not OK req_data->error_buf=SSL certificate problem: self-signed certificate" failures. This was specifically seen doing a realm pull from a secondary site. Workaround (if any): None Result: The automatic cert generation from the "generate_cert" option in the RGW spec is self-signed and may cause issues when pulling from the secondary site.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-25 09:13:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hemanth Sai 2024-10-16 10:09:43 UTC 
Description of problem:
ream pull on secondary site is failing with "SSL certificate problem" with rgw ssl deployed with generate_cert
because of this multisite creation with rgw ssl(generate_cert) is blocked.

[cephuser@ceph-sec-hsm-ms-cb6rd2-node6 ~]$ radosgw-admin realm pull --rgw-realm india --url https://10.0.65.191:443 --access-key 21e86bce636c3aa0 --secret cf764951f1fdde5d --default --debug-rgw 20
2024-10-15T15:35:21.054-0400 7f0106206800 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2,1] modes=[2,1]
2024-10-15T15:35:21.054-0400 7f0106206800 20 rgw_check_secure_mon_conn(): method 1 is insecure
2024-10-15T15:35:21.059-0400 7effaeffd640 20 reqs_thread_entry: start
2024-10-15T15:35:21.059-0400 7f0106206800 20 NOTICE: cannot identify region for connection to: 10.0.65.191:443
2024-10-15T15:35:21.059-0400 7f0106206800 20 sign_request_v4():> HTTP_DATE -> Tue Oct 15 19:35:21 2024
2024-10-15T15:35:21.059-0400 7f0106206800 10 canonical headers format = date:Tue Oct 15 19:35:21 2024
x-amz-date:20241015T193521Z

2024-10-15T15:35:21.059-0400 7f0106206800 10 payload request hash = UNSIGNED-PAYLOAD
2024-10-15T15:35:21.059-0400 7f0106206800 10 canonical request = GET
/admin/realm
name=india
date:Tue Oct 15 19:35:21 2024
x-amz-date:20241015T193521Z

date;x-amz-date
UNSIGNED-PAYLOAD
2024-10-15T15:35:21.059-0400 7f0106206800 10 canonical request hash = 798a71cf7a5bc88bade1e8e3527b48dc9432a92de41c99dfd3a7d252e251dfb0
2024-10-15T15:35:21.059-0400 7f0106206800 10 string to sign = AWS4-HMAC-SHA256
20241015T193521Z
20241015//s3/aws4_request
798a71cf7a5bc88bade1e8e3527b48dc9432a92de41c99dfd3a7d252e251dfb0
2024-10-15T15:35:21.059-0400 7f0106206800 10 date_k    = b4e5467f4a2310d0eb6c77fa11dade2f2e34cd7a9a43c5f9a3b7a46e69e1951d
2024-10-15T15:35:21.059-0400 7f0106206800 10 region_k  = 949de0528140dcf9e3539b99093778877f1b41f02ad92d5c4f853b0bd0a99664
2024-10-15T15:35:21.059-0400 7f0106206800 10 service_k = b47c44183b5fc433677426d28ae6af008aac2588f05da28ec33071be81328625
2024-10-15T15:35:21.059-0400 7f0106206800 10 signing_k = 47ee044d3ba7e7958f7285fd379ad9f677a46353df7a8266fd27381ebf7e6eec
2024-10-15T15:35:21.059-0400 7f0106206800 10 generated signature = 35aa924137acc48ef2acd7375c016f69b77b4a3605048491bec152396706d4e3
2024-10-15T15:35:21.059-0400 7f0106206800 20 sign_request_v4(): sigv4 header: Authorization: AWS4-HMAC-SHA256 Credential=21e86bce636c3aa0/20241015//s3/aws4_request,SignedHeaders=date;x-amz-date,Signature=35aa924137acc48ef2acd7375c016f69b77b4a3605048491bec152396706d4e3
2024-10-15T15:35:21.059-0400 7f0106206800 20 sign_request_v4(): sigv4 header: x-amz-content-sha256: UNSIGNED-PAYLOAD
2024-10-15T15:35:21.059-0400 7f0106206800 20 sign_request_v4(): sigv4 header: x-amz-date: 20241015T193521Z
2024-10-15T15:35:21.059-0400 7f0106206800 20 sending request to https://10.0.65.191:443/admin/realm?name=india
2024-10-15T15:35:21.059-0400 7f0106206800 20 register_request mgr=0x7f00ec00ae60 req_data->id=0, curl_handle=0x56098f47db80
2024-10-15T15:35:21.059-0400 7effaeffd640 20 link_request req_data=0x56098f3c15f0 req_data->id=0, curl_handle=0x56098f47db80
2024-10-15T15:35:21.096-0400 7effaeffd640 20 ERROR: msg->data.result=60 req_data->id=0 http_status=0
2024-10-15T15:35:21.096-0400 7effaeffd640 20 ERROR: curl error: SSL peer certificate or SSH remote key was not OK req_data->error_buf=SSL certificate problem: self-signed certificate
request failed: (2200) Unknown error 2200
[cephuser@ceph-sec-hsm-ms-cb6rd2-node6 ~]$ 


Version-Release number of selected component (if applicable):
ceph version 19.2.0-24.el9cp

How reproducible:
always

Steps to Reproduce:
1.deploy rhcs8.0 ceph cluster
2.create a realm, zonegroup and zone on pri site. period update commit
3.deploy rgw ssl on pri site with generate_cert in the spec file
[cephuser@ceph-pri-hsm-ms-cb6rd2-node6 ~]$ cat rgw_spec_gen_cert.yaml 
service_type: rgw
service_id: shared.pri
service_name: rgw.shared.pri
placement:
  hosts:
    - ceph-pri-hsm-ms-cb6rd2-node5
spec:
  generate_cert: true
  ssl: true
  rgw_realm: india
  rgw_zone: primary
  rgw_zonegroup: shared
[cephuser@ceph-pri-hsm-ms-cb6rd2-node6 ~]$ 

4.perform realm pull on the secondary site for multisite creation. its failing with ssl certificate error
[cephuser@ceph-sec-hsm-ms-cb6rd2-node6 ~]$ radosgw-admin realm pull --rgw-realm india --url https://10.0.65.191:443 --access-key 21e86bce636c3aa0 --secret cf764951f1fdde5d --default --debug-rgw 20
2024-10-15T15:44:32.794-0400 7ff7127fc640 20 ERROR: curl error: SSL peer certificate or SSH remote key was not OK req_data->error_buf=SSL certificate problem: self-signed certificate
request failed: (2200) Unknown error 2200
[cephuser@ceph-sec-hsm-ms-cb6rd2-node6 ~]$ 


Actual results:
realm pull on sec site with rgw gen_ssl endpoint is failing with ssl error

Expected results:
expected realm pull is successful with rgw ssl (generate_cert) endpoint

Additional info:
log output of commands is captured here: https://docs.google.com/document/d/11VG1kbRymyGJ5E863ENfFg2ihpvO0AeksIq7QCx9DO4/edit?usp=sharing
Comment 17 errata-xmlrpc 2024-11-25 09:13:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216
Comment 18 Red Hat Bugzilla 2025-03-26 04:26:24 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days