Bug 2153199

Summary: External mode using SSL for RGW fails because nooba doesn't know about CA certificate
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Daniel Horák <dahorak>
Component: Multi-Cloud Object GatewayAssignee: Nimrod Becker <nbecker>
Status: ASSIGNED --- QA Contact: krishnaram Karthick <kramdoss>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.12CC: jalbo, nbecker, odf-bz-bot
Target Milestone: ---Flags: dahorak: needinfo? (jalbo)
Target Release: ODF 4.13.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.13.0-90 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Horák 2022-12-14 09:30:45 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
  I'm trying to configure External mode cluster with SSL enabled for RGW[1]
  and `noobaa` object gets stuck in `Configuring` phase with following reason:
    Put "https://<IP>:443/nb.1671007052593.apps.<cluster-url>": x509: certificate signed by unknown authority


Version of all relevant components (if applicable):
  OCP: 4.12.0-0.nightly-2022-12-13-205407
  ODF: 4.12.0-140


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
  I'm not able to configure the external mode cluster with SSL enabled for RGW.


Is there any workaround available to the best of your knowledge?
  I'm not sure, if there is any.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
  4


Can this issue reproducible?
  yes


Can this issue reproduce from the UI?
  N/A


Steps to Reproduce:
1. Prepare Ceph cluster with SSL enabled for RGW
2. Install ODF and run create-external-cluster-resources.py with proper
    parameters
    $ python3 create-external-cluster-resources.py \
      --rbd-data-pool-name rbd --rgw-endpoint <IP>:443 \
      --rgw-tls-cert-path /tmp/cephqe-ca.pem

3. Continue with the ODF deployment.


Actual results:
  $ oc get noobaa -n openshift-storage
  NAME     S3-ENDPOINTS                     STS-ENDPOINTS                    IMAGE                                                                                                            PHASE         AGE
  noobaa   ["https://<IP>:30788"]   ["https://<IP>:32388"]   quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:b495b59219d78ab468d1e1faedacfda59cb4b9fe13b253157897ff6899811de5   Configuring   80m


  $ oc describe noobaa -n openshift-storage
  Name:         noobaa
  Namespace:    openshift-storage
  Labels:       app=noobaa
  Annotations:  <none>
  API Version:  noobaa.io/v1alpha1
  Kind:         NooBaa
  ...
  Status:
    Accounts:
      Admin:
        Secret Ref:
          Name:       noobaa-admin
          Namespace:  openshift-storage
    Actual Image:     quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:b495b59219d78ab468d1e1faedacfda59cb4b9fe13b253157897ff6899811de5
    Conditions:
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Available
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                True
      Type:                  Progressing
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Degraded
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Upgradeable
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Status:                k8s
      Type:                  KMS-Type
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:44Z
      Status:                Sync
      Type:                  KMS-Status
    Observed Generation:     2
    Phase:                   Configuring
  ...


  $ oc logs -n openshift-storage noobaa-operator-558d485d8c-zj7r9
  time="2022-12-14T09:22:57Z" level=info msg="CephObjectStoreUser \"noobaa-ceph-objectstore-user\" created. Creating default backing store on ceph objectstore" func=ReconcileDefaultBackingStore sys=openshift-storage/noobaa
  time="2022-12-14T09:22:57Z" level=info msg="✅ Exists:  \"noobaa-ceph-objectstore-user\"\n"
  time="2022-12-14T09:22:57Z" level=info msg="✅ Exists:  \"rook-ceph-object-user-ocs-external-storagecluster-cephobjectstore-noobaa-ceph-objectstore-user\"\n"
  time="2022-12-14T09:22:57Z" level=info msg="Will connect to RGW at \"https://<IP>:443\"" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:57Z" level=info msg="creating bucket nb.1671009777800.apps.<cluster-url>" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=error msg="got error when trying to create bucket nb.1671009777800.apps.<cluster-url>. error: RequestError: send request failed\ncaused by: Put \"https://<IP>:443/nb.1671009777800.apps.<cluster-url>\": x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=warning msg="⏳ Temporary Error: RequestError: send request failed\ncaused by: Put \"https://<IP>:443/nb.1671009777800.apps.<cluster-url>\": x509: certificate signed by unknown authority" sys=openshift-storage/noobaa

  See also full noobaa-operator log[2].

Expected results:
  The noobaa will be correctly configured.


Additional info:
  must-gather logs[3]
  please check also following comment from Blaine in the original Jira[4]


[1] https://issues.redhat.com/browse/RHSTOR-2537
[2] https://url.corp.redhat.com/08df047
[3] https://url.corp.redhat.com/ef0e7c0
[4] https://issues.redhat.com/browse/RHSTOR-2537?focusedCommentId=21266903#comment-21266903