Bug 2153199 - External mode using SSL for RGW fails because nooba doesn't know about CA certificate [NEEDINFO]
Summary: External mode using SSL for RGW fails because nooba doesn't know about CA cer...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: Multi-Cloud Object Gateway
Version: 4.12
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ODF 4.13.0
Assignee: Nimrod Becker
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-12-14 09:30 UTC by Daniel Horák
Modified: 2023-08-09 16:49 UTC (History)
3 users (show)

Fixed In Version: 4.13.0-90
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:
dahorak: needinfo? (jalbo)


Attachments (Terms of Use)

Description Daniel Horák 2022-12-14 09:30:45 UTC
Description of problem (please be detailed as possible and provide log
snippests):
  I'm trying to configure External mode cluster with SSL enabled for RGW[1]
  and `noobaa` object gets stuck in `Configuring` phase with following reason:
    Put "https://<IP>:443/nb.1671007052593.apps.<cluster-url>": x509: certificate signed by unknown authority


Version of all relevant components (if applicable):
  OCP: 4.12.0-0.nightly-2022-12-13-205407
  ODF: 4.12.0-140


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
  I'm not able to configure the external mode cluster with SSL enabled for RGW.


Is there any workaround available to the best of your knowledge?
  I'm not sure, if there is any.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
  4


Can this issue reproducible?
  yes


Can this issue reproduce from the UI?
  N/A


Steps to Reproduce:
1. Prepare Ceph cluster with SSL enabled for RGW
2. Install ODF and run create-external-cluster-resources.py with proper
    parameters
    $ python3 create-external-cluster-resources.py \
      --rbd-data-pool-name rbd --rgw-endpoint <IP>:443 \
      --rgw-tls-cert-path /tmp/cephqe-ca.pem

3. Continue with the ODF deployment.


Actual results:
  $ oc get noobaa -n openshift-storage
  NAME     S3-ENDPOINTS                     STS-ENDPOINTS                    IMAGE                                                                                                            PHASE         AGE
  noobaa   ["https://<IP>:30788"]   ["https://<IP>:32388"]   quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:b495b59219d78ab468d1e1faedacfda59cb4b9fe13b253157897ff6899811de5   Configuring   80m


  $ oc describe noobaa -n openshift-storage
  Name:         noobaa
  Namespace:    openshift-storage
  Labels:       app=noobaa
  Annotations:  <none>
  API Version:  noobaa.io/v1alpha1
  Kind:         NooBaa
  ...
  Status:
    Accounts:
      Admin:
        Secret Ref:
          Name:       noobaa-admin
          Namespace:  openshift-storage
    Actual Image:     quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:b495b59219d78ab468d1e1faedacfda59cb4b9fe13b253157897ff6899811de5
    Conditions:
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Available
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                True
      Type:                  Progressing
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Degraded
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Message:               RequestError: send request failed
  caused by: Put "https://<IP>:443/nb.1671009473760.apps.<cluster-url>": x509: certificate signed by unknown authority
      Reason:                TemporaryError
      Status:                False
      Type:                  Upgradeable
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:43Z
      Status:                k8s
      Type:                  KMS-Type
      Last Heartbeat Time:   2022-12-14T07:56:43Z
      Last Transition Time:  2022-12-14T07:56:44Z
      Status:                Sync
      Type:                  KMS-Status
    Observed Generation:     2
    Phase:                   Configuring
  ...


  $ oc logs -n openshift-storage noobaa-operator-558d485d8c-zj7r9
  time="2022-12-14T09:22:57Z" level=info msg="CephObjectStoreUser \"noobaa-ceph-objectstore-user\" created. Creating default backing store on ceph objectstore" func=ReconcileDefaultBackingStore sys=openshift-storage/noobaa
  time="2022-12-14T09:22:57Z" level=info msg="✅ Exists:  \"noobaa-ceph-objectstore-user\"\n"
  time="2022-12-14T09:22:57Z" level=info msg="✅ Exists:  \"rook-ceph-object-user-ocs-external-storagecluster-cephobjectstore-noobaa-ceph-objectstore-user\"\n"
  time="2022-12-14T09:22:57Z" level=info msg="Will connect to RGW at \"https://<IP>:443\"" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:57Z" level=info msg="creating bucket nb.1671009777800.apps.<cluster-url>" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=error msg="got error when trying to create bucket nb.1671009777800.apps.<cluster-url>. error: RequestError: send request failed\ncaused by: Put \"https://<IP>:443/nb.1671009777800.apps.<cluster-url>\": x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa
  time="2022-12-14T09:22:58Z" level=warning msg="⏳ Temporary Error: RequestError: send request failed\ncaused by: Put \"https://<IP>:443/nb.1671009777800.apps.<cluster-url>\": x509: certificate signed by unknown authority" sys=openshift-storage/noobaa

  See also full noobaa-operator log[2].

Expected results:
  The noobaa will be correctly configured.


Additional info:
  must-gather logs[3]
  please check also following comment from Blaine in the original Jira[4]


[1] https://issues.redhat.com/browse/RHSTOR-2537
[2] https://url.corp.redhat.com/08df047
[3] https://url.corp.redhat.com/ef0e7c0
[4] https://issues.redhat.com/browse/RHSTOR-2537?focusedCommentId=21266903#comment-21266903


Note You need to log in before you can comment on or make changes to this bug.