Description of problem (please be detailed as possible and provide log snippests): External cluster deployment ailed with "noobaa-default-backing-store" not found Version of all relevant components (if applicable): odf-operator.v4.9.0-132.ci Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes Is there any workaround available to the best of your knowledge? NA Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 1/1 Can this issue reproduce from the UI? Not tried If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Installl OCS using ocs-ci 2. check the backingstore 3. Actual results: $ oc -n openshift-storage get backingstore No resources found in openshift-storage namespace. $ Expected results: noobaa-default-backing-store should be found Additional info: > csv are in succeeded phase $ oc get csv NAME DISPLAY VERSION REPLACES PHASE noobaa-operator.v4.9.0-132.ci NooBaa Operator 4.9.0-132.ci Succeeded ocs-operator.v4.9.0-132.ci OpenShift Container Storage 4.9.0-132.ci Succeeded odf-operator.v4.9.0-132.ci OpenShift Data Foundation 4.9.0-132.ci Succeeded > pods $ oc get pods NAME READY STATUS RESTARTS AGE csi-cephfsplugin-dwpfh 3/3 Running 0 16m csi-cephfsplugin-f46b2 3/3 Running 0 16m csi-cephfsplugin-gvh7d 3/3 Running 0 16m csi-cephfsplugin-provisioner-8dc9b74b5-psfn4 6/6 Running 0 16m csi-cephfsplugin-provisioner-8dc9b74b5-wdwb5 6/6 Running 0 16m csi-rbdplugin-nb2zm 3/3 Running 0 16m csi-rbdplugin-provisioner-56cb9bf6bf-9jxdm 6/6 Running 0 16m csi-rbdplugin-provisioner-56cb9bf6bf-s5cpd 6/6 Running 0 16m csi-rbdplugin-qxx6m 3/3 Running 0 16m csi-rbdplugin-thnk5 3/3 Running 0 16m must-gather-lnzvp-helper 1/1 Running 0 51s noobaa-core-0 1/1 Running 0 16m noobaa-db-pg-0 1/1 Running 0 16m noobaa-endpoint-67b4558dbd-jlc6m 1/1 Running 0 13m noobaa-operator-7bfdf6d8d7-c945q 1/1 Running 0 18m ocs-metrics-exporter-84f79d5986-4xdfq 1/1 Running 0 18m ocs-operator-794cf5d6bf-z2g9b 1/1 Running 0 18m odf-console-5d7786c7cb-zmvl5 2/2 Running 0 18m odf-operator-controller-manager-7dcc7456d7-md6g6 2/2 Running 0 18m rook-ceph-operator-797bb85f7-wgx7k 1/1 Running 0 18m > cephcluster is in connected state $ oc get cephcluster NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL ocs-external-storagecluster-cephcluster 16m Connected Cluster connected successfully HEALTH_OK true > $ oc -n openshift-storage get backingstore No resources found in openshift-storage namespace. $ > cephobjectstore is in progressing state $ oc get cephobjectstore NAME AGE ocs-external-storagecluster-cephobjectstore 23m [vavuthu@vavuthu rem]$ oc get cephobjectstore -o yaml apiVersion: v1 items: - apiVersion: ceph.rook.io/v1 kind: CephObjectStore metadata: creationTimestamp: "2021-09-14T07:41:23Z" finalizers: - cephobjectstore.ceph.rook.io generation: 1 managedFields: status: info: endpoint: https://rook-ceph-rgw-ocs-external-storagecluster-cephobjectstore.openshift-storage.svc:8080 phase: Progressing kind: List metadata: resourceVersion: "" selfLink: "" > $ oc get noobaa NAME MGMT-ENDPOINTS S3-ENDPOINTS IMAGE PHASE AGE noobaa ["https://10.1.160.236:32216"] ["https://10.1.160.132:31240"] quay.io/rhceph-dev/mcg-core@sha256:a23df063d713a7bce416f9e8635af52d2ec382485f7d4aaf404d6d7f209790f5 Configuring 35m > noooba operator log ( noobaa-operator-7bfdf6d8d7-c945q ) time="2021-09-14T08:04:32Z" level=info msg="❌ Not Found: BackingStore \"noobaa-default-backing-store\"\n" time="2021-09-14T08:04:32Z" level=info msg="CephObjectStoreUser \"noobaa-ceph-objectstore-user\" created. Creating default backing store on ceph objectstore" func=ReconcileDefaultBackingStore sys=openshift-storage/noobaa time="2021-09-14T08:04:32Z" level=info msg="✅ Exists: \"noobaa-ceph-objectstore-user\"\n" time="2021-09-14T08:04:32Z" level=info msg="Ceph objectstore user \"noobaa-ceph-objectstore-user\" is not ready. retry on next reconcile.." sys=openshift-storage/noobaa time="2021-09-14T08:04:32Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa time="2021-09-14T08:04:32Z" level=warning msg="⏳ Temporary Error: Ceph objectstore user \"noobaa-ceph-objectstore-user\" is not ready" sys=openshift-storage/noobaa time="2021-09-14T08:04:32Z" level=info msg="UpdateStatus: Done generation 1" sys=openshift-storage/noobaa > rook ceph operator ( rook-ceph-operator-797bb85f7-wgx7k ) has below error ( not sue its relevant or not ) 2021-09-14 07:41:38.838733 E | ceph-object-controller: failed to create bucket checker for CephObjectStore "openshift-storage/ocs-external-storagecluster-cephobjectstore": failed to fetch CA cert to establish TLS connection with object store "openshift-storage/ocs-external-storagecluster-cephobjectstore": failed to get secret ceph-rgw-tls-cert containing TLS certificate defined in ocs-external-storagecluster-cephobjectstore: secrets "ceph-rgw-tls-cert" not found must gather logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/vavuthuext1-odf/vavuthuext1-odf_20210914T064952/logs/failed_testcase_ocs_logs_1631603285/deployment_ocs_logs/ job: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/5931/console
Please share the JSON output of the external script.
Vijay, did you use the new flag `--rgw-tls-cert-path`, it looks like the cert is missing from the output.
The ObjectStore CR specifies: securePort: 8080 sslCertificateRef: ceph-rgw-tls-cert This means a secure connection is in place and thus a certificate is used. Typically for rgw to run with TLS, the certificate is located in /etc/ceph/private/rgw-cert.pem I don't see any bug here, just a misconfiguration. If you agree please close this BZ. Thanks.
It looks like ocs-op is setting SecurePort in the CephObjectStore CR but the external is not using TLS. ocs-op should check the presence of a Secret call "ceph-rgw-tls-cert" and then turn on TLS if needed. Arun PTAL
clearing needinfo based on comment #8
Taking the BZ. Will update.
A minor fix, PR: https://github.com/red-hat-storage/ocs-operator/pull/1346 is up Jose please take a look
[asandler@fedora ~]$ oc get csv No resources found in default namespace. [asandler@fedora ~]$ oc get csv -A \NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-operator-lifecycle-manager packageserver Package Server 0.18.3 Succeeded openshift-storage noobaa-operator.v4.9.0-158.ci NooBaa Operator 4.9.0-158.ci Succeeded openshift-storage ocs-operator.v4.9.0-158.ci OpenShift Container Storage 4.9.0-158.ci Succeeded openshift-storage odf-operator.v4.9.0-158.ci OpenShift Data Foundation 4.9.0-158.ci Succeeded [asandler@fedora ~]$ oc -n openshift-storage get backingstore NAME TYPE PHASE AGE noobaa-default-backing-store aws-s3 Ready 167m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:5086