Description of problem (please be detailed as possible and provide log snippests): After NooBaa creates a CephObjectStoreUser, there's an attempt to look for it. However, it fails - >$ oc describe noobaa >... >Ceph object user secret "rook-ceph-object-user-STORE_NAME-noobaa-ceph-objectstore-user" is not ready yet The secret name does not seem to contain anything under the "STORE_NAME" part - >$ oc get secret | grep object >rook-ceph-object-user--noobaa-ceph-objectstore-user Version of all relevant components (if applicable): OCP 4.5.0-0.nightly-2020-07-06-211538 OCS 4.5.0-480.ci on independent mode Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes, NooBaa fails to deploy Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? No If this is a regression, please provide more details to justify this: No, NooBaa still does not work on independent mode Steps to Reproduce: 1. Deploy an OCP cluster 2. Install OCS >4.5.0-477 3. Run `oc describe noobaa`, it's stuck with the error described above Actual results: NooBaa deployment fails Expected results: NooBaa finds the secret and uses it to create a default backingstore Additional info:
Hi Ben Independent mode in 4.5 has no CephObjectSotre installed in the OCS cluster. which mean the secret name should be "rook-ceph-object-user--noobaa-ceph-objectstore-user" Noobaa finds the RGW gateway by looking at a label named "rgw-endpoint" which the OCS operator set on the noobaa CR. If a label does not exist it assume it is converged mode. I think there is a mismatch with OCS-operator/NooBaa-operator versions here. To start the diagnosis lets check that an "rgw-endpoint" label is set on the NooBaa CR and what its value contains. You can check and post here or attach a must-gather or provide me with access to the env
$ oc get noobaa --show-labels NAME MGMT-ENDPOINTS S3-ENDPOINTS IMAGE PHASE AGE LABELS noobaa [https://10.70.60.44:31985] [https://10.70.60.49:32544] quay.io/rhceph-dev/mcg-core@sha256:86d27e39c293f14da564b7edac3804b6afb09490d24ab01b66d315f12ca7141b Configuring 125m app=noobaa,rgw-endpoint=10.1.8.96_8080
Proposing as a blocker since this might be the cause for the ocs-operator CSV to be stuck in Installing state $ oc get csv NAME DISPLAY VERSION REPLACES PHASE ocs-operator.v4.5.0-480.ci OpenShift Container Storage 4.5.0-480.ci Installing INFO[0000] CLI version: 2.3.0 INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0-rc3 INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0 INFO[0000] Namespace: openshift-storage INFO[0000] INFO[0000] CRD Status: INFO[0000] ✅ Exists: CustomResourceDefinition "noobaas.noobaa.io" INFO[0000] ✅ Exists: CustomResourceDefinition "backingstores.noobaa.io" INFO[0000] ✅ Exists: CustomResourceDefinition "bucketclasses.noobaa.io" INFO[0000] ✅ Exists: CustomResourceDefinition "objectbucketclaims.objectbucket.io" INFO[0000] ✅ Exists: CustomResourceDefinition "objectbuckets.objectbucket.io" INFO[0000] INFO[0000] Operator Status: INFO[0000] ✅ Exists: Namespace "openshift-storage" INFO[0000] ✅ Exists: ServiceAccount "noobaa" INFO[0000] ✅ Exists: Role "ocs-operator.v4.5.0-480.ci-86797d7d59" INFO[0000] ✅ Exists: RoleBinding "ocs-operator.v4.5.0-480.ci-86797d7d59-7fb5b5bbfb" INFO[0000] ✅ Exists: ClusterRole "ocs-operator.v4.5.0-480.ci-d4c5fc6b6" INFO[0000] ✅ Exists: ClusterRoleBinding "ocs-operator.v4.5.0-480.ci-d4c5fc6b6-7c454596f4" INFO[0000] ✅ Exists: Deployment "noobaa-operator" INFO[0000] INFO[0000] System Status: INFO[0000] ✅ Exists: NooBaa "noobaa" INFO[0000] ✅ Exists: StatefulSet "noobaa-core" INFO[0000] ✅ Exists: StatefulSet "noobaa-db" INFO[0000] ✅ Exists: Service "noobaa-mgmt" INFO[0000] ✅ Exists: Service "s3" INFO[0000] ✅ Exists: Service "noobaa-db" INFO[0000] ✅ Exists: Secret "noobaa-server" INFO[0000] ✅ Exists: Secret "noobaa-operator" INFO[0000] ✅ Exists: Secret "noobaa-endpoints" INFO[0000] ✅ Exists: Secret "noobaa-admin" INFO[0000] ❌ Not Found: StorageClass "openshift-storage.noobaa.io" INFO[0000] ❌ Not Found: BucketClass "noobaa-default-bucket-class" INFO[0000] ✅ Exists: Deployment "noobaa-endpoint" INFO[0000] ✅ Exists: HorizontalPodAutoscaler "noobaa-endpoint" INFO[0000] ⬛ (Optional) Not Found: BackingStore "noobaa-default-backing-store" INFO[0000] ⬛ (Optional) Not Found: CredentialsRequest "noobaa-aws-cloud-creds" INFO[0000] ⬛ (Optional) Not Found: CredentialsRequest "noobaa-azure-cloud-creds" INFO[0000] ⬛ (Optional) Not Found: Secret "noobaa-azure-container-creds" INFO[0000] ⬛ (Optional) Not Found: PrometheusRule "noobaa-prometheus-rules" INFO[0000] ⬛ (Optional) Not Found: ServiceMonitor "noobaa-service-monitor" INFO[0000] ✅ (Optional) Exists: Route "noobaa-mgmt" INFO[0000] ✅ (Optional) Exists: Route "s3" INFO[0000] ✅ Exists: PersistentVolumeClaim "db-noobaa-db-0" INFO[0000] ❌ System Phase is "Configuring" INFO[0000] ⏳ System Phase is "Configuring". Waiting for phase ready ...
A quick update, Noobaa is creating a CephObjectStoreUser with a store name of an empty string. If it cannot find the secret in the same reconcile loop it will schedule another reconcile loop in the future. In this new reconcile loop the CephObjectStoreUser already exists and then we have the bug where we do not erase the default store name which is "STORE_NAME". I am looking into the details now and will update as soon as I have a solution, preferably with an upstream PR in place.
A PR was opened on the upsream project (see links sections)
For traceability it would be good to add the backport PR to the BZ when moving to modified. https://github.com/noobaa/noobaa-operator/pull/363
From the linked output, it looks like the noobaa operator has finished a full system reconciliation, which means that the fix has helped. It also looks like noobaa is set up correctly. Regarding the CSV issue, I believe it is a separate bug and might not even be related to NooBaa. As such my suggestion is to move this BZ verified
Thank You Ohad & Nimrod for all the clarifications in the slack channel and the google chatrooms Based on the following status and comment#13 , moving the BZ to verified state. 1. noobaa status looks good and is setup properly 2. able to create OBCs, attach to pod and run some IO 3. IO status can be seen from the Noobaa console 4. Confirmed that for OCS 4.5, there is no CephObjectStore . There is only a CephObjectStoreUser $ oc describe noobaa noobaa|grep 'Ceph object user secret' $ --> No output $ oc describe noobaa |grep 'CephObjectStoreUser' $ --> no output >>Versions OCP = 4.5.0-0.nightly-2020-07-06-211538 OCS = ocs-operator.v4.5.0-482.ci RHCS = 4.1.0 (14.2.8-59.el8cp) $ oc get secret -n openshift-storage |grep object rook-ceph-object-user--noobaa-ceph-objectstore-user kubernetes.io/rook 2 6h5m $ oc get -n openshift-storage cephobjectStore No resources found. >>> Confirmed that there would not a cephobjectstore for Independent Mode in OCS 4.5 $ oc get -n openshift-storage cephobjectStoreUser NAME AGE noobaa-ceph-objectstore-user 6h6m $ oc get backingstore -A NAMESPACE NAME TYPE PHASE AGE openshift-storage noobaa-default-backing-store s3-compatible Ready 5h59m $ oc get bucketclass -A NAMESPACE NAME PLACEMENT PHASE AGE openshift-storage noobaa-default-bucket-class map[tiers:[map[backingStores:[noobaa-default-backing-store]]]] Ready 5h59m [nberry@localhost logs]$ >>> Noobaa-status $ noobaa status INFO[0000] CLI version: 2.3.0 INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0-rc3 INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0 INFO[0000] Namespace: openshift-storage INFO[0000] INFO[0000] CRD Status: INFO[0002] ✅ Exists: CustomResourceDefinition "noobaas.noobaa.io" INFO[0002] ✅ Exists: CustomResourceDefinition "backingstores.noobaa.io" INFO[0002] ✅ Exists: CustomResourceDefinition "bucketclasses.noobaa.io" INFO[0002] ✅ Exists: CustomResourceDefinition "objectbucketclaims.objectbucket.io" INFO[0002] ✅ Exists: CustomResourceDefinition "objectbuckets.objectbucket.io" INFO[0002] INFO[0002] Operator Status: INFO[0003] ✅ Exists: Namespace "openshift-storage" INFO[0003] ✅ Exists: ServiceAccount "noobaa" INFO[0004] ✅ Exists: Role "ocs-operator.v4.5.0-482.ci-86797d7d59" INFO[0004] ✅ Exists: RoleBinding "ocs-operator.v4.5.0-482.ci-86797d7d59-5984b4fcfd" INFO[0004] ✅ Exists: ClusterRole "ocs-operator.v4.5.0-482.ci-d4c5fc6b6" INFO[0005] ✅ Exists: ClusterRoleBinding "ocs-operator.v4.5.0-482.ci-d4c5fc6b6-5c5d7cc6c8" INFO[0005] ✅ Exists: Deployment "noobaa-operator" INFO[0005] INFO[0005] System Status: INFO[0005] ✅ Exists: NooBaa "noobaa" INFO[0005] ✅ Exists: StatefulSet "noobaa-core" INFO[0006] ✅ Exists: StatefulSet "noobaa-db" INFO[0006] ✅ Exists: Service "noobaa-mgmt" INFO[0006] ✅ Exists: Service "s3" INFO[0006] ✅ Exists: Service "noobaa-db" INFO[0007] ✅ Exists: Secret "noobaa-server" INFO[0007] ✅ Exists: Secret "noobaa-operator" INFO[0007] ✅ Exists: Secret "noobaa-endpoints" INFO[0007] ✅ Exists: Secret "noobaa-admin" INFO[0007] ✅ Exists: StorageClass "openshift-storage.noobaa.io" INFO[0008] ✅ Exists: BucketClass "noobaa-default-bucket-class" INFO[0008] ✅ Exists: Deployment "noobaa-endpoint" INFO[0008] ✅ Exists: HorizontalPodAutoscaler "noobaa-endpoint" INFO[0008] ✅ (Optional) Exists: BackingStore "noobaa-default-backing-store" INFO[0009] ⬛ (Optional) Not Found: CredentialsRequest "noobaa-aws-cloud-creds" INFO[0009] ⬛ (Optional) Not Found: CredentialsRequest "noobaa-azure-cloud-creds" INFO[0009] ⬛ (Optional) Not Found: Secret "noobaa-azure-container-creds" INFO[0009] ✅ (Optional) Exists: PrometheusRule "noobaa-prometheus-rules" INFO[0010] ✅ (Optional) Exists: ServiceMonitor "noobaa-service-monitor" INFO[0010] ✅ (Optional) Exists: Route "noobaa-mgmt" INFO[0010] ✅ (Optional) Exists: Route "s3" INFO[0011] ✅ Exists: PersistentVolumeClaim "db-noobaa-db-0" INFO[0011] ✅ System Phase is "Ready" INFO[0011] ✅ Exists: "noobaa-admin" #------------------# #- Mgmt Addresses -# #------------------# ExternalDNS : [https://noobaa-mgmt-openshift-storage.apps.sagrawal-dc3-ind.qe.rh-ocs.com] ExternalIP : [] NodePorts : [https://10.70.60.44:32708] InternalDNS : [https://noobaa-mgmt.openshift-storage.svc:443] InternalIP : [https://172.30.140.203:443] PodPorts : [https://10.129.2.32:8443] #--------------------# #- Mgmt Credentials -# #--------------------# email : admin password : oky9Fcx5YCLiMfD1IZ8+ug== #----------------# #- S3 Addresses -# #----------------# ExternalDNS : [https://s3-openshift-storage.apps.sagrawal-dc3-ind.qe.rh-ocs.com] ExternalIP : [] NodePorts : [https://10.70.60.44:31898] InternalDNS : [https://s3.openshift-storage.svc:443] InternalIP : [https://172.30.85.239:443] PodPorts : [https://10.129.2.33:6443] #------------------# #- S3 Credentials -# #------------------# AWS_ACCESS_KEY_ID : W7R8vYpkQFBn3GahhnuX AWS_SECRET_ACCESS_KEY : /HIbTPsdlQgxgB/2yrsGszlAmngI9X4zg8h63mVu #------------------# #- Backing Stores -# #------------------# NAME TYPE TARGET-BUCKET PHASE AGE noobaa-default-backing-store s3-compatible nb.1594277648909.apps.sagrawal-dc3-ind.qe.rh-ocs.com Ready 1h59m45s #------------------# #- Bucket Classes -# #------------------# NAME PLACEMENT PHASE AGE noobaa-default-bucket-class {Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]} Ready 1h59m45s #-----------------# #- Bucket Claims -# #-----------------# NAMESPACE NAME BUCKET-NAME STORAGE-CLASS BUCKET-CLASS PHASE test nb-test nb-test-91101f67-d968-41b8-add9-34689d06f8de openshift-storage.noobaa.io noobaa-default-bucket-class Bound test test-bucket test-bucket-4fe29880-6120-4a40-b81b-0f6b79f98150 ocs-independent-storagecluster-ceph-rgw Bound _________________________________________________________________________________ $ oc describe backingstore noobaa-default-backing-store Spec: s3Compatible: Endpoint: http://10.1.8.96:8080 Secret: Name: rook-ceph-object-user--noobaa-ceph-objectstore-user Namespace: openshift-storage Signature Version: v4 Target Bucket: nb.1594277648909.apps.sagrawal-dc3-ind.qe.rh-ocs.com Type: s3-compatible ______________________________________________________________________________________________ $ oc describe noobaa noobaa Name: noobaa Namespace: openshift-storage Labels: app=noobaa rgw-endpoint=10.1.8.96_8080 Annotations: <none> API Version: noobaa.io/v1alpha1 Kind: NooBaa Metadata: Creation Timestamp: 2020-07-09T06:46:37Z Finalizers: noobaa.io/graceful_finalizer Generation: 2 Manager: noobaa-operator Operation: Update Time: 2020-07-09T11:27:18Z Owner References: API Version: ocs.openshift.io/v1 Block Owner Deletion: true Controller: true Kind: StorageCluster Name: ocs-independent-storagecluster UID: a36749d5-35d5-4900-b291-50a9843f4f72 Resource Version: 1161170 Self Link: /apis/noobaa.io/v1alpha1/namespaces/openshift-storage/noobaas/noobaa UID: 57ad46ee-ae48-428f-9d00-cdb9777c3361 Spec: Affinity: Cleanup Policy: Core Resources: Limits: Cpu: 1 Memory: 4Gi Requests: Cpu: 1 Memory: 4Gi Db Image: registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:3292de73cb0cd935cb20118bd86e9ddbe8ff8c0a8d171cf712b46a0dbd54e169 Db Resources: Limits: Cpu: 500m Memory: 500Mi Requests: Cpu: 500m Memory: 500Mi Db Storage Class: ocs-independent-storagecluster-ceph-rbd Db Volume Resources: Requests: Storage: 50Gi Endpoints: Max Count: 1 Min Count: 1 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Image: quay.io/rhceph-dev/mcg-core@sha256:86d27e39c293f14da564b7edac3804b6afb09490d24ab01b66d315f12ca7141b Pv Pool Default Storage Class: ocs-independent-storagecluster-ceph-rbd Tolerations: Effect: NoSchedule Key: node.ocs.openshift.io/storage Operator: Equal Value: true Status: Accounts: Admin: Secret Ref: Name: noobaa-admin Namespace: openshift-storage Actual Image: quay.io/rhceph-dev/mcg-core@sha256:86d27e39c293f14da564b7edac3804b6afb09490d24ab01b66d315f12ca7141b Conditions: Last Heartbeat Time: 2020-07-09T06:46:38Z Last Transition Time: 2020-07-09T11:27:18Z Message: noobaa operator completed reconcile - system is ready Reason: SystemPhaseReady Status: True Type: Available Last Heartbeat Time: 2020-07-09T06:46:38Z Last Transition Time: 2020-07-09T11:27:18Z Message: noobaa operator completed reconcile - system is ready Reason: SystemPhaseReady Status: False Type: Progressing Last Heartbeat Time: 2020-07-09T06:46:38Z Last Transition Time: 2020-07-09T06:46:38Z Message: noobaa operator completed reconcile - system is ready Reason: SystemPhaseReady Status: False Type: Degraded Last Heartbeat Time: 2020-07-09T06:46:38Z Last Transition Time: 2020-07-09T11:27:18Z Message: noobaa operator completed reconcile - system is ready Reason: SystemPhaseReady Status: True Type: Upgradeable Endpoints: Ready Count: 1 Virtual Hosts: s3.openshift-storage.svc Observed Generation: 2 Phase: Ready Readme: Welcome to NooBaa! ----------------- NooBaa Core Version: 5.5.0-9089fd4 NooBaa Operator Version: 2.3.0 Lets get started: 1. Connect to Management console: Read your mgmt console login information (email & password) from secret: "noobaa-admin". kubectl get secret noobaa-admin -n openshift-storage -o json | jq '.data|map_values(@base64d)' Open the management console service - take External IP/DNS or Node Port or use port forwarding: kubectl port-forward -n openshift-storage service/noobaa-mgmt 11443:443 & open https://localhost:11443 2. Test S3 client: kubectl port-forward -n openshift-storage service/s3 10443:443 & NOOBAA_ACCESS_KEY=$(kubectl get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_ACCESS_KEY_ID|@base64d') NOOBAA_SECRET_KEY=$(kubectl get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_SECRET_ACCESS_KEY|@base64d') alias s3='AWS_ACCESS_KEY_ID=$NOOBAA_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$NOOBAA_SECRET_KEY aws --endpoint https://localhost:10443 --no-verify-ssl s3' s3 ls Services: Service Mgmt: External DNS: https://noobaa-mgmt-openshift-storage.apps.sagrawal-dc3-ind.qe.rh-ocs.com Internal DNS: https://noobaa-mgmt.openshift-storage.svc:443 Internal IP: https://172.30.140.203:443 Node Ports: https://10.70.60.44:32708 Pod Ports: https://10.129.2.32:8443 serviceS3: External DNS: https://s3-openshift-storage.apps.sagrawal-dc3-ind.qe.rh-ocs.com Internal DNS: https://s3.openshift-storage.svc:443 Internal IP: https://172.30.85.239:443 Node Ports: https://10.70.60.44:31898 Pod Ports: https://10.129.2.33:6443 Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754