Created attachment 1701378 [details] UI params Description of problem (please be detailed as possible and provide log snippests): ---------------------------------------------------------------------- While verifying the fix for Bug 1820297, we confirmed that backingstore create command ignores SC "openshift-storage.noobaa.io" which is an OBC only SC. But, it does not ignore the other OBC only Storageclass, i.e. "ocs-storagecluster-ceph-rgw" during PVC backed backingstore. Observations: ================= 1. The PVC and POD are in pending state as ocs-storagecluster-ceph-rgw is also a non-PVC based ool1-noobaa-pvc-2437e4c0 Pending ocs-storagecluster-ceph-rgw 8m12s pool1-noobaa-pvc-438f4e5e Pending ocs-storagecluster-ceph-rgw 8m13s pool1-noobaa-pod-2437e4c0 0/1 Pending 0 7m12s <none> <none> <none> <none> pool1-noobaa-pod-438f4e5e 0/1 Pending 0 7m12s <none> <none> <none> <none> More information in Additional section openshift-storage.noobaa.io Version of all relevant components (if applicable): ---------------------------------------------------------------------- cluster VMware+RHCOS, cluster channel: stable-4.5 OCP cluster version: 4.5.0-0.nightly-2020-07-14-213353 OCS = ocs-operator.v4.5.0-487.ci INFO[0000] CLI version: 2.3.0 INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0-rc3 INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0 INFO[0000] Namespace: openshift-storage Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? ---------------------------------------------------------------------- It leaves a Backingstore in Rejected state Is there any workaround available to the best of your knowledge? ---------------------------------------------------------------------- Not sure Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? ---------------------------------------------------------------------- 3 Can this issue reproducible? ---------------------------------------------------------------------- Yes Can this issue reproduce from the UI? ---------------------------------------------------------------------- Yes If this is a regression, please provide more details to justify this: ---------------------------------------------------------------------- Steps to Reproduce: ---------------------------------------------------------------------- From UI =========== 1. Create an OCP 4.5 + OCS 4.5 cluster on VMware/BM 2. Navigate to Installed Operators->Openshift COntainer Storage Operator-> Backing Store 3. Click on Create Backingstore Name = Add a name for the backingstore Provider = PVC StorageClass = Select the incorrect OBC only type SC = ocs-storagecluster-ceph-rgw 4. check PVC, pod and backingstore .. First 2 would be in pending and Backingstore will ultimately be in rejected state From CLI ========== <noobaa-cli> backingstore create pv-pool pool1 --num-volumes 2 --pv-size-gb 16 --storage-class ocs-storagecluster-ceph-rgw Actual results: ---------------------------------------------------------------------- the check for "ignoring OBC only SC" works for Sc=openshift-storage.noobaa.io but NOT for SC=ocs-storagecluster-ceph-rgw Expected results: ---------------------------------------------------------------------- backingstore creation should not proceed on using the OBC only SC =ocs-storagecluster-ceph-rgw Additional info: ---------------------------------------------------------------------- The fix for Bug 1820297 Works well for non-existent SC =========================================================== [nberry@localhost auth]$ /usr/local/bin/nooba-cli backingstore create pv-pool pool1 --num-volumes 2 --pv-size-gb 16 --storage-class jacky.noobaa INFO[0005] ✅ Exists: NooBaa "noobaa" INFO[0006] ❌ Not Found: StorageClass "jacky.noobaa" FATA[0006] ❌ Could not get StorageClass "jacky.noobaa" for system in namespace "openshift-storage" ______________________________________ Works good for openshift-storage.noobaa.io =============================================== [nberry@localhost auth]$ /usr/local/bin/nooba-cli backingstore create pv-pool pool2 --num-volumes 1 --pv-size-gb 16 --storage-class openshift-storage.noobaa.io INFO[0005] ✅ Exists: NooBaa "noobaa" INFO[0006] ✅ Exists: StorageClass "openshift-storage.noobaa.io" FATA[0006] ❌ Could not set StorageClass "openshift-storage.noobaa.io" for system in namespace "openshift-storage" - as this class reserved for obc only [nberry@localhost auth]$ __________________________________________________________ fails to ignore the other OBC only SC ocs-storagecluster-ceph-rgw ======================================================================= [nberry@localhost auth]$ /usr/local/bin/nooba-cli backingstore create pv-pool pool1 --num-volumes 2 --pv-size-gb 16 --storage-class ocs-storagecluster-ceph-rgw INFO[0005] ✅ Exists: NooBaa "noobaa" INFO[0006] ✅ Exists: StorageClass "ocs-storagecluster-ceph-rgw" INFO[0007] ✅ Created: BackingStore "pool1" INFO[0009] ✅ Created: Secret "backing-store-pv-pool-pool1" INFO[0009] INFO[0009] NOTE: INFO[0009] - This command has finished applying changes to the cluster. INFO[0009] - From now on, it only loops and reads the status, to monitor the operator work. INFO[0009] - You may Ctrl-C at any time to stop the loop and watch it manually. INFO[0009] INFO[0009] BackingStore Wait Ready: INFO[0009] ⏳ BackingStore "pool1" Phase is "Creating": TemporaryError BackingStore Still didn't connect all requested pods. 2 from 2 are pending INFO[0013] ⏳ BackingStore "pool1" Phase is "Creating": TemporaryError BackingStore Still didn't connect all requested pods. 2 from 2 are pending INFO[0016] ⏳ BackingStore "pool1" Phase is "Creating": TemporaryError BackingStore Still didn't connect all requested pods. 2 from 2 are pending INFO[0019] ⏳ BackingStore "pool1" Phase is "Creating": TemporaryError BackingStore Still didn't connect all requested pods. 2 from 2 are pending INFO[0022] ⏳ BackingStore "pool1" Phase is "Creating": TemporaryError BackingStore Still didn't connect all requested pods. 2 from 2 are pending ^C ______________________________________________________________________________________
My bad.. Even though I downloaded the latest noobaa-binary from noobaa-operator pod and copied in /usr/local/bin, I still ended up using the old noobaa-CLI Thanks @Jacky for looking into it and making me realise my mistake :) The fix works on below builds: OCS = ocs-operator.v4.5.0-494.ci OCP = Cluster version is 4.5.0-0.nightly-2020-07-22-074214 From CLI ============= 1. oc exec noobaa-operator-695884dc65-kldm5 -- cat /usr/local/bin/noobaa-operator >>noobaa-new ; chmod 777 noobaa-new 2. $ date --utc; ./noobaa-new backingstore create pv-pool rgw-new1 --num-volumes 2 --pv-size-gb 16 --storage-class ocs-storagecluster-ceph-rgw Wed Jul 22 14:53:26 UTC 2020 INFO[0005] ✅ Exists: NooBaa "noobaa" INFO[0006] ✅ Exists: StorageClass "ocs-storagecluster-ceph-rgw" FATA[0006] ❌ Could not set StorageClass "ocs-storagecluster-ceph-rgw" for system in namespace "openshift-storage" - as this class reserved for obc only 3. $ ./noobaa-new version INFO[0000] CLI version: 2.3.0 INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0 INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0 From UI =========== UI did not reject the OBC only SC. A bZ is already raised for tracking the issue. Bug 1858179 - In OCS UI: Backingstore creation should reject Provider :PVC with OBC only SC(ocs-storagecluster-ceph-rgw & openshift-storage.noobaa.io) Hence, moving the BZ to verified state. Again, apologies for causing so much noise.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754