This bug was initially created as a copy of Bug #1876633 I am copying this bug because the original bug was not properly fixed, and it's meaning changed a bit, so that we still need to have a bug open for the original problem so that it can be properly fixed: - as noted in https://bugzilla.redhat.com/show_bug.cgi?id=1876633#c18 the original bug #1876633 is representing cli only hack - proper approach how to address the original problem is noted in https://bugzilla.redhat.com/show_bug.cgi?id=1966231#c8 Description of problem ====================== When one tries to remove BackingStore hosted on a pv-pool, such attempt fails on: ``` DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached ``` With the backingstore stuck in Rejected state. Version-Release number of selected component ============================================ OCP 4.5.0-0.ci-2020-09-04-161649 OCS 4.5.0-546.ci Full version report ------------------- cluster channel: stable-4.5 cluster version: 4.5.0-0.ci-2020-09-04-161649 cluster image: registry.svc.ci.openshift.org/ocp/release@sha256:2664dc11a62754b04abb8d9bb67113af87084c31dba7495cd65f18e6a3a9a507 storage namespace openshift-cluster-storage-operator image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:4dbc8466b0f384078286dc9462bf60e4185ac30fe434d04a6988e3bfbb0a605b * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:4dbc8466b0f384078286dc9462bf60e4185ac30fe434d04a6988e3bfbb0a605b image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:e69949554a71f58b02984e14eecbad4a79812c7295cac1dda1382751b5bf818f * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:e69949554a71f58b02984e14eecbad4a79812c7295cac1dda1382751b5bf818f image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:29f8813cfe44206c2d66311e33075afb210f923fb6bf2c9f8f49e88c0b6e6da6 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:29f8813cfe44206c2d66311e33075afb210f923fb6bf2c9f8f49e88c0b6e6da6 storage namespace openshift-kube-storage-version-migrator image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:ead86cb945a675f34fef0fc468a1b76fe2c26302c8fd9f830c00ead7852596c9 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:ead86cb945a675f34fef0fc468a1b76fe2c26302c8fd9f830c00ead7852596c9 storage namespace openshift-kube-storage-version-migrator-operator image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:52a9da86b9b533da8355e6ef2183e83f12413596abfe1984ff903360eb01dfbe * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:52a9da86b9b533da8355e6ef2183e83f12413596abfe1984ff903360eb01dfbe storage namespace openshift-storage image quay.io/rhceph-dev/cephcsi@sha256:fa003ab56d59653b4143cd78997677854d227f38492e4f3ddbcd0a6262381494 * quay.io/rhceph-dev/cephcsi@sha256:a2e07bceff940cac650abca0f5eff13933fd8d147c6585d80bcb901f038614f3 image registry.redhat.io/openshift4/ose-csi-driver-registrar@sha256:424566e4a110a9fc5964f87e22ee6f87a00db2b00636780461bc5e234ca4e6e7 * registry.redhat.io/openshift4/ose-csi-driver-registrar@sha256:424566e4a110a9fc5964f87e22ee6f87a00db2b00636780461bc5e234ca4e6e7 image registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:2d89d6d1b9bc0f8a3603bb68859617cbc9a04584933f5534362ca68ab590e8ce * registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:27945464c6dd60bde78052294d456d0a1d6978ec095dfc5d14660b6fb2c0b532 image registry.redhat.io/openshift4/ose-csi-external-provisioner-rhel7@sha256:5d1a62e07f844d6c55a4983aeb1326c474622b0a5c886e897d1441b3ba74daa6 * registry.redhat.io/openshift4/ose-csi-external-provisioner-rhel7@sha256:2a0b6ed5bc6ee19f05b4d312a9017c831b280933aa8d2db5f9216139512f44ae image registry.redhat.io/openshift4/ose-csi-external-resizer-rhel7@sha256:279a12fb2095c7c7f7429135317c53a3f821d6a5a7b89b2f49fc4f84d5cfba42 * registry.redhat.io/openshift4/ose-csi-external-resizer-rhel7@sha256:279a12fb2095c7c7f7429135317c53a3f821d6a5a7b89b2f49fc4f84d5cfba42 image quay.io/rhceph-dev/mcg-core@sha256:fa9ab8465d698823e8eaa41f12384fb420819aa6c0849cbadda5e14d46495d27 * quay.io/rhceph-dev/mcg-core@sha256:e421dbc06483936da690d775b7eeef157d52336559179ace828a8e31a91d9115 image registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:ba74027bb4b244df0b0823ee29aa927d729da33edaa20ebdf51a2430cc6b4e95 * registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:ba74027bb4b244df0b0823ee29aa927d729da33edaa20ebdf51a2430cc6b4e95 image quay.io/rhceph-dev/mcg-operator@sha256:b38f1b44077e2f39ccc32486d517cbf0ccbeda93d3895a693f853756b6e0d6c0 * quay.io/rhceph-dev/mcg-operator@sha256:2cd43f974f68654a7d7380f802f63ebd5ea1963d8b097e3f728da57bdcc04861 image quay.io/rhceph-dev/ocs-operator@sha256:587772f3d8fa2712c89d80ba2af62b2eaa743eb0048368ad8c4ba1c9fb2f30b1 * quay.io/rhceph-dev/ocs-operator@sha256:587772f3d8fa2712c89d80ba2af62b2eaa743eb0048368ad8c4ba1c9fb2f30b1 image quay.io/rhceph-dev/rhceph@sha256:eafd1acb0ada5d7cf93699056118aca19ed7a22e4938411d307ef94048746cc8 * quay.io/rhceph-dev/rhceph@sha256:3def885ad9e8440c5bd6d5c830dafdd59edf9c9e8cce0042b0f44a5396b5b0f6 image quay.io/rhceph-dev/rook-ceph@sha256:a31d1ad207868fd9e4aa98133fd6c3b8addf03b5aefa837143c758abd3b033b6 * quay.io/rhceph-dev/rook-ceph@sha256:071e020ab4b628a86275869dd5e45ddc15e7ca531eedb318d7da9e450cdf6ed7 How reproducible ================ 1/1 Steps to Reproduce ================== 1. Install OCP/OCS cluster on GCP (this is the most simple way to get pv-pool backing store out of the box) 2. Go to OCP Console, navigate to list of backing stores 3. Locate noobaa-default-backing-store and use 3-dots menu to delete it Actual results ============== The noobaa-default-backing-store is not deleted, because of the following error: ``` DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached ``` And remains in Rejected state. The same state is visible via noobaa cli (so it's not reporting problem in OCP Console): ``` $ noobaa backingstore list -n openshift-storage NAME TYPE TARGET-BUCKET PHASE AGE gcp-backing-store google-cloud-storage mbukatov-2020-09-07-noobaa-backing-store-1 Ready 46m23s noobaa-default-backing-store pv-pool Rejected 59m43s ``` Expected results ================ The noobaa-default-backing-store is not deleted, because of the following error: ``` DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached ``` The removal is not even started and so the backing store remains in Ready state. Additional info =============== This is problematic because the removal got stuck somewhere in between, leaving the customer to figure out what to do now with such rejected backing store. The check which disallows the removal should either pass and then the backing store is deleted, or fail and then the removal is not started at all.
Fixing state of this bug: - dropping "CLI only" from the title, I must have overlooked that when cloning - moving it back to NEW, if we want to declare it as RFE and not track it in bugzilla, we would need to open a JIRA for the feature (that said, I'm not fully convinced that this is "a feature")
This is not an RFE, but a bug which depends on a feature which is not yet present. Let's keep it open, and for better tracking, mark the validation feature as a blocker for this bug.
Hi Martin, Few questions: - Is it reproducible? - I see OCS 4.5. do we see it on later versions too? - Any workaround available? - In the last comment you are saying that it depends on a feature that is not yet present - what is that feature? - Is this scenario was tested as part of a feature for 4.8 release? Thanks
(In reply to Raz Tamir from comment #10) > - Is it reproducible? Yes, but I haven't tried it since I stopped working on this and don't plan to retry. > - I see OCS 4.5. do we see it on later versions too? I assume yes. I haven't tried myself. > - Any workaround available? I'm not aware of any workaround. > - In the last comment you are saying that it depends on a feature that is > not yet present - what is that feature? I don't know myself as well. You would need to ask Eran or Nimrod. > - Is this scenario was tested as part of a feature for 4.8 release? No. The main problem here is that original BZ 1876633 was mishandled by development team, which wastes time for everyone. See: https://bugzilla.redhat.com/show_bug.cgi?id=1876633#c28
Testing with GCP cluster: OCP 4.10.0-0.nightly-2022-02-15-041303 ODF 4.10.0-156 And I see that when one attempts to delete `backingstore/noobaa-default-backing-store`, the action is rejected: ``` $ oc delete backingstore/noobaa-default-backing-store -n openshift-storage Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "noobaa-default-backing-store" in "IN_USE" state ``` The same applies to the attempt to remove the backing store via OCP Console, which reports: ``` An error occurred admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "noobaa-default-backing-store" in "IN_USE" state ```
Created attachment 1861440 [details] verification screenshot: removal of backingstore/noobaa-default-backing-store is rejected
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.10.0 enhancement, security & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1372