Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1966231

Summary: Attempt to remove pv-pool based noobaa-default-backing-store fails and makes this pool stuck in Rejected state
Product: OpenShift Container Platform Reporter: Martin Bukatovic <mbukatov>
Component: Console Storage PluginAssignee: Vineet <vbadrina>
Status: CLOSED NOTABUG QA Contact: Raz Tamir <ratamir>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, nbecker, nthomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-10 11:12:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1876633    
Attachments:
Description Flags
Default backing store stuck in Deleting phase none

Description Martin Bukatovic 2021-05-31 17:19:47 UTC 
This bug was initially created as a copy of Bug #1876633

I am copying this bug because: dev team indicated that to fix the problem, OCS plugin in OCP Console needs to be updated to use validation framework. See:

- https://bugzilla.redhat.com/show_bug.cgi?id=1876633#c12
- https://bugzilla.redhat.com/show_bug.cgi?id=1876633#c15

Description of problem
======================

When one tries to remove BackingStore hosted on a pv-pool, such attempt fails
on:

```
DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached
```

With the backingstore stuck in Rejected state.

Version-Release number of selected component
============================================

OCP 4.5.0-0.ci-2020-09-04-161649
OCS 4.5.0-546.ci

Full version report
-------------------

cluster channel: stable-4.5
cluster version: 4.5.0-0.ci-2020-09-04-161649
cluster image: registry.svc.ci.openshift.org/ocp/release@sha256:2664dc11a62754b04abb8d9bb67113af87084c31dba7495cd65f18e6a3a9a507

storage namespace openshift-cluster-storage-operator
image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:4dbc8466b0f384078286dc9462bf60e4185ac30fe434d04a6988e3bfbb0a605b
 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:4dbc8466b0f384078286dc9462bf60e4185ac30fe434d04a6988e3bfbb0a605b
image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:e69949554a71f58b02984e14eecbad4a79812c7295cac1dda1382751b5bf818f
 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:e69949554a71f58b02984e14eecbad4a79812c7295cac1dda1382751b5bf818f
image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:29f8813cfe44206c2d66311e33075afb210f923fb6bf2c9f8f49e88c0b6e6da6
 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:29f8813cfe44206c2d66311e33075afb210f923fb6bf2c9f8f49e88c0b6e6da6

storage namespace openshift-kube-storage-version-migrator
image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:ead86cb945a675f34fef0fc468a1b76fe2c26302c8fd9f830c00ead7852596c9
 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:ead86cb945a675f34fef0fc468a1b76fe2c26302c8fd9f830c00ead7852596c9

storage namespace openshift-kube-storage-version-migrator-operator
image registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:52a9da86b9b533da8355e6ef2183e83f12413596abfe1984ff903360eb01dfbe
 * registry.svc.ci.openshift.org/ocp/4.5-2020-09-04-161649@sha256:52a9da86b9b533da8355e6ef2183e83f12413596abfe1984ff903360eb01dfbe

storage namespace openshift-storage
image quay.io/rhceph-dev/cephcsi@sha256:fa003ab56d59653b4143cd78997677854d227f38492e4f3ddbcd0a6262381494
 * quay.io/rhceph-dev/cephcsi@sha256:a2e07bceff940cac650abca0f5eff13933fd8d147c6585d80bcb901f038614f3
image registry.redhat.io/openshift4/ose-csi-driver-registrar@sha256:424566e4a110a9fc5964f87e22ee6f87a00db2b00636780461bc5e234ca4e6e7
 * registry.redhat.io/openshift4/ose-csi-driver-registrar@sha256:424566e4a110a9fc5964f87e22ee6f87a00db2b00636780461bc5e234ca4e6e7
image registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:2d89d6d1b9bc0f8a3603bb68859617cbc9a04584933f5534362ca68ab590e8ce
 * registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:27945464c6dd60bde78052294d456d0a1d6978ec095dfc5d14660b6fb2c0b532
image registry.redhat.io/openshift4/ose-csi-external-provisioner-rhel7@sha256:5d1a62e07f844d6c55a4983aeb1326c474622b0a5c886e897d1441b3ba74daa6
 * registry.redhat.io/openshift4/ose-csi-external-provisioner-rhel7@sha256:2a0b6ed5bc6ee19f05b4d312a9017c831b280933aa8d2db5f9216139512f44ae
image registry.redhat.io/openshift4/ose-csi-external-resizer-rhel7@sha256:279a12fb2095c7c7f7429135317c53a3f821d6a5a7b89b2f49fc4f84d5cfba42
 * registry.redhat.io/openshift4/ose-csi-external-resizer-rhel7@sha256:279a12fb2095c7c7f7429135317c53a3f821d6a5a7b89b2f49fc4f84d5cfba42
image quay.io/rhceph-dev/mcg-core@sha256:fa9ab8465d698823e8eaa41f12384fb420819aa6c0849cbadda5e14d46495d27
 * quay.io/rhceph-dev/mcg-core@sha256:e421dbc06483936da690d775b7eeef157d52336559179ace828a8e31a91d9115
image registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:ba74027bb4b244df0b0823ee29aa927d729da33edaa20ebdf51a2430cc6b4e95
 * registry.redhat.io/rhscl/mongodb-36-rhel7@sha256:ba74027bb4b244df0b0823ee29aa927d729da33edaa20ebdf51a2430cc6b4e95
image quay.io/rhceph-dev/mcg-operator@sha256:b38f1b44077e2f39ccc32486d517cbf0ccbeda93d3895a693f853756b6e0d6c0
 * quay.io/rhceph-dev/mcg-operator@sha256:2cd43f974f68654a7d7380f802f63ebd5ea1963d8b097e3f728da57bdcc04861
image quay.io/rhceph-dev/ocs-operator@sha256:587772f3d8fa2712c89d80ba2af62b2eaa743eb0048368ad8c4ba1c9fb2f30b1
 * quay.io/rhceph-dev/ocs-operator@sha256:587772f3d8fa2712c89d80ba2af62b2eaa743eb0048368ad8c4ba1c9fb2f30b1
image quay.io/rhceph-dev/rhceph@sha256:eafd1acb0ada5d7cf93699056118aca19ed7a22e4938411d307ef94048746cc8
 * quay.io/rhceph-dev/rhceph@sha256:3def885ad9e8440c5bd6d5c830dafdd59edf9c9e8cce0042b0f44a5396b5b0f6
image quay.io/rhceph-dev/rook-ceph@sha256:a31d1ad207868fd9e4aa98133fd6c3b8addf03b5aefa837143c758abd3b033b6
 * quay.io/rhceph-dev/rook-ceph@sha256:071e020ab4b628a86275869dd5e45ddc15e7ca531eedb318d7da9e450cdf6ed7

How reproducible
================

1/1

Steps to Reproduce
==================

1. Install OCP/OCS cluster on GCP (this is the most simple way to get pv-pool
   backing store out of the box)
2. Go to OCP Console, navigate to list of backing stores
3. Locate noobaa-default-backing-store and use 3-dots menu to delete it

Actual results
==============

The noobaa-default-backing-store is not deleted, because of the following
error:

```
DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached
```

And remains in Rejected state.

The same state is visible via noobaa cli (so it's not reporting problem in OCP
Console):

```
$ noobaa backingstore list -n openshift-storage
NAME                           TYPE                   TARGET-BUCKET                                PHASE      AGE
gcp-backing-store              google-cloud-storage   mbukatov-2020-09-07-noobaa-backing-store-1   Ready      46m23s   
noobaa-default-backing-store   pv-pool                                                             Rejected   59m43s
```

Expected results
================

The noobaa-default-backing-store is not deleted, because of the following
error:

```
DeletePoolAPI cannot complete because pool "noobaa-default-backing-store" has buckets attached
```

The removal is not even started and so the backing store remains in Ready
state.

Additional info
===============

This is problematic because the removal got stuck somewhere in between, leaving
the customer to figure out what to do now with such rejected backing store.

The check which disallows the removal should either pass and then the backing
store is deleted, or fail and then the removal is not started at all.
Comment 2 Vineet 2021-06-07 10:33:23 UTC 
Created attachment 1789199 [details]
Default backing store stuck in Deleting phase
Comment 4 Nishanth Thomas 2021-06-09 08:09:58 UTC 
@mbukatov
Comment 5 Martin Bukatovic 2021-06-10 11:12:19 UTC 
(In reply to Nishanth Thomas from comment #4)
> @mbukatov

Ack. Thanks for the evaluation. So I see that the conclusion is that
there is nothing to be fixed in UI for this usecase. The fix needs
to be done in backend.

Closing this as a not a bug.