Bug 1858385
Summary: | POD and PVC stay back in terminating state on deleting a UI-based-Backingstore (Provider:PVC) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat OpenShift Container Storage | Reporter: | Neha Berry <nberry> | ||||||
Component: | Multi-Cloud Object Gateway | Assignee: | Ohad <omitrani> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Neha Berry <nberry> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 4.5 | CC: | ebenahar, etamir, nbecker, ocs-bugs, omitrani, ratamir | ||||||
Target Milestone: | --- | Keywords: | AutomationBackLog | ||||||
Target Release: | OCS 4.5.0 | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | v4.5.0-30.ci | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2020-09-15 10:18:18 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Neha Berry
2020-07-17 18:11:51 UTC
Proposing as a blocker for 4.5 as PV backingstore is the default fallback and as it is going to be fully supported in this version. It seems I found the reason for the problem The Backing store created from the UI is missing some metadata, the noobaa finalizer, the noobaa label and an ownerRef. For the sake of this bug the problem is the missing finalizer. Without it the deleting the backing store will take effect immediately and the noobaa operator will not have the opertunity to run the proper steps to allow deletion of the resources behind the backing store. Usualy we add the finalizer during our reconcile loop, but it seems we are not handling an edge case manifest in this bug. I will update the code to handle this edge case, will issue an upstream PR, and will update here A PR with a fix was issued on the upstream project (see links section) Created attachment 1703083 [details] backingstore yamls Verified the deletion of Backingstore created from UI is successful in OCS build - ocs-operator.v4.5.0-508.ci OCP build - 4.5.0-0.nightly-2020-07-30-213620 $ ../nooba508 version INFO[0000] CLI version: 2.3.0 INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0 INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0 tested on External Mode cluster(OCP on vmware) and Internam Mode cluster(AWS) 1. Created a BS with Provider:PVC from CLI with noobaa CLI </usr/local/bin/nooba-cli> backingstore create pv-pool nb-bs2-cli --num-volumes 1 --pv-size-gb 16 --storage-class ocs-storagecluster-ceph-rbd |tee cli-BS-creation.txt 2. Created a BS with Provider:PVC via UI. 3. Checked both the backingstores and their corresponding PVC and PODs are created. 4. Deleted the above 2 Backinsgores, either from CLI or UI. (I tested both and observed same behavior) Observation: Backingstore deletion -Success POd and PVC Deletion - success ___________________________________________________________________ Checked the BS created from UI and it had following new things: (Attached in the BZ) 1. Labels 2. Finalizers @Ohad But the UI based backingstore still doesnt have any OwnerReference set(Comment#4). Is this expected? ____________________________________________________________________________ After creation of 2 BS -------------------------- ========CSV ====== NAME DISPLAY VERSION REPLACES PHASE ocs-operator.v4.5.0-508.ci OpenShift Container Storage 4.5.0-508.ci Succeeded -------------- =======PODS ====== nb-bs2-cli-noobaa-pod-eb3d5c27 1/1 Running 0 14m 10.129.3.21 ip-10-0-135-115.us-east-2.compute.internal <none> <none> nb-ui-bs1-noobaa-pod-e51a27c7 1/1 Running 0 14m 10.131.0.251 ip-10-0-180-109.us-east-2.compute.internal <none> <none> -------------- ======= PVC ========== db-noobaa-db-0 Bound pvc-b3cecc87-2537-4d7d-93d2-276e4f21df7d 50Gi RWO ocs-storagecluster-ceph-rbd 4h16m nb-bs2-cli-noobaa-pvc-eb3d5c27 Bound pvc-e3530a31-a2f9-4e87-b202-33b422ef2c93 16Gi RWO ocs-storagecluster-ceph-rbd 14m nb-ui-bs1-noobaa-pvc-e51a27c7 Bound pvc-99c20328-0dfa-415f-aba1-c0f37eb8f532 50Gi RWO ocs-storagecluster-ceph-rbd 14m -------------- ======= backingstore ========== NAME TYPE PHASE AGE nb-bs2-cli pv-pool Ready 14m nb-ui-bs1 pv-pool Ready 14m After deletion of the 2 BS --------------------------- $ oc delete backingstore nb-ui-bs1 backingstore.noobaa.io "nb-ui-bs1" deleted $ oc delete backingstore nb-bs2-cli backingstore.noobaa.io "nb-bs2-cli" deleted _ $ oc get pods -o wide -n openshift-storage|grep nobbaa-pod [nberry@localhost akrai]$ oc get pvc -o wide -n openshift-storage|grep nobbaa-pvc [nberry@localhost akrai]$ oc get backingstore -o wide -n openshift-storage NAME TYPE PHASE AGE noobaa-default-backing-store aws-s3 Ready 4h34m __________________________________________________________________ --- apiVersion: noobaa.io/v1alpha1 kind: BackingStore metadata: creationTimestamp: "2020-07-31T09:13:38Z" finalizers: - noobaa.io/finalizer generation: 2 labels: app: noobaa managedFields: - apiVersion: noobaa.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:spec: .: {} f:pvPool: .: {} f:numVolumes: {} f:resources: .: {} f:requests: .: {} f:storage: {} f:storageClass: {} f:type: {} manager: Mozilla operation: Update time: "2020-07-31T09:13:38Z" - apiVersion: noobaa.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: {} f:labels: .: {} f:app: {} f:spec: f:pvPool: f:secret: {} f:status: .: {} f:conditions: {} f:mode: .: {} f:modeCode: {} f:timeStamp: {} f:phase: {} manager: noobaa-operator operation: Update time: "2020-07-31T09:20:27Z" name: nb-ui-bs1 namespace: openshift-storage resourceVersion: "225075" selfLink: /apis/noobaa.io/v1alpha1/namespaces/openshift-storage/backingstores/nb-ui-bs1 uid: 0f3a3cb9-a622-49f9-922e-872be1a6c8d8 spec: pvPool: numVolumes: 1 resources: requests: storage: 50Gi secret: {} ___________________________________________________________________________________________________________________ One small query on Ohad (In reply to Ohad from comment #4) > It seems I found the reason for the problem > > The Backing store created from the UI is missing some metadata, the noobaa > finalizer, the noobaa label and an ownerRef. Hi Ohad, is there a plan to add OwnerReference for the BS created from UI in next release ? that is still a difference we can see between the CLI and UI based yamls, hence wanted to confirm. Verified based on Comment#9. If a new bug for OwnerReference is needed, shall raise one based on Ohad's reply in Comment#10 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754 |