Bug 2186189

Summary: [RDR] [ACM-Tracker] App-set based workload remains stuck on deletion
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Aman Agrawal <amagrawa>
Component: odf-drAssignee: Benamar Mekhissi <bmekhiss>
odf-dr sub component: ramen QA Contact: Aman Agrawal <amagrawa>
Status: VERIFIED --- Docs Contact:
Severity: urgent    
Priority: unspecified CC: ebenahar, gshanmug, kseeger, muagarwa, odf-bz-bot, srangana
Version: 4.13   
Target Milestone: ---   
Target Release: ODF 4.13.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aman Agrawal 2023-04-12 11:22:46 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

Deletion was done after reporting https://bugzilla.redhat.com/show_bug.cgi?id=2185953
Workload is currently ruuning on C1.

Version of all relevant components (if applicable):
ACM 2.7.2
ODF 4.13.0-121.stable


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Navigate to ACM UI and selet the app-set based workloads and delete from kebab menu option.
2. Ensure that "Remove ApplicationSet related resources" was checked while initiating the delete.


Actual results: App-set based workload remains stuck on deletion.

Hub-
amagrawa:~$ date; oc get drpc -o yaml -A
Wednesday 12 April 2023 04:49:40 PM IST
apiVersion: v1
items:
- apiVersion: ramendr.openshift.io/v1alpha1
  kind: DRPlacementControl
  metadata:
    creationTimestamp: "2023-04-03T14:25:38Z"
    finalizers:
    - drpc.ramendr.openshift.io/finalizer
    generation: 5
    labels:
      cluster.open-cluster-management.io/backup: resource
    name: admin-placement-drpc
    namespace: openshift-gitops
    resourceVersion: "34773991"
    uid: d1f0d7cf-a444-4ab0-82a3-bbad4bde540c
  spec:
    action: Failover
    drPolicyRef:
      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRPolicy
      name: my-drpolicy-5
    failoverCluster: amagrawa-c1
    placementRef:
      apiVersion: cluster.open-cluster-management.io/v1beta1
      kind: Placement
      name: admin-placement
      namespace: openshift-gitops
    preferredCluster: amagrawa-c1
    pvcSelector:
      matchLabels:
        appname: busybox_app3
  status:
    actionDuration: 24m54.260545194s
    actionStartTime: "2023-04-11T13:07:26Z"
    conditions:
    - lastTransitionTime: "2023-04-11T13:32:20Z"
      message: Cleaned
      observedGeneration: 5
      reason: Success
      status: "True"
      type: PeerReady
    - lastTransitionTime: "2023-04-11T13:29:20Z"
      message: Completed
      observedGeneration: 5
      reason: FailedOver
      status: "True"
      type: Available
    lastUpdateTime: "2023-04-12T11:12:20Z"
    phase: FailedOver
    preferredDecision:
      clusterName: amagrawa-c1
      clusterNamespace: app-busybox-3
    progression: Completed
    resourceConditions:
      conditions:
      - lastTransitionTime: "2023-04-11T13:31:53Z"
        message: PVCs in the VolumeReplicationGroup are ready for use
        observedGeneration: 1
        reason: Ready
        status: "True"
        type: DataReady
      - lastTransitionTime: "2023-04-11T13:31:39Z"
        message: VolumeReplicationGroup is replicating
        observedGeneration: 1
        reason: Replicating
        status: "False"
        type: DataProtected
      - lastTransitionTime: "2023-04-11T13:29:04Z"
        message: Restored PV cluster data
        observedGeneration: 1
        reason: Restored
        status: "True"
        type: ClusterDataReady
      - lastTransitionTime: "2023-04-11T13:31:46Z"
        message: Kube objects protected
        observedGeneration: 1
        reason: Uploaded
        status: "True"
        type: ClusterDataProtected
      resourceMeta:
        generation: 1
        kind: VolumeReplicationGroup
        name: admin-placement-drpc
        namespace: app-busybox-3
        protectedpvcs:
        - busybox-pvc-55
        - busybox-pvc-58
        - busybox-pvc-45
        - busybox-pvc-54
        - busybox-pvc-51
        - busybox-pvc-50
        - busybox-pvc-49
        - busybox-pvc-60
        - busybox-pvc-52
        - busybox-pvc-53
        - busybox-pvc-56
        - busybox-pvc-48
        - busybox-pvc-46
        - busybox-pvc-59
        - busybox-pvc-42
        - busybox-pvc-44
        - busybox-pvc-57
        - busybox-pvc-43
        - busybox-pvc-47
        - busybox-pvc-41
kind: List
metadata:
  resourceVersion: ""

amagrawa:~$ date; oc get drpc -o wide -A
Wednesday 12 April 2023 04:51:01 PM IST
NAMESPACE          NAME                   AGE   PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION           PEER READY
openshift-gitops   admin-placement-drpc   8d    amagrawa-c1        amagrawa-c1       Failover       FailedOver     Completed     2023-04-11T13:07:26Z   24m54.260545194s   True



C2- orphan pods/pvcs remains stuck from BZ 2185953 and not expected to get cleaned. Just stating here as a FYI.

C1-

amagrawa:~$ oc get pods,pvc,vr,vrg -n app-busybox-3
NAME                                   STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/busybox-pvc-41   Terminating   pvc-d0b72be5-22f5-45ba-bbf6-2281fdebefbf   42Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-42   Terminating   pvc-9b9cf55a-a75c-4d30-97d7-4b9ff2722431   81Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-43   Terminating   pvc-dcf4c325-adc5-48bd-8419-e09e6a787a39   28Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-44   Terminating   pvc-e00a4ac9-e813-4a51-8c22-56b8731e4bb7   118Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-45   Terminating   pvc-13f42be9-36c4-414f-b492-0c0340b29afa   19Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-46   Terminating   pvc-03235a82-37e8-4d6c-ad48-470e8e98fdd7   129Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-47   Terminating   pvc-c59917a7-1d32-46f6-a4b1-59855ea47070   43Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-48   Terminating   pvc-5a4f302b-2cad-470f-9b8c-108150635fdf   57Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-49   Terminating   pvc-e28d6929-9446-4b02-bd41-5a24f0a28d2d   89Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-50   Terminating   pvc-8eedf9cb-e49b-487d-a4e1-97a41a30099c   124Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-51   Terminating   pvc-f18cf24a-5b40-490c-994d-b12ed9600a45   95Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-52   Terminating   pvc-5560146e-8134-4e2c-b5b6-368d5294f30c   129Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-53   Terminating   pvc-dc442fe7-d45f-4d95-86d0-2139b71a5e04   51Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-54   Terminating   pvc-36da406f-42c9-4761-8139-0131ea9d951b   30Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-55   Terminating   pvc-ad09eb08-cac7-4902-ba2c-b22acc1c7586   102Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-56   Terminating   pvc-d879c820-f6c2-4a3c-b7e8-2112b703e936   40Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-57   Terminating   pvc-dfbd0f5f-4f7b-4700-80fc-58ae403abc42   146Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-58   Terminating   pvc-86da9780-49c6-4bbf-b7de-053b9886568d   63Gi       RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-59   Terminating   pvc-11f13a73-f21c-467f-b678-52e168471e66   118Gi      RWO            ocs-storagecluster-ceph-rbd   21h
persistentvolumeclaim/busybox-pvc-60   Terminating   pvc-776748d4-ff0e-4a14-9598-31a9ef8019ab   25Gi       RWO            ocs-storagecluster-ceph-rbd   21h

NAME                                                                AGE   VOLUMEREPLICATIONCLASS                  PVCNAME          DESIREDSTATE   CURRENTSTATE
volumereplication.replication.storage.openshift.io/busybox-pvc-41   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-41   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-42   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-42   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-43   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-43   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-44   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-44   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-45   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-45   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-46   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-46   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-47   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-47   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-48   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-48   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-49   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-49   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-50   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-50   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-51   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-51   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-52   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-52   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-53   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-53   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-54   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-54   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-55   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-55   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-56   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-56   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-57   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-57   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-58   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-58   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-59   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-59   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-60   21h   rbd-volumereplicationclass-1625360775   busybox-pvc-60   primary        Primary

NAME                                                               DESIREDSTATE   CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/admin-placement-drpc   primary        Primary


Expected results: Deletion should complete.


Additional info: