2122700 – [RDR] DRPC status shows stale information on subsequent same DR actions

Bug 2122700 - [RDR] DRPC status shows stale information on subsequent same DR actions

Summary: [RDR] DRPC status shows stale information on subsequent same DR actions

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	odf-dr
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	ODF 4.12.0
Assignee:	Benamar Mekhissi
QA Contact:	kmanohar
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-08-30 16:04 UTC by Sidhant Agrawal
Modified:	2023-08-09 17:00 UTC (History)
CC List:	6 users (show)
Fixed In Version:	4.12.0-114
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-02-08 14:06:28 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	RamenDR ramen pull 606	0	None	open	Fix progression when failing over or relocating for the second time	2022-11-14 13:11:39 UTC
Github	red-hat-storage ramen pull 82	0	None	open	Bug 2122700: Fix status updates when changing actions	2022-11-14 22:32:35 UTC

Comment 3 Benamar Mekhissi 2022-11-13 14:48:45 UTC

https://github.com/RamenDR/ramen/pull/606

Comment 12 kmanohar 2023-01-04 13:23:33 UTC

Steps to Reproduce:
1. On a RDR setup, deploy a simple busybox application
2. Perform a Failover action
3. Again perform failover action and observe the drpc status by using the command "oc get drpc -o wide"


Actual results:
DRPC status displays stale information on subsequent and same DR action.

Expected results:
DRPC status shows updated information after each failover/relocate action.

______________________________________________________________________________________________________________________

Verification comments:

Verified on - 4.12.0-152
ACM - 2.7
CEPH version:- ceph version 16.2.10-82.el8cp

before relocate

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION            PEER READY
busybox-drpc   4h17m   kmanohar-clu1                                       Deployed       Completed     2023-01-04T05:22:17Z   3h0m35.785389123s   True


relocate 1:

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   4h24m   kmanohar-clu2                        Relocate       Relocated      Completed     2023-01-04T09:41:07Z   2m49.580970394s   True

relocate 2:

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   4h59m   kmanohar-clu1                        Relocate       Relocated      Completed     2023-01-04T10:15:01Z   3m23.740561449s   True


apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  creationTimestamp: "2023-01-04T05:22:17Z"
  finalizers:
  - drpc.ramendr.openshift.io/finalizer
  generation: 4
  labels:
    app: busybox-sample
    cluster.open-cluster-management.io/backup: resource
  name: busybox-drpc
  namespace: busybox-workloads-1
  resourceVersion: "2205293"
  uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
  action: Relocate
  drPolicyRef:
    name: dr-policy-10m
  placementRef:
    kind: PlacementRule
    name: busybox-placement
    namespace: busybox-workloads-1
  preferredCluster: kmanohar-clu1
  pvcSelector:
    matchLabels:
      appname: busybox
status:
  actionDuration: 3m23.740561449s
  actionStartTime: "2023-01-04T10:15:01Z"
  conditions:
  - lastTransitionTime: "2023-01-04T10:18:24Z"
    message: Completed
    observedGeneration: 4
    reason: Relocated
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-04T10:18:24Z"
    message: Cleaned
    observedGeneration: 4
    reason: Success
    status: "True"
    type: PeerReady
  lastGroupSyncTime: "2023-01-04T10:50:00Z"
  lastUpdateTime: "2023-01-04T10:52:57Z"
  phase: Relocated
  preferredDecision:
    clusterName: kmanohar-clu1
    clusterNamespace: kmanohar-clu1
  progression: Completed
  resourceConditions:
    conditions:
    - lastTransitionTime: "2023-01-04T10:17:36Z"
      message: PVCs in the VolumeReplicationGroup are ready for use
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: DataReady
    - lastTransitionTime: "2023-01-04T10:17:27Z"
      message: VolumeReplicationGroup is replicating
      observedGeneration: 1
      reason: Replicating
      status: "False"
      type: DataProtected
    - lastTransitionTime: "2023-01-04T10:16:56Z"
      message: Restored PV cluster data
      observedGeneration: 1
      reason: Restored
      status: "True"
      type: ClusterDataReady
    - lastTransitionTime: "2023-01-04T10:17:33Z"
      message: Kube objects protected
      observedGeneration: 1
      reason: Uploaded
      status: "True"
      type: ClusterDataProtected
    resourceMeta:
      generation: 1
      kind: VolumeReplicationGroup
      name: busybox-drpc
      namespace: busybox-workloads-1
      protectedpvcs:
      - busybox-pvc-1
      - busybox-pvc-20
      - busybox-pvc-10
      - busybox-pvc-12
      - busybox-pvc-15
      - busybox-pvc-16
      - busybox-pvc-18
      - busybox-pvc-11
      - busybox-pvc-13
      - busybox-pvc-14
      - busybox-pvc-17
      - busybox-pvc-19
      - busybox-pvc-2
      - busybox-pvc-8
      - busybox-pvc-9
      - busybox-pvc-3
      - busybox-pvc-4
      - busybox-pvc-5
      - busybox-pvc-6
      - busybox-pvc-7



relocate 3:

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   5h48m   kmanohar-clu2                        Relocate       Relocated      Completed     2023-01-04T11:03:30Z   2m58.199064636s   True


apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  creationTimestamp: "2023-01-04T05:22:17Z"
  finalizers:
  - drpc.ramendr.openshift.io/finalizer
  generation: 5
  labels:
    app: busybox-sample
    cluster.open-cluster-management.io/backup: resource
  name: busybox-drpc
  namespace: busybox-workloads-1
  resourceVersion: "2222538"
  uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
  action: Relocate
  drPolicyRef:
    name: dr-policy-10m
  placementRef:
    kind: PlacementRule
    name: busybox-placement
    namespace: busybox-workloads-1
  preferredCluster: kmanohar-clu2
  pvcSelector:
    matchLabels:
      appname: busybox
status:
  actionDuration: 2m58.199064636s
  actionStartTime: "2023-01-04T11:03:30Z"
  conditions:
  - lastTransitionTime: "2023-01-04T11:06:28Z"
    message: Completed
    observedGeneration: 5
    reason: Relocated
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-04T11:06:28Z"
    message: Cleaned
    observedGeneration: 5
    reason: Success
    status: "True"
    type: PeerReady
  lastUpdateTime: "2023-01-04T11:06:28Z"
  phase: Relocated
  preferredDecision:
    clusterName: kmanohar-clu1
    clusterNamespace: kmanohar-clu1
  progression: Completed
  resourceConditions:
    conditions:
    - lastTransitionTime: "2023-01-04T11:06:03Z"
      message: PVCs in the VolumeReplicationGroup are ready for use
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: DataReady
    - lastTransitionTime: "2023-01-04T11:05:56Z"
      message: VolumeReplicationGroup is replicating
      observedGeneration: 1
      reason: Replicating
      status: "False"
      type: DataProtected
    - lastTransitionTime: "2023-01-04T11:05:30Z"
      message: Restored PV cluster data
      observedGeneration: 1
      reason: Restored
      status: "True"
      type: ClusterDataReady
    - lastTransitionTime: "2023-01-04T11:06:02Z"
      message: Kube objects protected
      observedGeneration: 1
      reason: Uploaded
      status: "True"
      type: ClusterDataProtected
    resourceMeta:
      generation: 1
      kind: VolumeReplicationGroup
      name: busybox-drpc
      namespace: busybox-workloads-1
      protectedpvcs:
      - busybox-pvc-1
      - busybox-pvc-10
      - busybox-pvc-12
      - busybox-pvc-14
      - busybox-pvc-2
      - busybox-pvc-3
      - busybox-pvc-5
      - busybox-pvc-8
      - busybox-pvc-16
      - busybox-pvc-17
      - busybox-pvc-18
      - busybox-pvc-19
      - busybox-pvc-20
      - busybox-pvc-4
      - busybox-pvc-9
      - busybox-pvc-11
      - busybox-pvc-15
      - busybox-pvc-6
      - busybox-pvc-7
      - busybox-pvc-13


____________________________________________________________________

failover 1

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   6h16m   kmanohar-clu1      kmanohar-clu2     Failover       FailedOver     Completed     2023-01-04T11:33:13Z   3m45.742949056s   True



apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  creationTimestamp: "2023-01-04T05:22:17Z"
  finalizers:
  - drpc.ramendr.openshift.io/finalizer
  generation: 7
  labels:
    app: busybox-sample
    cluster.open-cluster-management.io/backup: resource
  name: busybox-drpc
  namespace: busybox-workloads-1
  resourceVersion: "2264589"
  uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
  action: Failover
  drPolicyRef:
    name: dr-policy-10m
  failoverCluster: kmanohar-clu2
  placementRef:
    kind: PlacementRule
    name: busybox-placement
    namespace: busybox-workloads-1
  preferredCluster: kmanohar-clu1
  pvcSelector:
    matchLabels:
      appname: busybox
status:
  actionDuration: 3m45.742949056s
  actionStartTime: "2023-01-04T11:33:13Z"
  conditions:
  - lastTransitionTime: "2023-01-04T11:33:25Z"
    message: Completed
    observedGeneration: 7
    reason: FailedOver
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-04T11:36:58Z"
    message: Cleaned
    observedGeneration: 7
    reason: Success
    status: "True"
    type: PeerReady
  lastGroupSyncTime: "2023-01-04T11:33:31Z"
  lastUpdateTime: "2023-01-04T11:39:51Z"
  phase: FailedOver
  preferredDecision:
    clusterName: kmanohar-clu1
    clusterNamespace: kmanohar-clu1
  progression: Completed
  resourceConditions:
    conditions:
    - lastTransitionTime: "2023-01-04T11:33:38Z"
      message: PVCs in the VolumeReplicationGroup are ready for use
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: DataReady
    - lastTransitionTime: "2023-01-04T11:33:27Z"
      message: VolumeReplicationGroup is replicating
      observedGeneration: 1
      reason: Replicating
      status: "False"
      type: DataProtected
    - lastTransitionTime: "2023-01-04T11:33:16Z"
      message: Restored PV cluster data
      observedGeneration: 1
      reason: Restored
      status: "True"
      type: ClusterDataReady
    - lastTransitionTime: "2023-01-04T11:33:32Z"
      message: Kube objects protected
      observedGeneration: 1
      reason: Uploaded
      status: "True"
      type: ClusterDataProtected
    resourceMeta:
      generation: 1
      kind: VolumeReplicationGroup
      name: busybox-drpc
      namespace: busybox-workloads-1
      protectedpvcs:
      - busybox-pvc-1
      - busybox-pvc-10
      - busybox-pvc-19
      - busybox-pvc-2
      - busybox-pvc-9
      - busybox-pvc-6
      - busybox-pvc-7
      - busybox-pvc-11
      - busybox-pvc-15
      - busybox-pvc-16
      - busybox-pvc-5
      - busybox-pvc-4
      - busybox-pvc-12
      - busybox-pvc-14
      - busybox-pvc-18
      - busybox-pvc-20
      - busybox-pvc-13
      - busybox-pvc-17
      - busybox-pvc-3
      - busybox-pvc-8


failover 2

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   7h45m   kmanohar-clu1      kmanohar-clu1     Failover       FailedOver     Completed     2023-01-04T12:58:56Z   3m58.740763547s   True



apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  creationTimestamp: "2023-01-04T05:22:17Z"
  finalizers:
  - drpc.ramendr.openshift.io/finalizer
  generation: 8
  labels:
    app: busybox-sample
    cluster.open-cluster-management.io/backup: resource
  name: busybox-drpc
  namespace: busybox-workloads-1
  resourceVersion: "2371102"
  uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
  action: Failover
  drPolicyRef:
    name: dr-policy-10m
  failoverCluster: kmanohar-clu1
  placementRef:
    kind: PlacementRule
    name: busybox-placement
    namespace: busybox-workloads-1
  preferredCluster: kmanohar-clu1
  pvcSelector:
    matchLabels:
      appname: busybox
status:
  actionDuration: 3m58.740763547s
  actionStartTime: "2023-01-04T12:58:56Z"
  conditions:
  - lastTransitionTime: "2023-01-04T12:59:00Z"
    message: Completed
    observedGeneration: 8
    reason: FailedOver
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-04T13:02:54Z"
    message: Cleaned
    observedGeneration: 8
    reason: Success
    status: "True"
    type: PeerReady
  lastGroupSyncTime: "2023-01-04T12:59:11Z"
  lastUpdateTime: "2023-01-04T13:04:30Z"
  phase: FailedOver
  preferredDecision:
    clusterName: kmanohar-clu1
    clusterNamespace: kmanohar-clu1
  progression: Completed
  resourceConditions:
    conditions:
    - lastTransitionTime: "2023-01-04T12:59:14Z"
      message: PVCs in the VolumeReplicationGroup are ready for use
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: DataReady
    - lastTransitionTime: "2023-01-04T12:59:01Z"
      message: VolumeReplicationGroup is replicating
      observedGeneration: 1
      reason: Replicating
      status: "False"
      type: DataProtected
    - lastTransitionTime: "2023-01-04T12:58:58Z"
      message: Restored PV cluster data
      observedGeneration: 1
      reason: Restored
      status: "True"
      type: ClusterDataReady
    - lastTransitionTime: "2023-01-04T12:59:07Z"
      message: Kube objects protected
      observedGeneration: 1
      reason: Uploaded
      status: "True"
      type: ClusterDataProtected
    resourceMeta:
      generation: 1
      kind: VolumeReplicationGroup
      name: busybox-drpc
      namespace: busybox-workloads-1
      protectedpvcs:
      - busybox-pvc-1
      - busybox-pvc-10
      - busybox-pvc-6
      - busybox-pvc-13
      - busybox-pvc-14
      - busybox-pvc-17
      - busybox-pvc-20
      - busybox-pvc-5
      - busybox-pvc-15
      - busybox-pvc-18
      - busybox-pvc-19
      - busybox-pvc-2
      - busybox-pvc-3
      - busybox-pvc-11
      - busybox-pvc-16
      - busybox-pvc-8
      - busybox-pvc-9
      - busybox-pvc-12
      - busybox-pvc-4
      - busybox-pvc-7



___________________________________________________________


failover 3

oc get drpc -n busybox-workloads-1 -o wide
NAME           AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-drpc   7h53m   kmanohar-clu1      kmanohar-clu2     Failover       FailedOver     Completed     2023-01-04T13:10:45Z   4m15.789209787s   True



apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
  creationTimestamp: "2023-01-04T05:22:17Z"
  finalizers:
  - drpc.ramendr.openshift.io/finalizer
  generation: 9
  labels:
    app: busybox-sample
    cluster.open-cluster-management.io/backup: resource
  name: busybox-drpc
  namespace: busybox-workloads-1
  resourceVersion: "2386427"
  uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
  action: Failover
  drPolicyRef:
    name: dr-policy-10m
  failoverCluster: kmanohar-clu2
  placementRef:
    kind: PlacementRule
    name: busybox-placement
    namespace: busybox-workloads-1
  preferredCluster: kmanohar-clu1
  pvcSelector:
    matchLabels:
      appname: busybox
status:
  actionDuration: 4m15.789209787s
  actionStartTime: "2023-01-04T13:10:45Z"
  conditions:
  - lastTransitionTime: "2023-01-04T13:10:54Z"
    message: Completed
    observedGeneration: 9
    reason: FailedOver
    status: "True"
    type: Available
  - lastTransitionTime: "2023-01-04T13:15:00Z"
    message: Cleaned
    observedGeneration: 9
    reason: Success
    status: "True"
    type: PeerReady
  lastGroupSyncTime: "2023-01-04T13:10:58Z"
  lastUpdateTime: "2023-01-04T13:16:24Z"
  phase: FailedOver
  preferredDecision:
    clusterName: kmanohar-clu1
    clusterNamespace: kmanohar-clu1
  progression: Completed
  resourceConditions:
    conditions:
    - lastTransitionTime: "2023-01-04T13:11:10Z"
      message: PVCs in the VolumeReplicationGroup are ready for use
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: DataReady
    - lastTransitionTime: "2023-01-04T13:10:55Z"
      message: VolumeReplicationGroup is replicating
      observedGeneration: 1
      reason: Replicating
      status: "False"
      type: DataProtected
    - lastTransitionTime: "2023-01-04T13:10:47Z"
      message: Restored PV cluster data
      observedGeneration: 1
      reason: Restored
      status: "True"
      type: ClusterDataReady
    - lastTransitionTime: "2023-01-04T13:11:01Z"
      message: Kube objects protected
      observedGeneration: 1
      reason: Uploaded
      status: "True"
      type: ClusterDataProtected
    resourceMeta:
      generation: 1
      kind: VolumeReplicationGroup
      name: busybox-drpc
      namespace: busybox-workloads-1
      protectedpvcs:
      - busybox-pvc-1
      - busybox-pvc-4
      - busybox-pvc-6
      - busybox-pvc-7
      - busybox-pvc-8
      - busybox-pvc-12
      - busybox-pvc-2
      - busybox-pvc-17
      - busybox-pvc-18
      - busybox-pvc-13
      - busybox-pvc-3
      - busybox-pvc-5
      - busybox-pvc-15
      - busybox-pvc-16
      - busybox-pvc-19
      - busybox-pvc-20
      - busybox-pvc-9
      - busybox-pvc-10
      - busybox-pvc-11
      - busybox-pvc-14

Note You need to log in before you can comment on or make changes to this bug.