Bug 2122700
| Summary: | [RDR] DRPC status shows stale information on subsequent same DR actions | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Sidhant Agrawal <sagrawal> |
| Component: | odf-dr | Assignee: | Benamar Mekhissi <bmekhiss> |
| odf-dr sub component: | ramen | QA Contact: | kmanohar |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | bmekhiss, ebenahar, muagarwa, ocs-bugs, odf-bz-bot, srangana |
| Version: | 4.11 | ||
| Target Milestone: | --- | ||
| Target Release: | ODF 4.12.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 4.12.0-114 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-02-08 14:06:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 3
Benamar Mekhissi
2022-11-13 14:48:45 UTC
Steps to Reproduce:
1. On a RDR setup, deploy a simple busybox application
2. Perform a Failover action
3. Again perform failover action and observe the drpc status by using the command "oc get drpc -o wide"
Actual results:
DRPC status displays stale information on subsequent and same DR action.
Expected results:
DRPC status shows updated information after each failover/relocate action.
______________________________________________________________________________________________________________________
Verification comments:
Verified on - 4.12.0-152
ACM - 2.7
CEPH version:- ceph version 16.2.10-82.el8cp
before relocate
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 4h17m kmanohar-clu1 Deployed Completed 2023-01-04T05:22:17Z 3h0m35.785389123s True
relocate 1:
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 4h24m kmanohar-clu2 Relocate Relocated Completed 2023-01-04T09:41:07Z 2m49.580970394s True
relocate 2:
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 4h59m kmanohar-clu1 Relocate Relocated Completed 2023-01-04T10:15:01Z 3m23.740561449s True
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
creationTimestamp: "2023-01-04T05:22:17Z"
finalizers:
- drpc.ramendr.openshift.io/finalizer
generation: 4
labels:
app: busybox-sample
cluster.open-cluster-management.io/backup: resource
name: busybox-drpc
namespace: busybox-workloads-1
resourceVersion: "2205293"
uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
action: Relocate
drPolicyRef:
name: dr-policy-10m
placementRef:
kind: PlacementRule
name: busybox-placement
namespace: busybox-workloads-1
preferredCluster: kmanohar-clu1
pvcSelector:
matchLabels:
appname: busybox
status:
actionDuration: 3m23.740561449s
actionStartTime: "2023-01-04T10:15:01Z"
conditions:
- lastTransitionTime: "2023-01-04T10:18:24Z"
message: Completed
observedGeneration: 4
reason: Relocated
status: "True"
type: Available
- lastTransitionTime: "2023-01-04T10:18:24Z"
message: Cleaned
observedGeneration: 4
reason: Success
status: "True"
type: PeerReady
lastGroupSyncTime: "2023-01-04T10:50:00Z"
lastUpdateTime: "2023-01-04T10:52:57Z"
phase: Relocated
preferredDecision:
clusterName: kmanohar-clu1
clusterNamespace: kmanohar-clu1
progression: Completed
resourceConditions:
conditions:
- lastTransitionTime: "2023-01-04T10:17:36Z"
message: PVCs in the VolumeReplicationGroup are ready for use
observedGeneration: 1
reason: Ready
status: "True"
type: DataReady
- lastTransitionTime: "2023-01-04T10:17:27Z"
message: VolumeReplicationGroup is replicating
observedGeneration: 1
reason: Replicating
status: "False"
type: DataProtected
- lastTransitionTime: "2023-01-04T10:16:56Z"
message: Restored PV cluster data
observedGeneration: 1
reason: Restored
status: "True"
type: ClusterDataReady
- lastTransitionTime: "2023-01-04T10:17:33Z"
message: Kube objects protected
observedGeneration: 1
reason: Uploaded
status: "True"
type: ClusterDataProtected
resourceMeta:
generation: 1
kind: VolumeReplicationGroup
name: busybox-drpc
namespace: busybox-workloads-1
protectedpvcs:
- busybox-pvc-1
- busybox-pvc-20
- busybox-pvc-10
- busybox-pvc-12
- busybox-pvc-15
- busybox-pvc-16
- busybox-pvc-18
- busybox-pvc-11
- busybox-pvc-13
- busybox-pvc-14
- busybox-pvc-17
- busybox-pvc-19
- busybox-pvc-2
- busybox-pvc-8
- busybox-pvc-9
- busybox-pvc-3
- busybox-pvc-4
- busybox-pvc-5
- busybox-pvc-6
- busybox-pvc-7
relocate 3:
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 5h48m kmanohar-clu2 Relocate Relocated Completed 2023-01-04T11:03:30Z 2m58.199064636s True
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
creationTimestamp: "2023-01-04T05:22:17Z"
finalizers:
- drpc.ramendr.openshift.io/finalizer
generation: 5
labels:
app: busybox-sample
cluster.open-cluster-management.io/backup: resource
name: busybox-drpc
namespace: busybox-workloads-1
resourceVersion: "2222538"
uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
action: Relocate
drPolicyRef:
name: dr-policy-10m
placementRef:
kind: PlacementRule
name: busybox-placement
namespace: busybox-workloads-1
preferredCluster: kmanohar-clu2
pvcSelector:
matchLabels:
appname: busybox
status:
actionDuration: 2m58.199064636s
actionStartTime: "2023-01-04T11:03:30Z"
conditions:
- lastTransitionTime: "2023-01-04T11:06:28Z"
message: Completed
observedGeneration: 5
reason: Relocated
status: "True"
type: Available
- lastTransitionTime: "2023-01-04T11:06:28Z"
message: Cleaned
observedGeneration: 5
reason: Success
status: "True"
type: PeerReady
lastUpdateTime: "2023-01-04T11:06:28Z"
phase: Relocated
preferredDecision:
clusterName: kmanohar-clu1
clusterNamespace: kmanohar-clu1
progression: Completed
resourceConditions:
conditions:
- lastTransitionTime: "2023-01-04T11:06:03Z"
message: PVCs in the VolumeReplicationGroup are ready for use
observedGeneration: 1
reason: Ready
status: "True"
type: DataReady
- lastTransitionTime: "2023-01-04T11:05:56Z"
message: VolumeReplicationGroup is replicating
observedGeneration: 1
reason: Replicating
status: "False"
type: DataProtected
- lastTransitionTime: "2023-01-04T11:05:30Z"
message: Restored PV cluster data
observedGeneration: 1
reason: Restored
status: "True"
type: ClusterDataReady
- lastTransitionTime: "2023-01-04T11:06:02Z"
message: Kube objects protected
observedGeneration: 1
reason: Uploaded
status: "True"
type: ClusterDataProtected
resourceMeta:
generation: 1
kind: VolumeReplicationGroup
name: busybox-drpc
namespace: busybox-workloads-1
protectedpvcs:
- busybox-pvc-1
- busybox-pvc-10
- busybox-pvc-12
- busybox-pvc-14
- busybox-pvc-2
- busybox-pvc-3
- busybox-pvc-5
- busybox-pvc-8
- busybox-pvc-16
- busybox-pvc-17
- busybox-pvc-18
- busybox-pvc-19
- busybox-pvc-20
- busybox-pvc-4
- busybox-pvc-9
- busybox-pvc-11
- busybox-pvc-15
- busybox-pvc-6
- busybox-pvc-7
- busybox-pvc-13
____________________________________________________________________
failover 1
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 6h16m kmanohar-clu1 kmanohar-clu2 Failover FailedOver Completed 2023-01-04T11:33:13Z 3m45.742949056s True
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
creationTimestamp: "2023-01-04T05:22:17Z"
finalizers:
- drpc.ramendr.openshift.io/finalizer
generation: 7
labels:
app: busybox-sample
cluster.open-cluster-management.io/backup: resource
name: busybox-drpc
namespace: busybox-workloads-1
resourceVersion: "2264589"
uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
action: Failover
drPolicyRef:
name: dr-policy-10m
failoverCluster: kmanohar-clu2
placementRef:
kind: PlacementRule
name: busybox-placement
namespace: busybox-workloads-1
preferredCluster: kmanohar-clu1
pvcSelector:
matchLabels:
appname: busybox
status:
actionDuration: 3m45.742949056s
actionStartTime: "2023-01-04T11:33:13Z"
conditions:
- lastTransitionTime: "2023-01-04T11:33:25Z"
message: Completed
observedGeneration: 7
reason: FailedOver
status: "True"
type: Available
- lastTransitionTime: "2023-01-04T11:36:58Z"
message: Cleaned
observedGeneration: 7
reason: Success
status: "True"
type: PeerReady
lastGroupSyncTime: "2023-01-04T11:33:31Z"
lastUpdateTime: "2023-01-04T11:39:51Z"
phase: FailedOver
preferredDecision:
clusterName: kmanohar-clu1
clusterNamespace: kmanohar-clu1
progression: Completed
resourceConditions:
conditions:
- lastTransitionTime: "2023-01-04T11:33:38Z"
message: PVCs in the VolumeReplicationGroup are ready for use
observedGeneration: 1
reason: Ready
status: "True"
type: DataReady
- lastTransitionTime: "2023-01-04T11:33:27Z"
message: VolumeReplicationGroup is replicating
observedGeneration: 1
reason: Replicating
status: "False"
type: DataProtected
- lastTransitionTime: "2023-01-04T11:33:16Z"
message: Restored PV cluster data
observedGeneration: 1
reason: Restored
status: "True"
type: ClusterDataReady
- lastTransitionTime: "2023-01-04T11:33:32Z"
message: Kube objects protected
observedGeneration: 1
reason: Uploaded
status: "True"
type: ClusterDataProtected
resourceMeta:
generation: 1
kind: VolumeReplicationGroup
name: busybox-drpc
namespace: busybox-workloads-1
protectedpvcs:
- busybox-pvc-1
- busybox-pvc-10
- busybox-pvc-19
- busybox-pvc-2
- busybox-pvc-9
- busybox-pvc-6
- busybox-pvc-7
- busybox-pvc-11
- busybox-pvc-15
- busybox-pvc-16
- busybox-pvc-5
- busybox-pvc-4
- busybox-pvc-12
- busybox-pvc-14
- busybox-pvc-18
- busybox-pvc-20
- busybox-pvc-13
- busybox-pvc-17
- busybox-pvc-3
- busybox-pvc-8
failover 2
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 7h45m kmanohar-clu1 kmanohar-clu1 Failover FailedOver Completed 2023-01-04T12:58:56Z 3m58.740763547s True
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
creationTimestamp: "2023-01-04T05:22:17Z"
finalizers:
- drpc.ramendr.openshift.io/finalizer
generation: 8
labels:
app: busybox-sample
cluster.open-cluster-management.io/backup: resource
name: busybox-drpc
namespace: busybox-workloads-1
resourceVersion: "2371102"
uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
action: Failover
drPolicyRef:
name: dr-policy-10m
failoverCluster: kmanohar-clu1
placementRef:
kind: PlacementRule
name: busybox-placement
namespace: busybox-workloads-1
preferredCluster: kmanohar-clu1
pvcSelector:
matchLabels:
appname: busybox
status:
actionDuration: 3m58.740763547s
actionStartTime: "2023-01-04T12:58:56Z"
conditions:
- lastTransitionTime: "2023-01-04T12:59:00Z"
message: Completed
observedGeneration: 8
reason: FailedOver
status: "True"
type: Available
- lastTransitionTime: "2023-01-04T13:02:54Z"
message: Cleaned
observedGeneration: 8
reason: Success
status: "True"
type: PeerReady
lastGroupSyncTime: "2023-01-04T12:59:11Z"
lastUpdateTime: "2023-01-04T13:04:30Z"
phase: FailedOver
preferredDecision:
clusterName: kmanohar-clu1
clusterNamespace: kmanohar-clu1
progression: Completed
resourceConditions:
conditions:
- lastTransitionTime: "2023-01-04T12:59:14Z"
message: PVCs in the VolumeReplicationGroup are ready for use
observedGeneration: 1
reason: Ready
status: "True"
type: DataReady
- lastTransitionTime: "2023-01-04T12:59:01Z"
message: VolumeReplicationGroup is replicating
observedGeneration: 1
reason: Replicating
status: "False"
type: DataProtected
- lastTransitionTime: "2023-01-04T12:58:58Z"
message: Restored PV cluster data
observedGeneration: 1
reason: Restored
status: "True"
type: ClusterDataReady
- lastTransitionTime: "2023-01-04T12:59:07Z"
message: Kube objects protected
observedGeneration: 1
reason: Uploaded
status: "True"
type: ClusterDataProtected
resourceMeta:
generation: 1
kind: VolumeReplicationGroup
name: busybox-drpc
namespace: busybox-workloads-1
protectedpvcs:
- busybox-pvc-1
- busybox-pvc-10
- busybox-pvc-6
- busybox-pvc-13
- busybox-pvc-14
- busybox-pvc-17
- busybox-pvc-20
- busybox-pvc-5
- busybox-pvc-15
- busybox-pvc-18
- busybox-pvc-19
- busybox-pvc-2
- busybox-pvc-3
- busybox-pvc-11
- busybox-pvc-16
- busybox-pvc-8
- busybox-pvc-9
- busybox-pvc-12
- busybox-pvc-4
- busybox-pvc-7
___________________________________________________________
failover 3
oc get drpc -n busybox-workloads-1 -o wide
NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY
busybox-drpc 7h53m kmanohar-clu1 kmanohar-clu2 Failover FailedOver Completed 2023-01-04T13:10:45Z 4m15.789209787s True
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata:
creationTimestamp: "2023-01-04T05:22:17Z"
finalizers:
- drpc.ramendr.openshift.io/finalizer
generation: 9
labels:
app: busybox-sample
cluster.open-cluster-management.io/backup: resource
name: busybox-drpc
namespace: busybox-workloads-1
resourceVersion: "2386427"
uid: 905776ca-2624-433a-adbf-67ee785ab2e0
spec:
action: Failover
drPolicyRef:
name: dr-policy-10m
failoverCluster: kmanohar-clu2
placementRef:
kind: PlacementRule
name: busybox-placement
namespace: busybox-workloads-1
preferredCluster: kmanohar-clu1
pvcSelector:
matchLabels:
appname: busybox
status:
actionDuration: 4m15.789209787s
actionStartTime: "2023-01-04T13:10:45Z"
conditions:
- lastTransitionTime: "2023-01-04T13:10:54Z"
message: Completed
observedGeneration: 9
reason: FailedOver
status: "True"
type: Available
- lastTransitionTime: "2023-01-04T13:15:00Z"
message: Cleaned
observedGeneration: 9
reason: Success
status: "True"
type: PeerReady
lastGroupSyncTime: "2023-01-04T13:10:58Z"
lastUpdateTime: "2023-01-04T13:16:24Z"
phase: FailedOver
preferredDecision:
clusterName: kmanohar-clu1
clusterNamespace: kmanohar-clu1
progression: Completed
resourceConditions:
conditions:
- lastTransitionTime: "2023-01-04T13:11:10Z"
message: PVCs in the VolumeReplicationGroup are ready for use
observedGeneration: 1
reason: Ready
status: "True"
type: DataReady
- lastTransitionTime: "2023-01-04T13:10:55Z"
message: VolumeReplicationGroup is replicating
observedGeneration: 1
reason: Replicating
status: "False"
type: DataProtected
- lastTransitionTime: "2023-01-04T13:10:47Z"
message: Restored PV cluster data
observedGeneration: 1
reason: Restored
status: "True"
type: ClusterDataReady
- lastTransitionTime: "2023-01-04T13:11:01Z"
message: Kube objects protected
observedGeneration: 1
reason: Uploaded
status: "True"
type: ClusterDataProtected
resourceMeta:
generation: 1
kind: VolumeReplicationGroup
name: busybox-drpc
namespace: busybox-workloads-1
protectedpvcs:
- busybox-pvc-1
- busybox-pvc-4
- busybox-pvc-6
- busybox-pvc-7
- busybox-pvc-8
- busybox-pvc-12
- busybox-pvc-2
- busybox-pvc-17
- busybox-pvc-18
- busybox-pvc-13
- busybox-pvc-3
- busybox-pvc-5
- busybox-pvc-15
- busybox-pvc-16
- busybox-pvc-19
- busybox-pvc-20
- busybox-pvc-9
- busybox-pvc-10
- busybox-pvc-11
- busybox-pvc-14
|