Bug 2255998 - [RDR] [Hub recovery] Failover for most cephfs and a few rbd workloads remain stuck at WaitingForResourceRestore
Summary: [RDR] [Hub recovery] Failover for most cephfs and a few rbd workloads remain ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: odf-dr
Version: 4.14
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ODF 4.16.0
Assignee: Benamar Mekhissi
QA Contact: Aman Agrawal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-12-27 17:13 UTC by Aman Agrawal
Modified: 2024-07-17 13:11 UTC (History)
5 users (show)

Fixed In Version: 4.15.0-146
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-07-17 13:11:49 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github RamenDR ramen pull 1203 0 None open Fix pvc restore failure for VolSync workload 2024-02-13 17:24:11 UTC
Github red-hat-storage ramen pull 195 0 None open Bug 2255998: Fix pvc restore failure for VolSync workload 2024-02-18 14:29:16 UTC
Red Hat Product Errata RHSA-2024:4591 0 None None None 2024-07-17 13:11:50 UTC

Comment 17 Benamar Mekhissi 2024-02-13 17:23:49 UTC
PR: https://github.com/RamenDR/ramen/pull/1203

Comment 27 Aman Agrawal 2024-05-31 19:30:07 UTC
Tested with following versions:

ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)
OCP 4.16.0-0.nightly-2024-05-23-173505
ACM 2.11.0-DOWNSTREAM-2024-05-23-15-16-26
MCE 2.6.0-104 
ODF 4.16.0-108.stable
Gitops v1.12.3 
Submariner 0.18.0 (image: brew.registry.redhat.io/rh-osbs/iib:722673)

Platform- VMware

When the steps to reproduce is repeated, multiple RBD and CephFS workloads with distinct states like Deployed, FailedOver and Relocated of both appset (pull) and subscription types were failedover in the final step from cluster amagrawa-c2-my28 to amagrawa-c1-28my and the failover was successful for all of them.

Workload also included 2 CNV(RBD) workloads of both appset(pull) and subscription types (cnv-sub-busybox18-placement-1-drpc and cnv-appset-busybox17-placement-drpc).



date -u; oc get drpc -o wide -A; echo "/////////////////////////////////////////////////////////////////////"
Fri May 31 19:11:38 UTC 2024
NAMESPACE               NAME                                      AGE   PREFERREDCLUSTER   FAILOVERCLUSTER    DESIREDSTATE   CURRENTSTATE   PROGRESSION                           START TIME             DURATION   PEER READY
busybox-workloads-101   rbd-sub-busybox101-placement-1-drpc       31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:03Z              False
busybox-workloads-103   cephfs-sub-busybox103-placement-1-drpc    31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:41Z              False
busybox-workloads-13    cephfs-sub-busybox13-placement-1-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:48Z              False
busybox-workloads-14    cephfs-sub-busybox14-placement-1-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:29Z              False
busybox-workloads-15    cephfs-sub-busybox15-placement-1-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:16Z              False
busybox-workloads-16    cephfs-sub-busybox16-placement-1-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:09Z              False
busybox-workloads-18    cnv-sub-busybox18-placement-1-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:06:24Z              False
busybox-workloads-20    rbd-sub-busybox20-placement-1-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:10Z              False
busybox-workloads-22    cephfs-sub-busybox22-placement-1-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:23Z              False
busybox-workloads-5     rbd-sub-busybox5-placement-1-drpc         31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:50Z              False
busybox-workloads-6     rbd-sub-busybox6-placement-1-drpc         31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:22Z              False
busybox-workloads-7     rbd-sub-busybox7-placement-1-drpc         31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:16Z              False
busybox-workloads-8     rbd-sub-busybox8-placement-1-drpc         31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:57Z              False
openshift-gitops        cephfs-appset-busybox10-placement-drpc    31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:53Z              False
openshift-gitops        cephfs-appset-busybox102-placement-drpc   31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:02:58Z              False
openshift-gitops        cephfs-appset-busybox11-placement-drpc    31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:03Z              False
openshift-gitops        cephfs-appset-busybox12-placement-drpc    31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:07Z              False
openshift-gitops        cephfs-appset-busybox21-placement-drpc    31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:14Z              False
openshift-gitops        cephfs-appset-busybox9-placement-drpc     31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:03:18Z              False
openshift-gitops        cnv-appset-busybox17-placement-drpc       31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:06:30Z              False
openshift-gitops        rbd-appset-busybox1-placement-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:28Z              False
openshift-gitops        rbd-appset-busybox100-placement-drpc      31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:34Z              False
openshift-gitops        rbd-appset-busybox19-placement-drpc       31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:39Z              False
openshift-gitops        rbd-appset-busybox2-placement-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:46Z              False
openshift-gitops        rbd-appset-busybox3-placement-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:50Z              False
openshift-gitops        rbd-appset-busybox4-placement-drpc        31m   amagrawa-c2-my28   amagrawa-c1-28my   Failover       FailedOver     Cleaning Up                           2024-05-31T19:04:54Z              False
/////////////////////////////////////////////////////////////////////

Since the cluster amagrawa-c2-my28 is still down, drpc progression is reporting Cleaning Up which is expected.

The issue reported in this BZ was not hit, therefore I am marking this bug as verified. Fix LGTM

Comment 30 errata-xmlrpc 2024-07-17 13:11:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591


Note You need to log in before you can comment on or make changes to this bug.