Bug 2251205

Summary: [4.14 clone] [RDR] [Hub recovery] Sync for all cephfs workloads stopped post hub recovery
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Karolin Seeger <kseeger>
Component: odf-drAssignee: Benamar Mekhissi <bmekhiss>
odf-dr sub component: ramen QA Contact: Aman Agrawal <amagrawa>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: amagrawa, bmekhiss, kramdoss, kseeger, muagarwa
Version: 4.14   
Target Milestone: ---   
Target Release: ODF 4.14.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.14.1-12 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 2250152 Environment:
Last Closed: 2023-12-07 13:21:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2250152    
Bug Blocks:    

Description Karolin Seeger 2023-11-23 13:10:40 UTC 
+++ This bug was initially created as a clone of Bug #2250152 +++

Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):
OCP 4.14.0-0.nightly-2023-11-09-204811
Volsync 0.8.0
Submariner 0.16.2
ACM quay.io:443/acm-d/acm-custom-registry:v2.9.0-RC2        
odf-multicluster-orchestrator.v4.14.1-rhodf  
ceph version 17.2.6-148.el9cp (badc1d27cb07762bea48f6554ad4f92b9d3fbb6b) quincy (stable)
Latency 50ms RTT


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:

**Active hub at neutral site**

1. Deployed multiple rbd and cephfs backed workloads of both appset and subscription types.
2. Failedover and relocated them in such a way that they are finally running on the primary managed cluster (which is expected to host all the workloads and can go under disaster). (A few of them are exception, check drpc -o wide status in Step 3).
3. Ensure that we have the workloads in distict states like deployed, failedover, relocated etc.

Here amagrawa-10n-1 is C1 primary managed cluster for me:

From active hub-

amagrawa:hub$ drpc
NAMESPACE              NAME                                                AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION          PEER READY
busybox-workloads-12   cephfs-sub-busybox-workloads-12-placement-1-drpc    7h18m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed     2023-11-16T08:54:29Z   5m59.196575462s   True
busybox-workloads-13   cephfs-sub-busybox-workloads-13-placement-1-drpc    7h17m   amagrawa-10n-1                       Relocate       Relocated      Completed     2023-11-16T12:12:36Z   5m58.842880173s   True
busybox-workloads-14   cephfs-sub-busybox-workloads-14-placement-1-drpc    7h16m   amagrawa-10n-1     amagrawa-10n-2    Failover       FailedOver     Completed     2023-11-16T08:29:07Z   3m19.098202668s   True
busybox-workloads-6    rbd-sub-busybox-workloads-6-placement-1-drpc        7h35m   amagrawa-10n-1                                      Deployed       Completed                                              True
busybox-workloads-7    rbd-sub-busybox-workloads-7-placement-1-drpc        7h34m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed     2023-11-16T08:53:38Z   9m59.85663627s    True
busybox-workloads-8    rbd-sub-busybox-workloads-8-placement-1-drpc        7h32m   amagrawa-10n-1                       Relocate       Relocated      Completed     2023-11-16T08:21:05Z   4m13.272955733s   True
openshift-gitops       cephfs-appset-busybox-workloads-10-placement-drpc   7h22m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed     2023-11-16T08:15:50Z   3m22.540081438s   True
openshift-gitops       cephfs-appset-busybox-workloads-11-placement-drpc   7h20m   amagrawa-10n-1                       Relocate       Relocated      Completed     2023-11-16T08:00:32Z   5m38.794985745s   True
openshift-gitops       cephfs-appset-busybox-workloads-9-placement-drpc    7h24m   amagrawa-10n-2                       Relocate       Relocated      Completed     2023-11-16T08:28:59Z   8m47.541429779s   True
openshift-gitops       rbd-appset-busybox-workloads-1-placement-drpc       7h43m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed     2023-11-16T08:16:14Z   8m31.330049487s   True
openshift-gitops       rbd-appset-busybox-workloads-2-placement-drpc       7h42m   amagrawa-10n-1                       Relocate       Relocated      Completed     2023-11-16T08:16:28Z   7m59.477897296s   True
openshift-gitops       rbd-appset-busybox-workloads-3-placement-drpc       7h41m   amagrawa-10n-2     amagrawa-10n-1    Failover       FailedOver     Completed     2023-11-16T08:27:18Z   7m4.760183798s    True
openshift-gitops       rbd-appset-busybox-workloads-4-placement-drpc       7h39m   amagrawa-10n-1                                      Deployed       Completed                                              True

4. Let the latest backups be taken at least 1 or 2 (at each 1 hr) for all the different states of the workloads (when progression is completed and no action is going on any of the workloads). Also ensure sync for all the workloads when on active hub is working fine and cluster is healthy. Note drpc -o wide, lastGroupSyncTime, download backups from S3, etc.

amagrawa:hub$ group|grep SyncTime
    lastGroupSyncTime: "2023-11-16T14:01:32Z"
    lastGroupSyncTime: "2023-11-16T14:06:09Z"
    lastGroupSyncTime: "2023-11-16T14:01:03Z"
    lastGroupSyncTime: "2023-11-16T13:45:09Z"
    lastGroupSyncTime: "2023-11-16T13:50:51Z"
    lastGroupSyncTime: "2023-11-16T13:50:40Z"
    lastGroupSyncTime: "2023-11-16T14:00:51Z"
    lastGroupSyncTime: "2023-11-16T14:06:12Z"
    lastGroupSyncTime: "2023-11-16T13:01:45Z"
    lastGroupSyncTime: "2023-11-16T13:50:36Z"
    lastGroupSyncTime: "2023-11-16T13:45:16Z"
    lastGroupSyncTime: "2023-11-16T13:56:22Z"
    lastGroupSyncTime: "2023-11-16T13:45:11Z"


amagrawa:hub$ date -u
Thursday 16 November 2023 02:12:11 PM UTC

5. Bring active hub completely down, move to passive hub. Restore backps, ensure velero backup reports successful restoration. Make sure both the managed clusters are successfully reported, drpolicy gets validated.
6. Wait for drpc to be restored, check if all the workloads are in their last backedup state or not.

They seem to have retained their last state which was backedup. So everything is fine so far.

amagrawa:~$ drpc
NAMESPACE              NAME                                                AGE     PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME   DURATION   PEER READY
busybox-workloads-12   cephfs-sub-busybox-workloads-12-placement-1-drpc    4h16m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed                             True
busybox-workloads-13   cephfs-sub-busybox-workloads-13-placement-1-drpc    4h16m   amagrawa-10n-1                       Relocate       Relocated      Completed                             True
busybox-workloads-14   cephfs-sub-busybox-workloads-14-placement-1-drpc    4h16m   amagrawa-10n-1     amagrawa-10n-2    Failover       FailedOver     Completed                             True
busybox-workloads-6    rbd-sub-busybox-workloads-6-placement-1-drpc        4h16m   amagrawa-10n-1                                      Deployed       Completed                             True
busybox-workloads-7    rbd-sub-busybox-workloads-7-placement-1-drpc        4h16m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed                             True
busybox-workloads-8    rbd-sub-busybox-workloads-8-placement-1-drpc        4h16m   amagrawa-10n-1                       Relocate       Relocated      Completed                             True
openshift-gitops       cephfs-appset-busybox-workloads-10-placement-drpc   4h16m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed                             True
openshift-gitops       cephfs-appset-busybox-workloads-11-placement-drpc   4h16m   amagrawa-10n-1                       Relocate       Relocated      Completed                             True
openshift-gitops       cephfs-appset-busybox-workloads-9-placement-drpc    4h16m   amagrawa-10n-2                       Relocate       Relocated      Completed                             True
openshift-gitops       rbd-appset-busybox-workloads-1-placement-drpc       4h16m   amagrawa-10n-1     amagrawa-10n-1    Failover       FailedOver     Completed                             True
openshift-gitops       rbd-appset-busybox-workloads-2-placement-drpc       4h16m   amagrawa-10n-1                       Relocate       Relocated      Completed                             True
openshift-gitops       rbd-appset-busybox-workloads-3-placement-drpc       4h16m   amagrawa-10n-2     amagrawa-10n-1    Failover       FailedOver     Completed                             True
openshift-gitops       rbd-appset-busybox-workloads-4-placement-drpc       4h16m   amagrawa-10n-1                                      Deployed       Completed                             True

7. Let IOs continue for a few hours. We observed that data sync for rbd based workloads were progressing just fine but sync stopped for all the cephfs based workloads be it of subsciption or appset type.


Actual results: Sync for all cephfs workloads stopped post hub recovery.

Logs- http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-aman/16nov23/logs/

VolumeSyncronizationDelay alert fires on passive hub for all cephfs workloads when monitoring label is applied.

Expected results: Sync for all cephfs workloads should continue without any issues post hub recovery.


Additional info:

--- Additional comment from RHEL Program Management on 2023-11-16 19:05:30 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.15.0' to '?', and so is being proposed to be fixed at the ODF 4.15.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from RHEL Program Management on 2023-11-16 19:05:30 UTC ---

Since this bug has severity set to 'urgent', it is being proposed as a blocker for the currently set release flag. Please resolve ASAP.

--- Additional comment from Benamar Mekhissi on 2023-11-17 02:42:37 UTC ---

Submariner lighthouse agent log is showing port conflicts every a couple of seconds.  The same message can be seen in the ReplicationDestination ServiceExport:

```
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  creationTimestamp: "2023-11-16T08:20:32Z"
  generation: 1
  name: volsync-rsync-tls-dst-busybox-pvc-4
  namespace: busybox-workloads-10
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    kind: ReplicationDestination
    name: busybox-pvc-4
    uid: 6c48fffa-4375-4845-9272-6801f94b5ac3
  resourceVersion: "4909434"
  uid: 72bfc6a0-1fed-434b-bfb5-7ed9674eba9e
status:
  conditions:
  - lastTransitionTime: "2023-11-16T08:20:42Z"
    message: ""
    reason: ""
    status: "True"
    type: Valid
  - lastTransitionTime: "2023-11-16T08:20:52Z"
    message: Service was successfully exported to the broker
    reason: ""
    status: "True"
    type: Ready
  - lastTransitionTime: "2023-11-16T14:35:02Z"
    message: 'The service ports conflict between the constituent clusters [amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2,
      amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1,
      amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-1, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-2, amagrawa-10n-2,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1, amagrawa-10n-1,
      amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-2, amagrawa-10n-1,
      amagrawa-10n-1]. The service will expose the intersection of all the ports: '
    reason: ConflictingPorts
    status: "True"
    type: Conflict
```

Submariner team is engaged. We've been asked to run `subctl verify`.  @amagrawa can you please run the "subctl verify" test and capture the output.

I ran the verification manually https://submariner.io/getting-started/quickstart/kind/#verify-manually and that seems to timeout.

--- Additional comment from Aman Agrawal on 2023-11-17 08:00:30 UTC ---

subctl verify connectivity check with context to both the managed clusters passed

--- Additional comment from Benamar Mekhissi on 2023-11-17 19:58:21 UTC ---

There are two issues contributing to this problem. The first, mentioned earlier, involves a Submariner issue that doesn't impact functionality but can be misleading and challenging to diagnose. The Submariner team is addressing it in subsequent releases.

The second issue relates to ACM/ODF, specifically the automatic backup of `policies.policy.open-cluster-management.io`. These ACM policies, are responsible for delivering the volsync mover secret from the hub to managed clusters. The volsync mover secrets aren't backed up. This results in a new active hub having a policy that doesn't point to the secret, causing the mover pod to fail in establishing a TLS connection.

We will find a suitable solution for version 4.15. Meanwhile, the workaround involves deleting these policies from the hub, prompting their regeneration with the correct secrets. Additionally, the mover pods on the destination cluster have be deleted as well.

For example; on the new hub, I have the following policies:
```
oc get policy -A | grep vs-secret | grep amagrawa-10n
amagrawa-10n-1                   busybox-workloads-12.vs-secret-db29abb9d806bf0ea8690fda8bc7d9cb                        Compliant          28h
amagrawa-10n-1                   busybox-workloads-13.vs-secret-8f116547aba3d58497d1c6440090c6c4                        Compliant          14m
amagrawa-10n-1                   busybox-workloads-14.vs-secret-3afb3c0dc0b9646dceef4fe1bd3da4a6                        Compliant          28h
amagrawa-10n-1                   openshift-gitops.vs-secret-6492ecc11b403505c9f30eb84106ac1b                            Compliant          28h
amagrawa-10n-1                   openshift-gitops.vs-secret-9dbea625099ac8d1b144c293ee183452                            Compliant          28h
amagrawa-10n-1                   openshift-gitops.vs-secret-eadf8882df7cd2a1a52db6e4631e5e98                            Compliant          28h
amagrawa-10n-2                   busybox-workloads-12.vs-secret-db29abb9d806bf0ea8690fda8bc7d9cb                        NonCompliant       28m
amagrawa-10n-2                   busybox-workloads-13.vs-secret-8f116547aba3d58497d1c6440090c6c4                        Compliant          37m
amagrawa-10n-2                   busybox-workloads-14.vs-secret-3afb3c0dc0b9646dceef4fe1bd3da4a6                        NonCompliant       28h
amagrawa-10n-2                   openshift-gitops.vs-secret-6492ecc11b403505c9f30eb84106ac1b                            NonCompliant       28h
amagrawa-10n-2                   openshift-gitops.vs-secret-9dbea625099ac8d1b144c293ee183452                            NonCompliant       28h
amagrawa-10n-2                   openshift-gitops.vs-secret-eadf8882df7cd2a1a52db6e4631e5e98                            NonCompliant       28h
```

Any policy that shows "NonCompliant" for the Compliance state should be deleted. New ones will be regenerated with the right secrets.

Also note the cluster name in the first column, and delete all the jobs that are pending on that cluster for the workload namespace.

Example:
```
oc -n busybox-workloads-12 delete jobs --all
job.batch "volsync-rsync-tls-dst-busybox-pvc-1" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-10" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-11" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-12" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-13" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-14" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-15" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-16" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-17" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-18" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-19" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-2" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-20" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-3" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-4" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-5" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-6" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-7" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-8" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-9" deleted
```

--- Additional comment from Aman Agrawal on 2023-11-18 07:13:39 UTC ---

Benamar had already applied the workaround for workloads-12, 13, and 14 and had also deleted all the NonCompliant secrets.

When I checked (on passive hub)-
amagrawa:~$ oc get policy -A | grep vs-secret | grep amagrawa-10n
amagrawa-10n-1                   benamar-ns.busybox-drpc-vs-secret                                                      Compliant          10h
amagrawa-10n-1                   busybox-workloads-12.vs-secret-db29abb9d806bf0ea8690fda8bc7d9cb                        Compliant          10h
amagrawa-10n-1                   busybox-workloads-13.vs-secret-8f116547aba3d58497d1c6440090c6c4                        Compliant          10h
amagrawa-10n-1                   busybox-workloads-14.vs-secret-3afb3c0dc0b9646dceef4fe1bd3da4a6                        Compliant          10h
amagrawa-10n-1                   openshift-gitops.vs-secret-6492ecc11b403505c9f30eb84106ac1b                            Compliant          10h
amagrawa-10n-1                   openshift-gitops.vs-secret-9dbea625099ac8d1b144c293ee183452                            Compliant          10h
amagrawa-10n-1                   openshift-gitops.vs-secret-eadf8882df7cd2a1a52db6e4631e5e98                            Compliant          10h
amagrawa-10n-2                   benamar-ns.busybox-drpc-vs-secret                                                      Compliant          10h
amagrawa-10n-2                   busybox-workloads-12.vs-secret-db29abb9d806bf0ea8690fda8bc7d9cb                        Compliant          10h
amagrawa-10n-2                   busybox-workloads-13.vs-secret-8f116547aba3d58497d1c6440090c6c4                        Compliant          10h
amagrawa-10n-2                   busybox-workloads-14.vs-secret-3afb3c0dc0b9646dceef4fe1bd3da4a6                        Compliant          10h
amagrawa-10n-2                   openshift-gitops.vs-secret-6492ecc11b403505c9f30eb84106ac1b                            Compliant          10h
amagrawa-10n-2                   openshift-gitops.vs-secret-9dbea625099ac8d1b144c293ee183452                            Compliant          10h
amagrawa-10n-2                   openshift-gitops.vs-secret-eadf8882df7cd2a1a52db6e4631e5e98                            Compliant          10h


All the secrets were complaint.

So I moved to the secondary cluster where destination pods were running but stuck for 40+hrs and then ran the below cmd:

amagrawa$ oc -n busybox-workloads-10 delete jobs --all
job.batch "volsync-rsync-tls-dst-busybox-pvc-1" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-10" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-11" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-12" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-13" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-14" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-15" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-16" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-17" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-18" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-19" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-2" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-20" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-3" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-4" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-5" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-6" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-7" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-8" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-9" deleted


amagrawa$ oc -n busybox-workloads-11 delete jobs --all
job.batch "volsync-rsync-tls-dst-busybox-pvc-1" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-10" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-11" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-12" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-13" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-14" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-15" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-16" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-17" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-18" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-19" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-2" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-20" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-3" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-4" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-5" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-6" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-7" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-8" deleted
job.batch "volsync-rsync-tls-dst-busybox-pvc-9" deleted


It re-created the dst pods on the secondary cluster and data sync resumes as soon as the next sync interval in reached.
I can confirm that the workaround works well and data sync is working fine for all the impacted cephfs workloads. 

@bmekhiss thank you so much for putting so much time and efforts on issues like this, you are a real saviour. :)
Comment 11 errata-xmlrpc 2023-12-07 13:21:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.14.1 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7696