Description of problem (please be detailed as possible and provide log
snippests):
[RDR][CEPHFS] sync/replication is getting stopped for some pvc rsync: connection unexpectedly closed

Version of all relevant components (if applicable):

OCP version:- 4.12.0-0.nightly-2023-01-19-110743
ODF version:- 4.12.0-167
CEPH version:- ceph version 16.2.10-94.el8cp (48ce8ed67474ea50f10c019b9445be7f49749d23) pacific (stable)
ACM version:- v2.7.0
SUBMARINER version:- v0.14.1
VOLSYNC version:- volsync-product.v0.6.0

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy RDR cluster
2.Create CEPHFS workload
3.check replicationsource status after 4-5 days 


Actual results:
oc get replicationsources --all-namespaces

NAMESPACE             NAME             SOURCE           LAST SYNC              DURATION          NEXT SYNC
busybox-workloads-1   busybox-pvc-1    busybox-pvc-1    2023-01-31T18:29:42Z   4m42.43785306s    2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-10   busybox-pvc-10   2023-01-31T18:29:44Z   4m44.270959207s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-11   busybox-pvc-11   2023-01-31T18:29:39Z   4m39.561215075s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-12   busybox-pvc-12   2023-01-31T18:29:39Z   4m39.505161496s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-13   busybox-pvc-13   2023-01-31T18:29:39Z   4m39.392102602s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-14   busybox-pvc-14   2023-01-29T09:23:46Z   3m46.2123443s     2023-01-29T09:25:00Z
busybox-workloads-1   busybox-pvc-15   busybox-pvc-15   2023-01-31T18:29:38Z   4m38.684680219s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-16   busybox-pvc-16   2023-01-31T18:29:44Z   4m44.940480011s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-17   busybox-pvc-17   2023-01-31T18:29:46Z   4m46.939628275s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-18   busybox-pvc-18   2023-01-31T18:29:49Z   4m49.386471097s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-19   busybox-pvc-19   2023-01-31T18:29:46Z   4m46.9576977s     2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-2    busybox-pvc-2    2023-01-31T18:29:44Z   4m44.210365128s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-20   busybox-pvc-20   2023-01-31T18:29:20Z   4m20.176716233s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-3    busybox-pvc-3    2023-01-31T18:29:44Z   4m44.252856471s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-4    busybox-pvc-4    2023-01-31T18:29:42Z   4m42.463537101s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-5    busybox-pvc-5    2023-01-31T18:29:39Z   4m39.544973999s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-6    busybox-pvc-6    2023-01-31T18:29:41Z   4m41.734123311s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-7    busybox-pvc-7    2023-01-31T18:29:24Z   4m24.746735678s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-8    busybox-pvc-8    2023-01-31T18:29:23Z   4m23.763610806s   2023-01-31T18:30:00Z
busybox-workloads-1   busybox-pvc-9    busybox-pvc-9    2023-01-31T18:29:24Z   4m24.778197239s   2023-01-31T18:30:00Z
busybox-workloads-1   mysql-pv-claim   mysql-pv-claim   2023-01-31T18:25:49Z   5m49.531404022s   2023-01-31T18:30:00Z



$oc logs volsync-rsync-src-busybox-pvc-14-xjxj6                                                    
VolSync rsync container version: ACM-0.6.0-ce9a280
Syncing data to volsync-rsync-dst-busybox-pvc-14.busybox-workloads-1.svc.clusterset.local:22 ...
ssh: Could not resolve hostname volsync-rsync-dst-busybox-pvc-14.busybox-workloads-1.svc.clusterset.local: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
Syncronization failed. Retrying in 2 seconds. Retry 1/5.
ssh: Could not resolve hostname volsync-rsync-dst-busybox-pvc-14.busybox-workloads-1.svc.clusterset.local: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
Syncronization failed. Retrying in 4 seconds. Retry 2/5.
ssh: Could not resolve hostname volsync-rsync-dst-busybox-pvc-14.busybox-workloads-1.svc.clusterset.local: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
Syncronization failed. Retrying in 8 seconds. Retry 3/5.
ssh: Could not resolve hostname volsync-rsync-dst-busybox-pvc-14.busybox-workloads-1.svc.clusterset.local: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]
Syncronization failed. Retrying in 16 seconds. Retry 4/5.


$replication is stop for volsync-rsync-dst-busybox-pvc-14-pz5jk   1/1       Running   0          2d9h



Expected results:


Additional info: