Description of problem (please be detailed as possible and provide log
snippests):
[RDR][CEPHFS] volsync-src-pvc  ROX clone is taking more time than expected

Version of all relevant components (if applicable):

OCP version:- 4.12.0-0.nightly-2022-12-16-075715
ODF version:- 4.12.0-145
CEPH version:- ceph version 16.2.10-82.el8cp (cfa37337c5ab43bd0ad7bb4418af8791c83b2785) pacific (stable)
ACM version:- 2.7
SUBMARINER version:- 0.14
VOLSYNC version:- volsync-product.v0.6.0


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy RDR cluster
2.Deploy cephfs workload
3.check PVC status of workload after 1,2 days of running workload 


Actual results:
oc describe pvc volsync-dd-io-pvc-2-src
Name:          volsync-dd-io-pvc-2-src
Namespace:     busybox-workloads-8
StorageClass:  ocs-storagecluster-cephfs-vrg
Status:        Pending
Volume:        
Labels:        app.kubernetes.io/created-by=volsync
               volsync.backube/cleanup=3a8540b7-d043-439d-a7dc-54a98a694e2e
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      volsync-dd-io-pvc-2-src
Used By:     volsync-rsync-src-dd-io-pvc-2-gsw4c
Events:
  Type     Reason                Age                   From                                                                                                                      Message
  ----     ------                ----                  ----                                                                                                                      -------
  Warning  ProvisioningFailed    6m8s (x2 over 10m)    openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-76b7d8bb97-n2jsg_27f2fb73-60e8-4b5e-b97c-4ce77aec0bf2  failed to provision volume with StorageClass "ocs-storagecluster-cephfs-vrg": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  ProvisioningFailed    6m8s (x6 over 10m)    openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-76b7d8bb97-n2jsg_27f2fb73-60e8-4b5e-b97c-4ce77aec0bf2  failed to provision volume with StorageClass "ocs-storagecluster-cephfs-vrg": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-fcfdda07-84ee-417a-bf5f-3ee28bc3e288 already exists
  Warning  ProvisioningFailed    5m1s (x2 over 8m38s)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-76b7d8bb97-n2jsg_27f2fb73-60e8-4b5e-b97c-4ce77aec0bf2  failed to provision volume with StorageClass "ocs-storagecluster-cephfs-vrg": rpc error: code = Internal desc = volume not found
  Normal   ExternalProvisioning  2m38s (x44 over 12m)  persistentvolume-controller                                                                                               waiting for a volume to be created, either by external provisioner "openshift-storage.cephfs.csi.ceph.com" or manually created by system administrator
  Normal   Provisioning          45s (x11 over 12m)    openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-76b7d8bb97-n2jsg_27f2fb73-60e8-4b5e-b97c-4ce77aec0bf2  External provisioner is provisioning volume for claim "busybox-workloads-8/volsync-dd-io-pvc-2-src"


oc get pvc -w                                           
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
dd-io-pvc-1               Bound     pvc-2d8417f0-c42c-44d0-8c86-3e87b6e1860b   117Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-2               Bound     pvc-36651ad3-d2d7-4acc-81a7-81a1535bfca8   143Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-3               Bound     pvc-211d79db-1878-44ac-87a1-f27ffcef3d7b   134Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-4               Bound     pvc-40ead61d-43f6-4d2f-86e6-5abfc168ad1c   106Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-5               Bound     pvc-48e6e532-94b7-4520-b6ef-1887785aec04   115Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-6               Bound     pvc-c95b8881-c419-4677-9c0f-56ea42940af3   129Gi      RWX            ocs-storagecluster-cephfs       3d13h
dd-io-pvc-7               Bound     pvc-be4a69e7-8c77-473f-afe7-e464db59b1a8   149Gi      RWX            ocs-storagecluster-cephfs       3d13h
volsync-dd-io-pvc-2-src   Pending                                                                        ocs-storagecluster-cephfs-vrg   53m
volsync-dd-io-pvc-3-src   Pending                                                                        ocs-storagecluster-cephfs-vrg   2m22s
volsync-dd-io-pvc-6-src   Pending                                                                        ocs-storagecluster-cephfs-vrg   2m22s
volsync-dd-io-pvc-7-src   Pending                                                                        ocs-storagecluster-cephfs-vrg   2m22s



Expected results:


Additional info: