Description of problem (please be detailed as possible and provide log
snippests):
[RDR] Osd migration has halted in between

Version of all relevant components (if applicable):

OCP version:- 4.15.0-0.nightly-2024-01-13-050900
ODF version:- 4.15.0-113
CEPH version:- ceph version 17.2.6-167.el9cp (5ef1496ea3e9daaa9788809a172bd5a1c3192cf7) quincy (stable)
ACM version:- 2.9.1
SUBMARINER version:- v0.16.2

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?
Restart rook-operator pod

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy Cluster without enabling it for RDR 
2.Write some data
3.Start Migation from UI


Actual results:
Migration of osd stopped in between 

storeType:
          bluestore: 2
          bluestore-rdr: 4

#cephstatus
  cluster:
    id:     303b3709-c0e9-4a3c-9352-923b1e9eef1f
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum d,e,f (age 25h)
    mgr: a(active, since 5d), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 6 osds: 6 up (since 21h), 6 in (since 21h)
    rgw: 1 daemon active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 169 pgs
    objects: 100.64k objects, 384 GiB
    usage:   1.1 TiB used, 11 TiB / 12 TiB avail
    pgs:     169 active+clean
 
  io:
    client:   305 MiB/s rd, 6.5 MiB/s wr, 598 op/s rd, 21 op/s wr
 


Expected results:
Migation should not stopped in between 

Additional info:

Must-gather:- Logs location: http://rhsqe-repo.lab.eng.blr.redhat.com/ocs4qe/pratik/bz/migation_stop/jan19/19-01-2024_11-58-17