Description of problem (please be detailed as possible and provide log
snippets):

mgr daemons crash

Version of all relevant components (if applicable):
OCP version:- 4.10.0-0.nightly-2022-05-26-102501
ODF version:- 4.10.3-4
CEPH version:- ceph version 16.2.7-112.el8cp (e18db2ff03ac60c64a18f3315c032b9d5a0a3b8f) pacific (stable)


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy RDR cluster
2. Run io for a longer time 
3.


Actual results:

$ ceph crash ls
ID                                                                ENTITY  NEW  
2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7  mgr.a    *   

bash-4.4$ ceph crash info 2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7
{
    "assert_condition": "m_pending_ops == 0",
    "assert_file": "/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc",
    "assert_func": "AsyncOpTracker::~AsyncOpTracker()",
    "assert_line": 13,
    "assert_msg": "/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f1344655700 time 2022-06-06T22:00:11.836444+0000\n/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)\n",
    "assert_thread_name": "io_context_pool",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7f13efddcce0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f13f0faed4f]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x276f18) [0x7f13f0faef18]",
        "(AsyncOpTracker::~AsyncOpTracker()+0x53) [0x7f13f103bd23]",
        "/lib64/librbd.so.1(+0x13955b) [0x7f13d8eb455b]",
        "/lib64/librbd.so.1(+0x1395cd) [0x7f13d8eb45cd]",
        "/lib64/librbd.so.1(+0x10fa2a) [0x7f13d8e8aa2a]",
        "/lib64/librbd.so.1(+0x26fee0) [0x7f13d8feaee0]",
        "/lib64/librbd.so.1(+0x275838) [0x7f13d8ff0838]",
        "/lib64/librbd.so.1(+0x1074bd) [0x7f13d8e824bd]",
        "/lib64/librbd.so.1(+0x12a186) [0x7f13d8ea5186]",
        "/lib64/librbd.so.1(+0x3c9e05) [0x7f13d9144e05]",
        "/lib64/librados.so.2(+0xc3692) [0x7f13e12d2692]",
        "/lib64/librados.so.2(+0xc909a) [0x7f13e12d809a]",
        "/lib64/libstdc++.so.6(+0xc2ba3) [0x7f13ef20dba3]",
        "/lib64/libpthread.so.0(+0x81cf) [0x7f13efdd21cf]",
        "clone()"
    ],
    "ceph_version": "16.2.7-112.el8cp",
    "crash_id": "2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7",
    "entity_name": "mgr.a",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "8.6 (Ootpa)",
    "os_version_id": "8.6",
    "process_name": "ceph-mgr",
    "stack_sig": "74c11b1432d4df3936d01e0ec855e7a779d2807d5641b694e257fc3d2db07db7",
    "timestamp": "2022-06-06T22:00:11.838611Z",
    "utsname_hostname": "rook-ceph-mgr-a-766ff4df84-2c6vm",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-305.45.1.el8_4.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Wed Apr 6 13:48:37 EDT 2022"
}


Expected results:
There should not be any crash

Additional info: