Description of problem (please be detailed as possible and provide log
snippests):

ceph-mon crashed while running e2e flow tests

[root@compute-2 /]# ceph crash  info 2021-05-21_15:32:56.924290Z_dca2e592-cf0b-4d65-97f0-02f8694f94ae 
{
    "crash_id": "2021-05-21_15:32:56.924290Z_dca2e592-cf0b-4d65-97f0-02f8694f94ae",
    "timestamp": "2021-05-21 15:32:56.924290Z",
    "process_name": "ceph-mon",
    "entity_name": "mon.c",
    "ceph_version": "14.2.11-147.el8cp",
    "utsname_hostname": "rook-ceph-mon-c-597dc9bb4b-7jwvz",
    "utsname_sysname": "Linux",
    "utsname_release": "4.18.0-305.el8.x86_64",
    "utsname_version": "#1 SMP Thu Apr 29 08:54:30 EDT 2021",
    "utsname_machine": "x86_64",
    "os_name": "Red Hat Enterprise Linux",
    "os_id": "rhel",
    "os_version_id": "8.3",
    "os_version": "8.3 (Ootpa)",
    "assert_condition": "abort",
    "assert_func": "int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)",
    "assert_file": "/builddir/build/BUILD/ceph-14.2.11/src/mon/MonitorDBStore.h",
    "assert_line": 354,
    "assert_thread_name": "safe_timer",
    "assert_msg": "/builddir/build/BUILD/ceph-14.2.11/src/mon/MonitorDBStore.h: In function 'int MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)' thread 7fe07e800700 time 2021-05-21 15:32:56.921312\n/builddir/build/BUILD/ceph-14.2.11/src/mon/MonitorDBStore.h: 354: ceph_abort_msg(\"failed to write to db\")\n",
    "backtrace": [
        "(()+0x12b30) [0x7fe08880ab30]",
        "(gsignal()+0x10f) [0x7fe08747284f]",
        "(abort()+0x127) [0x7fe08745cc45]",
        "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b4) [0x7fe08bb01015]",
        "(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0xe8c) [0x5569eca613fc]",
        "(Elector::persist_connectivity_scores()+0xe9) [0x5569ecb29979]",
        "(ConnectionTracker::increase_version()+0x1bd) [0x5569ecb3576d]",
        "(ConnectionTracker::report_dead_connection(int, double)+0xf0) [0x5569ecb35f60]",
        "(Elector::dead_ping(int)+0x15e) [0x5569ecb28c4e]",
        "(C_MonContext::finish(int)+0x3d) [0x5569eca6479d]",
        "(Context::complete(int)+0xd) [0x5569ecaa951d]",
        "(SafeTimer::timer_thread()+0x1aa) [0x7fe08bbdd37a]",
        "(SafeTimerThread::entry()+0x11) [0x7fe08bbdecc1]",
        "(()+0x815a) [0x7fe08880015a]",
        "(clone()+0x43) [0x7fe087537f73]"
    ]
}
Version of all relevant components (if applicable):

ocp 4.8.0-0.nightly-2021-05-19-123944
rook: 4.8-133.5daada2.release_4.8
ceph version 14.2.11-147.el8cp (1f54d52f20d93c1b91f1ec6af4c67a4b81402800) nautilus (stable)
ocs-operator.v4.8.0-399.ci

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
NA

Is there any workaround available to the best of your knowledge?
Run ceph crash archive 2021-05-21_15:32:56.924290Z_dca2e592-cf0b-4d65-97f0-02f8694f94ae 
Ceph health will be OK

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?
NA

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. https://github.com/red-hat-storage/ocs-ci/pull/4019/files#diff-2940650849aca221f3591c75030e3b4b3f68e7a7dd0f4389895bf93e0850f73dR174
        Test covers the following flow operations while running workloads in the background:
        1. Node reboot
        2. Device replacement
        3. Nooba Bucket policy: put, modify, delete
        4. Node drain
        5. Node n/w failure --> Crash occurred while this running step
        6. nooba core pod delete with obc IO
2.
3.


Actual results:
Ceph-mon crash

Expected results:
Ceph-mon should not have crashed

Additional info:
After running ceph archive cmd, IO operations worked fine.