Bug 2094441

Summary: [DR] mgr daemons crash
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Pratik Surve <prsurve>
Component: cephAssignee: Prasanna Kumar Kalever <prasanna.kalever>
ceph sub component: RBD-Mirror QA Contact: Elad <ebenahar>
Status: CLOSED WORKSFORME Docs Contact:
Severity: unspecified    
Priority: unspecified CC: amagrawa, bniver, idryomov, muagarwa, odf-bz-bot, prasanna.kalever
Version: 4.10Flags: amagrawa: needinfo? (prasanna.kalever)
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-08 12:58:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pratik Surve 2022-06-07 15:33:44 UTC 
Description of problem (please be detailed as possible and provide log
snippets):

mgr daemons crash

Version of all relevant components (if applicable):
OCP version:- 4.10.0-0.nightly-2022-05-26-102501
ODF version:- 4.10.3-4
CEPH version:- ceph version 16.2.7-112.el8cp (e18db2ff03ac60c64a18f3315c032b9d5a0a3b8f) pacific (stable)


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy RDR cluster
2. Run io for a longer time 
3.


Actual results:

$ ceph crash ls
ID                                                                ENTITY  NEW  
2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7  mgr.a    *   

bash-4.4$ ceph crash info 2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7
{
    "assert_condition": "m_pending_ops == 0",
    "assert_file": "/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc",
    "assert_func": "AsyncOpTracker::~AsyncOpTracker()",
    "assert_line": 13,
    "assert_msg": "/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f1344655700 time 2022-06-06T22:00:11.836444+0000\n/builddir/build/BUILD/ceph-16.2.7/src/common/AsyncOpTracker.cc: 13: FAILED ceph_assert(m_pending_ops == 0)\n",
    "assert_thread_name": "io_context_pool",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7f13efddcce0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f13f0faed4f]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x276f18) [0x7f13f0faef18]",
        "(AsyncOpTracker::~AsyncOpTracker()+0x53) [0x7f13f103bd23]",
        "/lib64/librbd.so.1(+0x13955b) [0x7f13d8eb455b]",
        "/lib64/librbd.so.1(+0x1395cd) [0x7f13d8eb45cd]",
        "/lib64/librbd.so.1(+0x10fa2a) [0x7f13d8e8aa2a]",
        "/lib64/librbd.so.1(+0x26fee0) [0x7f13d8feaee0]",
        "/lib64/librbd.so.1(+0x275838) [0x7f13d8ff0838]",
        "/lib64/librbd.so.1(+0x1074bd) [0x7f13d8e824bd]",
        "/lib64/librbd.so.1(+0x12a186) [0x7f13d8ea5186]",
        "/lib64/librbd.so.1(+0x3c9e05) [0x7f13d9144e05]",
        "/lib64/librados.so.2(+0xc3692) [0x7f13e12d2692]",
        "/lib64/librados.so.2(+0xc909a) [0x7f13e12d809a]",
        "/lib64/libstdc++.so.6(+0xc2ba3) [0x7f13ef20dba3]",
        "/lib64/libpthread.so.0(+0x81cf) [0x7f13efdd21cf]",
        "clone()"
    ],
    "ceph_version": "16.2.7-112.el8cp",
    "crash_id": "2022-06-06T22:00:11.838611Z_1f241902-1850-42fd-b370-15a65ad104d7",
    "entity_name": "mgr.a",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "8.6 (Ootpa)",
    "os_version_id": "8.6",
    "process_name": "ceph-mgr",
    "stack_sig": "74c11b1432d4df3936d01e0ec855e7a779d2807d5641b694e257fc3d2db07db7",
    "timestamp": "2022-06-06T22:00:11.838611Z",
    "utsname_hostname": "rook-ceph-mgr-a-766ff4df84-2c6vm",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-305.45.1.el8_4.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Wed Apr 6 13:48:37 EDT 2022"
}


Expected results:
There should not be any crash

Additional info:
Comment 3 Mudit Agarwal 2022-07-05 13:18:22 UTC 
Not a 4.11 blocker
Comment 5 Mudit Agarwal 2022-10-26 03:31:48 UTC 
Prasannna, can you take a look?