Bug 2226557

Summary: Mgr and OSD Daemons crashed while running the tests on NFS mount
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: Ceph-Mgr PluginsAssignee: Radoslaw Zarzynski <rzarzyns>
Ceph-Mgr Plugins sub component: crash QA Contact: Sayalee <saraut>
Status: NEW --- Docs Contact: ceph-docs <ceph-docs>
Severity: urgent    
Priority: unspecified CC: cephqe-warriors, kkeithle, nojha, vdas
Version: 6.1   
Target Milestone: ---   
Target Release: 7.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2023-07-25 23:24:36 UTC 
Description of problem:
=============

** Raising this BZ under NFS component initially. Please change the component if required..


5 mgr modules and 3 osd daemons crashed while running specStorage tool on NFS mount.

[ceph: root@ceph-mani-fbdbej-node1-installer /]# ceph crash ls-new
ID                                                                ENTITY                                       NEW  
2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317  mgr.ceph-mani-fbdbej-node3.giemdg             *   
2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a  mgr.ceph-mani-fbdbej-node1-installer.ehufrt   *   
2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8  osd.11                                        *   
2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306  osd.5                                         *   
2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d  osd.8                                         *   
2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf  mgr.ceph-mani-fbdbej-node1-installer.ehufrt   *   
2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f  mgr.ceph-mani-fbdbej-node3.giemdg             *   
2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6  mgr.ceph-mani-fbdbej-node3.giemdg             *   




Version-Release number of selected component (if applicable):

========

# ceph --version
ceph version 17.2.6-76.el9cp (7d277f1e8500eb73e50260771e11b7bd7d6f34af) quincy (stable)

# rpm -qa | grep ceph
libcephfs2-17.2.6-76.el9cp.x86_64
libcephsqlite-17.2.6-76.el9cp.x86_64
python3-ceph-common-17.2.6-76.el9cp.x86_64
python3-ceph-argparse-17.2.6-76.el9cp.x86_64
python3-cephfs-17.2.6-76.el9cp.x86_64
cephadm-17.2.6-76.el9cp.noarch
ceph-prometheus-alerts-17.2.6-76.el9cp.noarch
ceph-grafana-dashboards-17.2.6-76.el9cp.noarch
ceph-common-17.2.6-76.el9cp.x86_64
ceph-base-17.2.6-76.el9cp.x86_64
ceph-selinux-17.2.6-76.el9cp.x86_64
ceph-mgr-cephadm-17.2.6-76.el9cp.noarch
ceph-mgr-dashboard-17.2.6-76.el9cp.noarch
ceph-mgr-diskprediction-local-17.2.6-76.el9cp.noarch
ceph-mgr-k8sevents-17.2.6-76.el9cp.noarch
ceph-mgr-modules-core-17.2.6-76.el9cp.noarch
ceph-mgr-17.2.6-76.el9cp.x86_64
ceph-mgr-rook-17.2.6-76.el9cp.noarch
ceph-volume-17.2.6-76.el9cp.noarch
ceph-osd-17.2.6-76.el9cp.x86_64
ceph-radosgw-17.2.6-76.el9cp.x86_64
ceph-immutable-object-cache-17.2.6-76.el9cp.x86_64
ceph-exporter-17.2.6-76.el9cp.x86_64
ceph-mds-17.2.6-76.el9cp.x86_64
ceph-mon-17.2.6-76.el9cp.x86_64
cephfs-mirror-17.2.6-76.el9cp.x86_64
nfs-ganesha-ceph-5.3-1.el9cp.x86_64
cephfs-top-17.2.6-76.el9cp.noarch



How reproducible:


Steps to Reproduce:
1.Deploy 3 node ceph cluster and configure ganesha cluster on 1 node

# ceph nfs cluster info nfsganesha
{
    "nfsganesha": {
        "virtual_ip": null,
        "backend": [
            {
                "hostname": "ceph-mani-fbdbej-node2",
                "ip": "10.0.211.204",
                "port": 2049
            },
            {
                "hostname": "ceph-mani-fbdbej-node3",
                "ip": "10.0.208.236",
                "port": 2049
            }
        ]
    }
}


2. Export an cephfs volume

# ceph nfs export get nfsganesha /ceph  
{
  "export_id": 1,
  "path": "/",
  "cluster_id": "nfsganesha",
  "pseudo": "/ceph",
  "access_type": "RW",
  "squash": "none",
  "security_label": true,
  "protocols": [
    4
  ],
  "transports": [
    "TCP"
  ],
  "fsal": {
    "name": "CEPH",
    "user_id": "nfs.nfsganesha.1",
    "fs_name": "cephfs"
  },
  "clients": []
}


3.Mount the volume on client vers=4.1 

10.0.211.204:/ceph  86810624 2670592  84140032   4% /mnt/ganesha
tmpfs                 374800       0    374800   0% /run/user/0


4. Run specStorage tool on NFS-mount



Actual results:
=======
Mgr and osd daemons got crashed


Expected results:
======
No daemons should be crashed


Additional info:
------------

# ceph -s
  cluster:
    id:     905ded00-279e-11ee-89a6-fa163eb4f962
    health: HEALTH_WARN
            3 daemons have recently crashed
            5 mgr modules have recently crashed
 
  services:
    mon: 3 daemons, quorum ceph-mani-fbdbej-node1-installer,ceph-mani-fbdbej-node3,ceph-mani-fbdbej-node2 (age 67m)
    mgr: ceph-mani-fbdbej-node3.giemdg(active, since 26m), standbys: ceph-mani-fbdbej-node1-installer.ehufrt
    mds: 1/1 daemons up, 1 standby
    osd: 18 osds: 18 up (since 67m), 18 in (since 4d)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 241 pgs
    objects: 164.68k objects, 2.7 GiB
    usage:   12 GiB used, 258 GiB / 270 GiB avail
    pgs:     241 active+clean
 
  io:
    client:   170 B/s rd, 0 op/s rd, 0 op/s wr

===========================

Crash logs -
============================

# ceph crash info 2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n    if self.db_ready() and self.enable_monitoring:",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n    return self.db is not None",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n    self._db = self.open_db()",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n    self.configure_db(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1202, in configure_db\n    self.load_schema(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1188, in load_schema\n    self.create_skeleton_schema(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1129, in create_skeleton_schema\n    db.executescript(SQL)",
        "sqlite3.InternalError: unknown operation"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317",
    "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg",
    "mgr_module": "devicehealth",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "InternalError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-mgr",
    "stack_sig": "b172482b84937fd5ed6878988d42619765d745a0dd0c326018e99429e3abfbfd",
    "timestamp": "2023-07-24T22:02:53.030749Z",
    "utsname_hostname": "ceph-mani-fbdbej-node3",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}

-----------

# ceph crash info 2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a


{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n    if self.db_ready() and self.enable_monitoring:",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n    return self.db is not None",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n    self._db = self.open_db()",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n    self.configure_db(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1197, in configure_db\n    db.execute('PRAGMA JOURNAL_MODE = PERSIST')",
        "sqlite3.OperationalError: disk I/O error"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a",
    "entity_name": "mgr.ceph-mani-fbdbej-node1-installer.ehufrt",
    "mgr_module": "devicehealth",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "OperationalError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-mgr",
    "stack_sig": "5af484bd2ccb176d87ab56dbe6df8a9d6a592c3af56ebf0fb27d242cddc7245e",
    "timestamp": "2023-07-24T22:12:16.010086Z",
    "utsname_hostname": "ceph-mani-fbdbej-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}

------------------


# ceph crash info 2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 623, in serve\n    serve.serve()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 72, in serve\n    self._refresh_hosts_and_daemons()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 274, in _refresh_hosts_and_daemons\n    refresh(self.mgr.cache.get_hosts())",
        "  File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 85, in forall_hosts_wrapper\n    return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 364, in map\n    return self._map_async(func, iterable, mapstar, chunksize).get()",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 771, in get\n    raise self._value",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 125, in worker\n    result = (True, func(*args, **kwds))",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 48, in mapstar\n    return list(map(*args))",
        "  File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 79, in do_work\n    return f(*arg)",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 223, in refresh\n    r = self._refresh_host_daemons(host)",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 321, in _refresh_host_daemons\n    ls = self.mgr.wait_async(self._run_cephadm_json(",
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 635, in wait_async\n    return self.event_loop.get_result(coro, timeout)",
        "  File \"/usr/share/ceph/mgr/cephadm/ssh.py\", line 63, in get_result\n    return future.result(timeout)",
        "  File \"/lib64/python3.9/concurrent/futures/_base.py\", line 448, in result\n    raise TimeoutError()",
        "concurrent.futures._base.TimeoutError"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf",
    "entity_name": "mgr.ceph-mani-fbdbej-node1-installer.ehufrt",
    "mgr_module": "cephadm",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "TimeoutError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-mgr",
    "stack_sig": "d703d70abf65004104ed51a965966558d39bdd6eba65a48554e6dbd25c7c54fc",
    "timestamp": "2023-07-25T04:45:09.656829Z",
    "utsname_hostname": "ceph-mani-fbdbej-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}


-------------------------


# ceph crash info  2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8



{
    "backtrace": [
        "/lib64/libc.so.6(+0x54df0) [0x7f47f32fedf0]",
        "/lib64/libc.so.6(+0x9c560) [0x7f47f3346560]",
        "pthread_mutex_lock()",
        "(rocksdb::port::Mutex::Lock()+0x9) [0x558b9dd363e9]",
        "(rocksdb::InstrumentedMutex::Lock()+0xa9) [0x558b9dc7b289]",
        "(rocksdb::DBImpl::GetIntPropertyInternal(rocksdb::ColumnFamilyData*, rocksdb::DBPropertyInfo const&, bool, unsigned long*)+0x35) [0x558b9db2fe15]",
        "(RocksDBStore::get_property(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long*)+0x3b) [0x558b9da0812b]",
        "(BlueStore::BlueStoreThrottle::emit_initial_tracepoint(KeyValueDB&, BlueStore::TransContext&, std::chrono::time_point<ceph::mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x11b) [0x558b9d62992b]",
        "(BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xa82) [0x558b9d60f862]",
        "(ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xb37) [0x558b9d4bf847]",
        "(ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1c7) [0x558b9d4b9617]",
        "(PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x56) [0x558b9d322596]",
        "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x79d) [0x558b9d29013d]",
        "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x191) [0x558b9d1ed3b1]",
        "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x6d) [0x558b9d3c8edd]",
        "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf7) [0x558b9d2035e7]",
        "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2a3) [0x558b9d71d7d3]",
        "/usr/bin/ceph-osd(+0xa7ed74) [0x558b9d71dd74]",
        "/lib64/libc.so.6(+0x9f802) [0x7f47f3349802]",
        "/lib64/libc.so.6(+0x3f450) [0x7f47f32e9450]"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8",
    "entity_name": "osd.11",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-osd",
    "stack_sig": "e639c225ff1709ed8ca59e3a0888ac1a956809690995d0688383763f2e95b1bc",
    "timestamp": "2023-07-25T00:18:55.467396Z",
    "utsname_hostname": "ceph-mani-fbdbej-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}
-------------------

# ceph crash info 2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n    if self.db_ready() and self.enable_monitoring:",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n    return self.db is not None",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n    self._db = self.open_db()",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n    self.configure_db(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1202, in configure_db\n    self.load_schema(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1188, in load_schema\n    self.create_skeleton_schema(db)",
        "  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1129, in create_skeleton_schema\n    db.executescript(SQL)",
        "sqlite3.InternalError: unknown operation"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f",
    "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg",
    "mgr_module": "devicehealth",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "InternalError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-mgr",
    "stack_sig": "b172482b84937fd5ed6878988d42619765d745a0dd0c326018e99429e3abfbfd",
    "timestamp": "2023-07-25T05:12:16.182527Z",
    "utsname_hostname": "ceph-mani-fbdbej-node3",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}

---------------

# ceph crash info 2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 623, in serve\n    serve.serve()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 72, in serve\n    self._refresh_hosts_and_daemons()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 274, in _refresh_hosts_and_daemons\n    refresh(self.mgr.cache.get_hosts())",
        "  File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 85, in forall_hosts_wrapper\n    return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 364, in map\n    return self._map_async(func, iterable, mapstar, chunksize).get()",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 771, in get\n    raise self._value",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 125, in worker\n    result = (True, func(*args, **kwds))",
        "  File \"/lib64/python3.9/multiprocessing/pool.py\", line 48, in mapstar\n    return list(map(*args))",
        "  File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 79, in do_work\n    return f(*arg)",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 223, in refresh\n    r = self._refresh_host_daemons(host)",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 321, in _refresh_host_daemons\n    ls = self.mgr.wait_async(self._run_cephadm_json(",
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 635, in wait_async\n    return self.event_loop.get_result(coro, timeout)",
        "  File \"/usr/share/ceph/mgr/cephadm/ssh.py\", line 63, in get_result\n    return future.result(timeout)",
        "  File \"/lib64/python3.9/concurrent/futures/_base.py\", line 448, in result\n    raise TimeoutError()",
        "concurrent.futures._base.TimeoutError"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6",
    "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg",
    "mgr_module": "cephadm",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "TimeoutError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-mgr",
    "stack_sig": "d703d70abf65004104ed51a965966558d39bdd6eba65a48554e6dbd25c7c54fc",
    "timestamp": "2023-07-25T19:05:45.684434Z",
    "utsname_hostname": "ceph-mani-fbdbej-node3",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}

----------

# ceph crash info 2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d
{
    "backtrace": [
        "/lib64/libc.so.6(+0x54df0) [0x7f02e6ee1df0]",
        "/lib64/libc.so.6(+0x9c560) [0x7f02e6f29560]",
        "pthread_mutex_lock()",
        "(PG::lock(bool) const+0x2b) [0x56389add067b]",
        "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x45d) [0x56389ada7e4d]",
        "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2a3) [0x56389b2c27d3]",
        "/usr/bin/ceph-osd(+0xa7ed74) [0x56389b2c2d74]",
        "/lib64/libc.so.6(+0x9f802) [0x7f02e6f2c802]",
        "/lib64/libc.so.6(+0x3f450) [0x7f02e6ecc450]"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d",
    "entity_name": "osd.8",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-osd",
    "stack_sig": "1b35d485fce29e5a7b370206b12bed88e2a2243e53252b49251772880c953e2e",
    "timestamp": "2023-07-25T00:18:55.479102Z",
    "utsname_hostname": "ceph-mani-fbdbej-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"

-------------

# ceph crash info 2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306
{
    "backtrace": [
        "/lib64/libc.so.6(+0x54df0) [0x7f7e05a88df0]",
        "/lib64/libc.so.6(+0xa154c) [0x7f7e05ad554c]",
        "(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x27b) [0x55649bfbfe8b]",
        "(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >)+0x72) [0x55649bfbff82]",
        "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0x55649bfd37c0]",
        "/usr/bin/ceph-osd(+0xa7ed74) [0x55649bfd3d74]",
        "/lib64/libc.so.6(+0x9f802) [0x7f7e05ad3802]",
        "/lib64/libc.so.6(+0x3f450) [0x7f7e05a73450]"
    ],
    "ceph_version": "17.2.6-76.el9cp",
    "crash_id": "2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306",
    "entity_name": "osd.5",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.2 (Plow)",
    "os_version_id": "9.2",
    "process_name": "ceph-osd",
    "stack_sig": "888002e9e1b0600789f244e0281cf9b8de10b8587b13301b5b41854cf1891534",
    "timestamp": "2023-07-25T00:18:55.478671Z",
    "utsname_hostname": "ceph-mani-fbdbej-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.18.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023"
}

-------