Description of problem: ============= ** Raising this BZ under NFS component initially. Please change the component if required.. 5 mgr modules and 3 osd daemons crashed while running specStorage tool on NFS mount. [ceph: root@ceph-mani-fbdbej-node1-installer /]# ceph crash ls-new ID ENTITY NEW 2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317 mgr.ceph-mani-fbdbej-node3.giemdg * 2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a mgr.ceph-mani-fbdbej-node1-installer.ehufrt * 2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8 osd.11 * 2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306 osd.5 * 2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d osd.8 * 2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf mgr.ceph-mani-fbdbej-node1-installer.ehufrt * 2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f mgr.ceph-mani-fbdbej-node3.giemdg * 2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6 mgr.ceph-mani-fbdbej-node3.giemdg * Version-Release number of selected component (if applicable): ======== # ceph --version ceph version 17.2.6-76.el9cp (7d277f1e8500eb73e50260771e11b7bd7d6f34af) quincy (stable) # rpm -qa | grep ceph libcephfs2-17.2.6-76.el9cp.x86_64 libcephsqlite-17.2.6-76.el9cp.x86_64 python3-ceph-common-17.2.6-76.el9cp.x86_64 python3-ceph-argparse-17.2.6-76.el9cp.x86_64 python3-cephfs-17.2.6-76.el9cp.x86_64 cephadm-17.2.6-76.el9cp.noarch ceph-prometheus-alerts-17.2.6-76.el9cp.noarch ceph-grafana-dashboards-17.2.6-76.el9cp.noarch ceph-common-17.2.6-76.el9cp.x86_64 ceph-base-17.2.6-76.el9cp.x86_64 ceph-selinux-17.2.6-76.el9cp.x86_64 ceph-mgr-cephadm-17.2.6-76.el9cp.noarch ceph-mgr-dashboard-17.2.6-76.el9cp.noarch ceph-mgr-diskprediction-local-17.2.6-76.el9cp.noarch ceph-mgr-k8sevents-17.2.6-76.el9cp.noarch ceph-mgr-modules-core-17.2.6-76.el9cp.noarch ceph-mgr-17.2.6-76.el9cp.x86_64 ceph-mgr-rook-17.2.6-76.el9cp.noarch ceph-volume-17.2.6-76.el9cp.noarch ceph-osd-17.2.6-76.el9cp.x86_64 ceph-radosgw-17.2.6-76.el9cp.x86_64 ceph-immutable-object-cache-17.2.6-76.el9cp.x86_64 ceph-exporter-17.2.6-76.el9cp.x86_64 ceph-mds-17.2.6-76.el9cp.x86_64 ceph-mon-17.2.6-76.el9cp.x86_64 cephfs-mirror-17.2.6-76.el9cp.x86_64 nfs-ganesha-ceph-5.3-1.el9cp.x86_64 cephfs-top-17.2.6-76.el9cp.noarch How reproducible: Steps to Reproduce: 1.Deploy 3 node ceph cluster and configure ganesha cluster on 1 node # ceph nfs cluster info nfsganesha { "nfsganesha": { "virtual_ip": null, "backend": [ { "hostname": "ceph-mani-fbdbej-node2", "ip": "10.0.211.204", "port": 2049 }, { "hostname": "ceph-mani-fbdbej-node3", "ip": "10.0.208.236", "port": 2049 } ] } } 2. Export an cephfs volume # ceph nfs export get nfsganesha /ceph { "export_id": 1, "path": "/", "cluster_id": "nfsganesha", "pseudo": "/ceph", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfsganesha.1", "fs_name": "cephfs" }, "clients": [] } 3.Mount the volume on client vers=4.1 10.0.211.204:/ceph 86810624 2670592 84140032 4% /mnt/ganesha tmpfs 374800 0 374800 0% /run/user/0 4. Run specStorage tool on NFS-mount Actual results: ======= Mgr and osd daemons got crashed Expected results: ====== No daemons should be crashed Additional info: ------------ # ceph -s cluster: id: 905ded00-279e-11ee-89a6-fa163eb4f962 health: HEALTH_WARN 3 daemons have recently crashed 5 mgr modules have recently crashed services: mon: 3 daemons, quorum ceph-mani-fbdbej-node1-installer,ceph-mani-fbdbej-node3,ceph-mani-fbdbej-node2 (age 67m) mgr: ceph-mani-fbdbej-node3.giemdg(active, since 26m), standbys: ceph-mani-fbdbej-node1-installer.ehufrt mds: 1/1 daemons up, 1 standby osd: 18 osds: 18 up (since 67m), 18 in (since 4d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 9 pools, 241 pgs objects: 164.68k objects, 2.7 GiB usage: 12 GiB used, 258 GiB / 270 GiB avail pgs: 241 active+clean io: client: 170 B/s rd, 0 op/s rd, 0 op/s wr =========================== Crash logs - ============================ # ceph crash info 2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317 { "backtrace": [ " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n if self.db_ready() and self.enable_monitoring:", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n return self.db is not None", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n self._db = self.open_db()", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n self.configure_db(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1202, in configure_db\n self.load_schema(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1188, in load_schema\n self.create_skeleton_schema(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1129, in create_skeleton_schema\n db.executescript(SQL)", "sqlite3.InternalError: unknown operation" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-24T22:02:53.030749Z_2f21e62c-4a2d-4a3d-a23b-a1a14f72f317", "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg", "mgr_module": "devicehealth", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "InternalError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-mgr", "stack_sig": "b172482b84937fd5ed6878988d42619765d745a0dd0c326018e99429e3abfbfd", "timestamp": "2023-07-24T22:02:53.030749Z", "utsname_hostname": "ceph-mani-fbdbej-node3", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } ----------- # ceph crash info 2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a { "backtrace": [ " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n if self.db_ready() and self.enable_monitoring:", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n return self.db is not None", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n self._db = self.open_db()", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n self.configure_db(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1197, in configure_db\n db.execute('PRAGMA JOURNAL_MODE = PERSIST')", "sqlite3.OperationalError: disk I/O error" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-24T22:12:16.010086Z_4a7e5949-6017-4f4e-8920-25907461ae0a", "entity_name": "mgr.ceph-mani-fbdbej-node1-installer.ehufrt", "mgr_module": "devicehealth", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "OperationalError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-mgr", "stack_sig": "5af484bd2ccb176d87ab56dbe6df8a9d6a592c3af56ebf0fb27d242cddc7245e", "timestamp": "2023-07-24T22:12:16.010086Z", "utsname_hostname": "ceph-mani-fbdbej-node1-installer", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } ------------------ # ceph crash info 2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf { "backtrace": [ " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 623, in serve\n serve.serve()", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 72, in serve\n self._refresh_hosts_and_daemons()", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 274, in _refresh_hosts_and_daemons\n refresh(self.mgr.cache.get_hosts())", " File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 85, in forall_hosts_wrapper\n return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 364, in map\n return self._map_async(func, iterable, mapstar, chunksize).get()", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 771, in get\n raise self._value", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 125, in worker\n result = (True, func(*args, **kwds))", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 48, in mapstar\n return list(map(*args))", " File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 79, in do_work\n return f(*arg)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 223, in refresh\n r = self._refresh_host_daemons(host)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 321, in _refresh_host_daemons\n ls = self.mgr.wait_async(self._run_cephadm_json(", " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 635, in wait_async\n return self.event_loop.get_result(coro, timeout)", " File \"/usr/share/ceph/mgr/cephadm/ssh.py\", line 63, in get_result\n return future.result(timeout)", " File \"/lib64/python3.9/concurrent/futures/_base.py\", line 448, in result\n raise TimeoutError()", "concurrent.futures._base.TimeoutError" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T04:45:09.656829Z_7804c089-fadb-4367-8d6a-0f33c619a8bf", "entity_name": "mgr.ceph-mani-fbdbej-node1-installer.ehufrt", "mgr_module": "cephadm", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "TimeoutError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-mgr", "stack_sig": "d703d70abf65004104ed51a965966558d39bdd6eba65a48554e6dbd25c7c54fc", "timestamp": "2023-07-25T04:45:09.656829Z", "utsname_hostname": "ceph-mani-fbdbej-node1-installer", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } ------------------------- # ceph crash info 2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8 { "backtrace": [ "/lib64/libc.so.6(+0x54df0) [0x7f47f32fedf0]", "/lib64/libc.so.6(+0x9c560) [0x7f47f3346560]", "pthread_mutex_lock()", "(rocksdb::port::Mutex::Lock()+0x9) [0x558b9dd363e9]", "(rocksdb::InstrumentedMutex::Lock()+0xa9) [0x558b9dc7b289]", "(rocksdb::DBImpl::GetIntPropertyInternal(rocksdb::ColumnFamilyData*, rocksdb::DBPropertyInfo const&, bool, unsigned long*)+0x35) [0x558b9db2fe15]", "(RocksDBStore::get_property(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long*)+0x3b) [0x558b9da0812b]", "(BlueStore::BlueStoreThrottle::emit_initial_tracepoint(KeyValueDB&, BlueStore::TransContext&, std::chrono::time_point<ceph::mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x11b) [0x558b9d62992b]", "(BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xa82) [0x558b9d60f862]", "(ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0xb37) [0x558b9d4bf847]", "(ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x1c7) [0x558b9d4b9617]", "(PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x56) [0x558b9d322596]", "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x79d) [0x558b9d29013d]", "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x191) [0x558b9d1ed3b1]", "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x6d) [0x558b9d3c8edd]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf7) [0x558b9d2035e7]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2a3) [0x558b9d71d7d3]", "/usr/bin/ceph-osd(+0xa7ed74) [0x558b9d71dd74]", "/lib64/libc.so.6(+0x9f802) [0x7f47f3349802]", "/lib64/libc.so.6(+0x3f450) [0x7f47f32e9450]" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T00:18:55.467396Z_d229fbe5-83df-476b-9207-c2e0e27539a8", "entity_name": "osd.11", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-osd", "stack_sig": "e639c225ff1709ed8ca59e3a0888ac1a956809690995d0688383763f2e95b1bc", "timestamp": "2023-07-25T00:18:55.467396Z", "utsname_hostname": "ceph-mani-fbdbej-node1-installer", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } ------------------- # ceph crash info 2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f { "backtrace": [ " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 338, in serve\n if self.db_ready() and self.enable_monitoring:", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1219, in db_ready\n return self.db is not None", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1231, in db\n self._db = self.open_db()", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1212, in open_db\n self.configure_db(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1202, in configure_db\n self.load_schema(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1188, in load_schema\n self.create_skeleton_schema(db)", " File \"/usr/share/ceph/mgr/mgr_module.py\", line 1129, in create_skeleton_schema\n db.executescript(SQL)", "sqlite3.InternalError: unknown operation" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T05:12:16.182527Z_da4c4e13-a557-492f-b31e-04f536a2401f", "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg", "mgr_module": "devicehealth", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "InternalError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-mgr", "stack_sig": "b172482b84937fd5ed6878988d42619765d745a0dd0c326018e99429e3abfbfd", "timestamp": "2023-07-25T05:12:16.182527Z", "utsname_hostname": "ceph-mani-fbdbej-node3", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } --------------- # ceph crash info 2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6 { "backtrace": [ " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 623, in serve\n serve.serve()", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 72, in serve\n self._refresh_hosts_and_daemons()", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 274, in _refresh_hosts_and_daemons\n refresh(self.mgr.cache.get_hosts())", " File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 85, in forall_hosts_wrapper\n return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 364, in map\n return self._map_async(func, iterable, mapstar, chunksize).get()", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 771, in get\n raise self._value", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 125, in worker\n result = (True, func(*args, **kwds))", " File \"/lib64/python3.9/multiprocessing/pool.py\", line 48, in mapstar\n return list(map(*args))", " File \"/usr/share/ceph/mgr/cephadm/utils.py\", line 79, in do_work\n return f(*arg)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 223, in refresh\n r = self._refresh_host_daemons(host)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 321, in _refresh_host_daemons\n ls = self.mgr.wait_async(self._run_cephadm_json(", " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 635, in wait_async\n return self.event_loop.get_result(coro, timeout)", " File \"/usr/share/ceph/mgr/cephadm/ssh.py\", line 63, in get_result\n return future.result(timeout)", " File \"/lib64/python3.9/concurrent/futures/_base.py\", line 448, in result\n raise TimeoutError()", "concurrent.futures._base.TimeoutError" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T19:05:45.684434Z_69a93e17-1607-400d-9b24-72c05a6b7cf6", "entity_name": "mgr.ceph-mani-fbdbej-node3.giemdg", "mgr_module": "cephadm", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "TimeoutError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-mgr", "stack_sig": "d703d70abf65004104ed51a965966558d39bdd6eba65a48554e6dbd25c7c54fc", "timestamp": "2023-07-25T19:05:45.684434Z", "utsname_hostname": "ceph-mani-fbdbej-node3", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } ---------- # ceph crash info 2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d { "backtrace": [ "/lib64/libc.so.6(+0x54df0) [0x7f02e6ee1df0]", "/lib64/libc.so.6(+0x9c560) [0x7f02e6f29560]", "pthread_mutex_lock()", "(PG::lock(bool) const+0x2b) [0x56389add067b]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x45d) [0x56389ada7e4d]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2a3) [0x56389b2c27d3]", "/usr/bin/ceph-osd(+0xa7ed74) [0x56389b2c2d74]", "/lib64/libc.so.6(+0x9f802) [0x7f02e6f2c802]", "/lib64/libc.so.6(+0x3f450) [0x7f02e6ecc450]" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T00:18:55.479102Z_aee09b8f-de50-43b1-941a-f4a89d1d332d", "entity_name": "osd.8", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-osd", "stack_sig": "1b35d485fce29e5a7b370206b12bed88e2a2243e53252b49251772880c953e2e", "timestamp": "2023-07-25T00:18:55.479102Z", "utsname_hostname": "ceph-mani-fbdbej-node1-installer", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" ------------- # ceph crash info 2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306 { "backtrace": [ "/lib64/libc.so.6(+0x54df0) [0x7f7e05a88df0]", "/lib64/libc.so.6(+0xa154c) [0x7f7e05ad554c]", "(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x27b) [0x55649bfbfe8b]", "(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >)+0x72) [0x55649bfbff82]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0x55649bfd37c0]", "/usr/bin/ceph-osd(+0xa7ed74) [0x55649bfd3d74]", "/lib64/libc.so.6(+0x9f802) [0x7f7e05ad3802]", "/lib64/libc.so.6(+0x3f450) [0x7f7e05a73450]" ], "ceph_version": "17.2.6-76.el9cp", "crash_id": "2023-07-25T00:18:55.478671Z_024e6421-816b-4aec-8893-da8e79ec5306", "entity_name": "osd.5", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.2 (Plow)", "os_version_id": "9.2", "process_name": "ceph-osd", "stack_sig": "888002e9e1b0600789f244e0281cf9b8de10b8587b13301b5b41854cf1891534", "timestamp": "2023-07-25T00:18:55.478671Z", "utsname_hostname": "ceph-mani-fbdbej-node1-installer", "utsname_machine": "x86_64", "utsname_release": "5.14.0-284.18.1.el9_2.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed May 31 10:39:18 EDT 2023" } -------