Description of problem: Encountered while performing a fill workload with warp client(https://github.com/shreyanshjain7174/warp/releases/download/v0.7.4-put/warp-0.7.4.WorkloadDFG-0.x86_64.rpm) on 6.1 BSV3 private build on baremetal bruuni cluster Cluster is configured to run on a 10000Mb/s speed NIC, with 3 SSD OSDs on each host Note: Fill was successfully completed and issue was observed only during closure. STDOUT of the fill test - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump/fill_stdout17July.txt Cluster was not purged and redeployed before each fill round as the testing was first being carried out to evaluate the behavior of new warp binary # ceph orch host ls HOST ADDR LABELS STATUS bruuni002 172.20.20.12 osdrgw _admin bruuni003 172.20.20.13 monmgrosd _admin bruuni004 172.20.20.14 monmgrosd _admin bruuni006 172.20.20.16 osdrgw _admin bruuni007 172.20.20.17 osdrgw _admin bruuni011 172.20.20.21 osd _admin bruuni012 172.20.20.22 _admin monmgrosd 7 hosts in cluster # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 ssd 0.87329 1.00000 894 GiB 8.7 GiB 6.0 GiB 120 MiB 2.5 GiB 886 GiB 0.97 0.97 259 up 8 ssd 0.87329 1.00000 894 GiB 9.1 GiB 6.0 GiB 61 MiB 3.0 GiB 885 GiB 1.02 1.02 262 up 14 ssd 0.87329 1.00000 894 GiB 8.6 GiB 5.9 GiB 293 MiB 2.4 GiB 886 GiB 0.97 0.96 259 up 1 ssd 0.87329 1.00000 894 GiB 9.0 GiB 6.0 GiB 295 MiB 2.6 GiB 885 GiB 1.00 1.00 262 up 7 ssd 0.87329 1.00000 894 GiB 8.6 GiB 6.0 GiB 161 MiB 2.4 GiB 886 GiB 0.96 0.96 260 up 17 ssd 0.87329 1.00000 894 GiB 9.7 GiB 6.0 GiB 301 MiB 3.3 GiB 885 GiB 1.08 1.08 253 up 4 ssd 0.87329 1.00000 894 GiB 9.2 GiB 6.1 GiB 126 MiB 3.0 GiB 885 GiB 1.02 1.02 260 up 10 ssd 0.87329 1.00000 894 GiB 9.4 GiB 6.1 GiB 265 MiB 3.1 GiB 885 GiB 1.05 1.05 248 up 16 ssd 0.87329 1.00000 894 GiB 8.9 GiB 6.0 GiB 258 MiB 2.6 GiB 885 GiB 0.99 0.99 242 up 0 ssd 0.87329 1.00000 894 GiB 8.7 GiB 6.0 GiB 207 MiB 2.4 GiB 886 GiB 0.97 0.97 249 up 9 ssd 0.87329 1.00000 894 GiB 8.5 GiB 6.0 GiB 153 MiB 2.3 GiB 886 GiB 0.95 0.95 258 up 15 ssd 0.87329 1.00000 894 GiB 8.5 GiB 5.9 GiB 245 MiB 2.4 GiB 886 GiB 0.96 0.95 256 up 6 ssd 0.87329 1.00000 894 GiB 8.6 GiB 5.9 GiB 147 MiB 2.6 GiB 886 GiB 0.97 0.96 257 up 13 ssd 0.87329 1.00000 894 GiB 8.5 GiB 6.0 GiB 123 MiB 2.4 GiB 886 GiB 0.95 0.95 258 up 19 ssd 0.87329 1.00000 894 GiB 9.1 GiB 6.0 GiB 74 MiB 3.0 GiB 885 GiB 1.01 1.01 256 up 5 ssd 0.87329 1.00000 894 GiB 9.2 GiB 6.0 GiB 86 MiB 3.1 GiB 885 GiB 1.02 1.02 258 up 11 ssd 0.87329 1.00000 894 GiB 9.7 GiB 6.1 GiB 141 MiB 3.4 GiB 885 GiB 1.08 1.08 265 up 18 ssd 0.87329 1.00000 894 GiB 9.0 GiB 6.1 GiB 339 MiB 2.6 GiB 885 GiB 1.01 1.01 253 up 3 ssd 0.87329 1.00000 894 GiB 9.5 GiB 6.1 GiB 243 MiB 3.1 GiB 885 GiB 1.06 1.06 255 up 12 ssd 0.87329 1.00000 894 GiB 9.2 GiB 6.1 GiB 228 MiB 2.8 GiB 885 GiB 1.03 1.03 260 up 20 ssd 0.87329 1.00000 894 GiB 8.4 GiB 6.0 GiB 108 MiB 2.3 GiB 886 GiB 0.94 0.94 249 up TOTAL 18 TiB 188 GiB 127 GiB 3.9 GiB 57 GiB 18 TiB 1.00 MIN/MAX VAR: 0.94/1.08 STDDEV: 0.04 # ceph status cluster: id: 6bdae73a-1f24-11ee-9652-0cc47af96454 health: HEALTH_WARN 1 daemons have recently crashed services: mon: 3 daemons, quorum bruuni012,bruuni004,bruuni003 (age 47h) mgr: bruuni012.kutabx(active, since 47h), standbys: bruuni004.xdldff, bruuni003.tnuzhr osd: 21 osds: 21 up (since 47h), 21 in (since 5d) rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 7 pools, 1281 pgs objects: 1.20M objects, 76 GiB usage: 188 GiB used, 18 TiB / 18 TiB avail pgs: 1281 active+clean # coredumpctl list TIME PID UID GID SIG COREFILE EXE SIZE Mon 2023-06-26 05:58:49 UTC 1934897 167 167 SIGABRT none /usr/bin/ceph-osd - Tue 2023-06-27 05:43:25 UTC 43648 0 0 SIGABRT none /usr/bin/ceph-exporter - Sat 2023-07-08 13:25:09 UTC 3744643 167 167 SIGSEGV none ceph-osd - Sat 2023-07-08 19:26:14 UTC 1151010 982 982 SIGABRT none /usr/sbin/haproxy - Thu 2023-07-13 14:01:20 UTC 90792 982 982 SIGABRT none /usr/sbin/haproxy - Thu 2023-07-13 15:25:59 UTC 175526 982 982 SIGABRT none /usr/sbin/haproxy - Thu 2023-07-13 16:28:33 UTC 212733 982 982 SIGABRT none /usr/sbin/haproxy - Mon 2023-07-17 12:51:19 UTC 1754618 0 0 SIGSEGV present /usr/bin/radosgw-admin 1.8M Mon 2023-07-17 13:02:51 UTC 1734017 982 982 SIGABRT none /usr/sbin/haproxy - Version-Release number of selected component (if applicable): # ceph versions { "mon": { "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3 }, "osd": { "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 21 }, "mds": {}, "rgw": { "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3 }, "overall": { "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 30 } } How reproducible: 1/1 Steps to Reproduce: 1. Configure a RHCS 6.1 cluster on a baremetal setup 2. Install and configure warp, haproxy and all the pre-requisites to conduct a fill test with warp over RGW 3. Initiate multiple rounds of fill test without redeploying the cluster, remove the pools and restart the RGW and OSD daemons before every round. Actual results: Expected results: Additional info: Build information - Image: http://registry-proxy.engineering.redhat.com/rh-osbs/rhceph:6-164.0.TEST.bz2119217 Repo: http://download.eng.bos.redhat.com/rcm-guest/ceph-drops/testfixes/bz2119217-0-el9/bz2119217-0-el9.repo Stack trace of core dump - PID: 1754618 (radosgw-admin) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Mon 2023-07-17 12:51:18 UTC (2h 4min ago) Command Line: radosgw-admin bucket stats Executable: /usr/bin/radosgw-admin Control Group: /machine.slice/libpod-ec2ae13bec335825115671c1c1f61f61da5c4de4354ac64fcef6e04f585c7e15.scope/container Unit: libpod-ec2ae13bec335825115671c1c1f61f61da5c4de4354ac64fcef6e04f585c7e15.scope Slice: machine.slice Boot ID: 20fba749b4c54d64a29e5e722e68b161 Machine ID: 526bde07eb67437695e5cd6071793e48 Hostname: bruuni011 Storage: /var/lib/systemd/coredump/core.radosgw-admin.0.20fba749b4c54d64a29e5e722e68b161.1754618.1689598278000000.zst (present) Size on Disk: 1.8M Message: Process 1754618 (radosgw-admin) of user 0 dumped core. Stack trace of thread 7: #0 0x0000150ab113eeb6 pthread_getname_np@@GLIBC_2.34 (libc.so.6 + 0xa0eb6) #1 0x0000150ab2493735 _ZN4ceph7logging3Log11dump_recentEv (libceph-common.so.2 + 0x42b735) #2 0x000055803b1f0074 _ZL27handle_oneshot_fatal_signali (radosgw-admin + 0x441074) #3 0x0000150ab10f2df0 __restore_rt (libc.so.6 + 0x54df0) #4 0x0000150ab113f54c __pthread_kill_implementation (libc.so.6 + 0xa154c) #5 0x0000150ab10f2d46 raise (libc.so.6 + 0x54d46) #6 0x0000150ab10c67f3 abort (libc.so.6 + 0x287f3) #7 0x0000150ab10c7130 __libc_message.cold (libc.so.6 + 0x29130) #8 0x0000150ab1149617 malloc_printerr (libc.so.6 + 0xab617) #9 0x0000150ab114b37a _int_free (libc.so.6 + 0xad37a) #10 0x0000150ab114d955 free (libc.so.6 + 0xaf955) #11 0x0000150ab2290282 _ZNK4ceph6common16RefCountedObject3putEv (libceph-common.so.2 + 0x228282) #12 0x0000150ab2b154da _ZN8Objecter19handle_osd_op_replyEP11MOSDOpReply (librados.so.2 + 0xe94da) #13 0x0000150ab2b0a2ab _ZN8Objecter11ms_dispatchEP7Message (librados.so.2 + 0xde2ab) #14 0x0000150ab2ad755c _ZN10Dispatcher17ms_fast_dispatch2ERKN5boost13intrusive_ptrI7MessageEE (librados.so.2 + 0xab55c) #15 0x0000150ab2396b08 _ZN13DispatchQueue13fast_dispatchERKN5boost13intrusive_ptrI7MessageEE (libceph-common.so.2 + 0x32eb08) #16 0x0000150ab2421a75 _ZN13DispatchQueue13fast_dispatchEP7Message (libceph-common.so.2 + 0x3b9a75) #17 0x0000150ab2450c40 _ZN10ProtocolV214handle_messageEv (libceph-common.so.2 + 0x3e8c40) #18 0x0000150ab2445619 _ZN10ProtocolV216run_continuationER2CtIS_E (libceph-common.so.2 + 0x3dd619) #19 0x0000150ab2425d4b _ZN15AsyncConnection7processEv (libceph-common.so.2 + 0x3bdd4b) #20 0x0000150ab246d231 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405231) #21 0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6) #22 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #23 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #24 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 12: #0 0x0000150ab113a319 __futex_abstimed_wait_common (libc.so.6 + 0x9c319) #1 0x0000150ab1143d8f pthread_rwlock_wrlock@@GLIBC_2.34 (libc.so.6 + 0xa5d8f) #2 0x0000150ab2aff734 _ZN8Objecter14_linger_submitEPNS_8LingerOpERN4ceph13shunique_lockISt12shared_mutexEE (librados.so.2 + 0xd3734) #3 0x0000150ab2b6092d _ZN8Objecter12linger_watchEPNS_8LingerOpER15ObjectOperationRK11SnapContextNSt6chrono10time_pointIN4ceph10real_clockENS7_8durationImSt5ratioILl1ELl1000000000EEEEEERNS9_6buffer7v15_2_04listEOSt10unique_ptrINS9_5async10CompletionIFvN5boost6system10error_codeESI_EvEESt14default_deleteISR_EEPm.isra.0 (librados.so.2 + 0x13492d) #4 0x0000150ab2ad34f2 _ZN8librados9IoCtxImpl5watchERK8object_tPmPNS_7v14_2_08WatchCtxEPNS5_9WatchCtx2Ejb (librados.so.2 + 0xa74f2) #5 0x0000150ab2aac08a _ZN8librados7v14_2_05IoCtx6watch2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPmPNS0_9WatchCtx2E (librados.so.2 + 0x8008a) #6 0x000055803b6c22a5 _ZN10RGWWatcher13C_ReinitWatch6finishEi (radosgw-admin + 0x9132a5) #7 0x000055803b1bba5d _ZN7Context8completeEi (radosgw-admin + 0x40ca5d) #8 0x0000150ab2258325 _ZN8Finisher21finisher_thread_entryEv (libceph-common.so.2 + 0x1f0325) #9 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #10 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 17: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe) #3 0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 19: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab23972d2 _ZN13DispatchQueue5entryEv (libceph-common.so.2 + 0x32f2d2) #4 0x0000150ab2422c61 _ZN13DispatchQueue14DispatchThread5entryEv (libceph-common.so.2 + 0x3bac61) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 20: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab2396d49 _ZN13DispatchQueue18run_local_deliveryEv (libceph-common.so.2 + 0x32ed49) #4 0x0000150ab2422c81 _ZN13DispatchQueue19LocalDeliveryThread5entryEv (libceph-common.so.2 + 0x3bac81) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 5: #0 0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe) #1 0x0000150ab246eb18 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (libceph-common.so.2 + 0x406b18) #2 0x0000150ab246d176 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405176) #3 0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 59: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe) #3 0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 60: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe) #3 0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 3: #0 0x0000150ab113a560 __GI___lll_lock_wait (libc.so.6 + 0x9c560) #1 0x0000150ab1140c22 __pthread_mutex_lock.5 (libc.so.6 + 0xa2c22) #2 0x0000150ab24905fd _ZN4ceph7logging3Log5flushEv (libceph-common.so.2 + 0x4285fd) #3 0x0000150ab24907d9 _ZN4ceph7logging3Log5entryEv (libceph-common.so.2 + 0x4287d9) #4 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #5 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 6: #0 0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe) #1 0x0000150ab246eb18 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (libceph-common.so.2 + 0x406b18) #2 0x0000150ab246d176 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405176) #3 0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 21: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172) #2 0x0000150ab229444b _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c44b) #3 0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1) #4 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #5 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 16: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe) #3 0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 11: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172) #2 0x0000150ab22ad480 _ZN4ceph6common24CephContextServiceThread5entryEv (libceph-common.so.2 + 0x245480) #3 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #4 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 2: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab2ad133f _ZN8librados9IoCtxImpl7unwatchEm (librados.so.2 + 0xa533f) #4 0x000055803b6bff9d _ZN12RGWSI_Notify7unwatchERN11RGWSI_RADOS3ObjEm (radosgw-admin + 0x910f9d) #5 0x000055803b6c2700 _ZN12RGWSI_Notify14finalize_watchEv (radosgw-admin + 0x913700) #6 0x000055803b6c277e _ZN12RGWSI_Notify8shutdownEv.part.0 (radosgw-admin + 0x91377e) #7 0x000055803b21cf6c _ZN15RGWServices_Def8shutdownEv.part.0 (radosgw-admin + 0x46df6c) #8 0x000055803b4995a6 _ZN8RGWRados8finalizeEv (radosgw-admin + 0x6ea5a6) #9 0x000055803b5a0a16 _ZN12StoreManager13close_storageEPN3rgw3sal5StoreE (radosgw-admin + 0x7f1a16) #10 0x000055803b0f4c0d main (radosgw-admin + 0x345c0d) #11 0x0000150ab10ddeb0 __libc_start_call_main (libc.so.6 + 0x3feb0) #12 0x0000150ab10ddf60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60) #13 0x000055803b14e1d5 _start (radosgw-admin + 0x39f1d5) Stack trace of thread 18: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172) #2 0x0000150ab2b22f3b _ZN4ceph5timerINS_17coarse_mono_clockEE12timer_threadEv (librados.so.2 + 0xf6f3b) #3 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #4 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #5 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 62: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab23972d2 _ZN13DispatchQueue5entryEv (libceph-common.so.2 + 0x32f2d2) #4 0x0000150ab2422c61 _ZN13DispatchQueue14DispatchThread5entryEv (libceph-common.so.2 + 0x3bac61) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 22: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab2294170 _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c170) #4 0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 64: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172) #2 0x0000150ab229444b _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c44b) #3 0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1) #4 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #5 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 55: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172) #2 0x000055803b66874f _ZN17RGWDataChangesLog9renew_runEv (radosgw-admin + 0x8b974f) #3 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #4 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #5 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 65: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x0000150ab2294170 _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c170) #4 0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 70: #0 0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a) #1 0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0) #2 0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0) #3 0x000055803b49925d _ZN25RGWIndexCompletionManager7processEv (radosgw-admin + 0x6ea25d) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 71: #0 0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe) #1 0x000055803b19ec09 _ZN5boost4asio6detail13epoll_reactor3runElRNS1_8op_queueINS1_19scheduler_operationEEE (radosgw-admin + 0x3efc09) #2 0x000055803bb26f4a _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (radosgw-admin + 0xd77f4a) #3 0x000055803b3d51dd _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN3rgw6notify7ManagerC4EPN4ceph6common11CephContextEjjjjjjjjPNS3_3sal10RadosStoreEEUlvE0_EEEEE6_M_runEv (radosgw-admin + 0x6261dd) #4 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4) #5 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802) #6 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 72: #0 0x0000150ab11e096f __poll (libc.so.6 + 0x14296f) #1 0x0000150ab2c1492a n/a (/usr/lib64/libcurl.so.4.7.0 + 0x3d92a) ELF object binary architecture: AMD x86-64 Core dump file - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump/core.radosgw-admin.0.20fba749b4c54d64a29e5e722e68b161.1754618.1689598278000000.zst Consolidated location of all relevant files - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump
Thanks Adam for providing the fix. QE estimate for verification is 2 days with the same machine setup used previously.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:7780
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days