Bug 2224078

Summary: radosgw-admin generated core-dump after fill workload using Warp
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Harsh Kumar <hakumar>
Component: RGWAssignee: Adam C. Emerson <aemerson>
Status: CLOSED ERRATA QA Contact: Harsh Kumar <hakumar>
Severity: medium Docs Contact: Rivka Pollack <rpollack>
Priority: unspecified    
Version: 6.1CC: aemerson, akraj, ceph-eng-bugs, cephqe-warriors, kdreyer, mbenjamin, mkogan, tserlin, vumrao
Target Milestone: ---Keywords: Reopened
Target Release: 7.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ceph-18.2.0-77.el9cp Doc Type: Bug Fix
Doc Text:
.Initialization now only unregisters watches that were previously registered Previously, in some cases, an error in initialization could cause an attempt to unregister a watch that was never registered. This would result in some command line tools crashing unpredictably. With this fix, only previously registered watches are unregistered.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-13 15:20:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2237662    

Description Harsh Kumar 2023-07-19 18:07:57 UTC 
Description of problem:
Encountered while performing a fill workload with warp client(https://github.com/shreyanshjain7174/warp/releases/download/v0.7.4-put/warp-0.7.4.WorkloadDFG-0.x86_64.rpm) on 6.1 BSV3 private build on baremetal bruuni cluster

Cluster is configured to run on a 10000Mb/s speed NIC, with 3 SSD OSDs on each host
Note: Fill was successfully completed and issue was observed only during closure.
STDOUT of the fill test - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump/fill_stdout17July.txt
Cluster was not purged and redeployed before each fill round as the testing was first being carried out to evaluate the behavior of new warp binary

# ceph orch host ls
HOST       ADDR          LABELS            STATUS  
bruuni002  172.20.20.12  osdrgw _admin             
bruuni003  172.20.20.13  monmgrosd _admin          
bruuni004  172.20.20.14  monmgrosd _admin          
bruuni006  172.20.20.16  osdrgw _admin             
bruuni007  172.20.20.17  osdrgw _admin             
bruuni011  172.20.20.21  osd _admin                
bruuni012  172.20.20.22  _admin monmgrosd          
7 hosts in cluster

# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE  VAR   PGS  STATUS
 2    ssd  0.87329   1.00000  894 GiB  8.7 GiB  6.0 GiB  120 MiB  2.5 GiB  886 GiB  0.97  0.97  259      up
 8    ssd  0.87329   1.00000  894 GiB  9.1 GiB  6.0 GiB   61 MiB  3.0 GiB  885 GiB  1.02  1.02  262      up
14    ssd  0.87329   1.00000  894 GiB  8.6 GiB  5.9 GiB  293 MiB  2.4 GiB  886 GiB  0.97  0.96  259      up
 1    ssd  0.87329   1.00000  894 GiB  9.0 GiB  6.0 GiB  295 MiB  2.6 GiB  885 GiB  1.00  1.00  262      up
 7    ssd  0.87329   1.00000  894 GiB  8.6 GiB  6.0 GiB  161 MiB  2.4 GiB  886 GiB  0.96  0.96  260      up
17    ssd  0.87329   1.00000  894 GiB  9.7 GiB  6.0 GiB  301 MiB  3.3 GiB  885 GiB  1.08  1.08  253      up
 4    ssd  0.87329   1.00000  894 GiB  9.2 GiB  6.1 GiB  126 MiB  3.0 GiB  885 GiB  1.02  1.02  260      up
10    ssd  0.87329   1.00000  894 GiB  9.4 GiB  6.1 GiB  265 MiB  3.1 GiB  885 GiB  1.05  1.05  248      up
16    ssd  0.87329   1.00000  894 GiB  8.9 GiB  6.0 GiB  258 MiB  2.6 GiB  885 GiB  0.99  0.99  242      up
 0    ssd  0.87329   1.00000  894 GiB  8.7 GiB  6.0 GiB  207 MiB  2.4 GiB  886 GiB  0.97  0.97  249      up
 9    ssd  0.87329   1.00000  894 GiB  8.5 GiB  6.0 GiB  153 MiB  2.3 GiB  886 GiB  0.95  0.95  258      up
15    ssd  0.87329   1.00000  894 GiB  8.5 GiB  5.9 GiB  245 MiB  2.4 GiB  886 GiB  0.96  0.95  256      up
 6    ssd  0.87329   1.00000  894 GiB  8.6 GiB  5.9 GiB  147 MiB  2.6 GiB  886 GiB  0.97  0.96  257      up
13    ssd  0.87329   1.00000  894 GiB  8.5 GiB  6.0 GiB  123 MiB  2.4 GiB  886 GiB  0.95  0.95  258      up
19    ssd  0.87329   1.00000  894 GiB  9.1 GiB  6.0 GiB   74 MiB  3.0 GiB  885 GiB  1.01  1.01  256      up
 5    ssd  0.87329   1.00000  894 GiB  9.2 GiB  6.0 GiB   86 MiB  3.1 GiB  885 GiB  1.02  1.02  258      up
11    ssd  0.87329   1.00000  894 GiB  9.7 GiB  6.1 GiB  141 MiB  3.4 GiB  885 GiB  1.08  1.08  265      up
18    ssd  0.87329   1.00000  894 GiB  9.0 GiB  6.1 GiB  339 MiB  2.6 GiB  885 GiB  1.01  1.01  253      up
 3    ssd  0.87329   1.00000  894 GiB  9.5 GiB  6.1 GiB  243 MiB  3.1 GiB  885 GiB  1.06  1.06  255      up
12    ssd  0.87329   1.00000  894 GiB  9.2 GiB  6.1 GiB  228 MiB  2.8 GiB  885 GiB  1.03  1.03  260      up
20    ssd  0.87329   1.00000  894 GiB  8.4 GiB  6.0 GiB  108 MiB  2.3 GiB  886 GiB  0.94  0.94  249      up
                       TOTAL   18 TiB  188 GiB  127 GiB  3.9 GiB   57 GiB   18 TiB  1.00                   
MIN/MAX VAR: 0.94/1.08  STDDEV: 0.04

# ceph status
  cluster:
    id:     6bdae73a-1f24-11ee-9652-0cc47af96454
    health: HEALTH_WARN
            1 daemons have recently crashed
 
  services:
    mon: 3 daemons, quorum bruuni012,bruuni004,bruuni003 (age 47h)
    mgr: bruuni012.kutabx(active, since 47h), standbys: bruuni004.xdldff, bruuni003.tnuzhr
    osd: 21 osds: 21 up (since 47h), 21 in (since 5d)
    rgw: 3 daemons active (3 hosts, 1 zones)
 
  data:
    pools:   7 pools, 1281 pgs
    objects: 1.20M objects, 76 GiB
    usage:   188 GiB used, 18 TiB / 18 TiB avail
    pgs:     1281 active+clean

# coredumpctl list
TIME                            PID UID GID SIG     COREFILE EXE                    SIZE
Mon 2023-06-26 05:58:49 UTC 1934897 167 167 SIGABRT none     /usr/bin/ceph-osd         -
Tue 2023-06-27 05:43:25 UTC   43648   0   0 SIGABRT none     /usr/bin/ceph-exporter    -
Sat 2023-07-08 13:25:09 UTC 3744643 167 167 SIGSEGV none     ceph-osd                  -
Sat 2023-07-08 19:26:14 UTC 1151010 982 982 SIGABRT none     /usr/sbin/haproxy         -
Thu 2023-07-13 14:01:20 UTC   90792 982 982 SIGABRT none     /usr/sbin/haproxy         -
Thu 2023-07-13 15:25:59 UTC  175526 982 982 SIGABRT none     /usr/sbin/haproxy         -
Thu 2023-07-13 16:28:33 UTC  212733 982 982 SIGABRT none     /usr/sbin/haproxy         -
Mon 2023-07-17 12:51:19 UTC 1754618   0   0 SIGSEGV present  /usr/bin/radosgw-admin 1.8M
Mon 2023-07-17 13:02:51 UTC 1734017 982 982 SIGABRT none     /usr/sbin/haproxy         -



Version-Release number of selected component (if applicable):
# ceph versions
{
    "mon": {
        "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3
    },
    "mgr": {
        "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3
    },
    "osd": {
        "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 21
    },
    "mds": {},
    "rgw": {
        "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 3
    },
    "overall": {
        "ceph version 17.2.6-58.0.TEST.bz2119217.el9cp (7da3e6ae59de2dacd4d7dc88c7421d9016259fea) quincy (stable)": 30
    }
}

How reproducible:
1/1

Steps to Reproduce:
1. Configure a RHCS 6.1 cluster on a baremetal setup
2. Install and configure warp, haproxy and all the pre-requisites to conduct a fill test with warp over RGW
3. Initiate multiple rounds of fill test without redeploying the cluster, remove the pools and restart the RGW and OSD daemons before every round.

Actual results:


Expected results:


Additional info:
Build information -
Image: http://registry-proxy.engineering.redhat.com/rh-osbs/rhceph:6-164.0.TEST.bz2119217
Repo: http://download.eng.bos.redhat.com/rcm-guest/ceph-drops/testfixes/bz2119217-0-el9/bz2119217-0-el9.repo

Stack trace of core dump - 
           PID: 1754618 (radosgw-admin)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2023-07-17 12:51:18 UTC (2h 4min ago)
  Command Line: radosgw-admin bucket stats
    Executable: /usr/bin/radosgw-admin
 Control Group: /machine.slice/libpod-ec2ae13bec335825115671c1c1f61f61da5c4de4354ac64fcef6e04f585c7e15.scope/container
          Unit: libpod-ec2ae13bec335825115671c1c1f61f61da5c4de4354ac64fcef6e04f585c7e15.scope
         Slice: machine.slice
       Boot ID: 20fba749b4c54d64a29e5e722e68b161
    Machine ID: 526bde07eb67437695e5cd6071793e48
      Hostname: bruuni011
       Storage: /var/lib/systemd/coredump/core.radosgw-admin.0.20fba749b4c54d64a29e5e722e68b161.1754618.1689598278000000.zst (present)
  Size on Disk: 1.8M
       Message: Process 1754618 (radosgw-admin) of user 0 dumped core.
                
                Stack trace of thread 7:
                #0  0x0000150ab113eeb6 pthread_getname_np@@GLIBC_2.34 (libc.so.6 + 0xa0eb6)
                #1  0x0000150ab2493735 _ZN4ceph7logging3Log11dump_recentEv (libceph-common.so.2 + 0x42b735)
                #2  0x000055803b1f0074 _ZL27handle_oneshot_fatal_signali (radosgw-admin + 0x441074)
                #3  0x0000150ab10f2df0 __restore_rt (libc.so.6 + 0x54df0)
                #4  0x0000150ab113f54c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #5  0x0000150ab10f2d46 raise (libc.so.6 + 0x54d46)
                #6  0x0000150ab10c67f3 abort (libc.so.6 + 0x287f3)
                #7  0x0000150ab10c7130 __libc_message.cold (libc.so.6 + 0x29130)
                #8  0x0000150ab1149617 malloc_printerr (libc.so.6 + 0xab617)
                #9  0x0000150ab114b37a _int_free (libc.so.6 + 0xad37a)
                #10 0x0000150ab114d955 free (libc.so.6 + 0xaf955)
                #11 0x0000150ab2290282 _ZNK4ceph6common16RefCountedObject3putEv (libceph-common.so.2 + 0x228282)
                #12 0x0000150ab2b154da _ZN8Objecter19handle_osd_op_replyEP11MOSDOpReply (librados.so.2 + 0xe94da)
                #13 0x0000150ab2b0a2ab _ZN8Objecter11ms_dispatchEP7Message (librados.so.2 + 0xde2ab)
                #14 0x0000150ab2ad755c _ZN10Dispatcher17ms_fast_dispatch2ERKN5boost13intrusive_ptrI7MessageEE (librados.so.2 + 0xab55c)
                #15 0x0000150ab2396b08 _ZN13DispatchQueue13fast_dispatchERKN5boost13intrusive_ptrI7MessageEE (libceph-common.so.2 + 0x32eb08)
                #16 0x0000150ab2421a75 _ZN13DispatchQueue13fast_dispatchEP7Message (libceph-common.so.2 + 0x3b9a75)
                #17 0x0000150ab2450c40 _ZN10ProtocolV214handle_messageEv (libceph-common.so.2 + 0x3e8c40)
                #18 0x0000150ab2445619 _ZN10ProtocolV216run_continuationER2CtIS_E (libceph-common.so.2 + 0x3dd619)
                #19 0x0000150ab2425d4b _ZN15AsyncConnection7processEv (libceph-common.so.2 + 0x3bdd4b)
                #20 0x0000150ab246d231 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405231)
                #21 0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6)
                #22 0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #23 0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #24 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 12:
                #0  0x0000150ab113a319 __futex_abstimed_wait_common (libc.so.6 + 0x9c319)
                #1  0x0000150ab1143d8f pthread_rwlock_wrlock@@GLIBC_2.34 (libc.so.6 + 0xa5d8f)
                #2  0x0000150ab2aff734 _ZN8Objecter14_linger_submitEPNS_8LingerOpERN4ceph13shunique_lockISt12shared_mutexEE (librados.so.2 + 0xd3734)
                #3  0x0000150ab2b6092d _ZN8Objecter12linger_watchEPNS_8LingerOpER15ObjectOperationRK11SnapContextNSt6chrono10time_pointIN4ceph10real_clockENS7_8durationImSt5ratioILl1ELl1000000000EEEEEERNS9_6buffer7v15_2_04listEOSt10unique_ptrINS9_5async10CompletionIFvN5boost6system10error_codeESI_EvEESt14default_deleteISR_EEPm.isra.0 (librados.so.2 + 0x13492d)
                #4  0x0000150ab2ad34f2 _ZN8librados9IoCtxImpl5watchERK8object_tPmPNS_7v14_2_08WatchCtxEPNS5_9WatchCtx2Ejb (librados.so.2 + 0xa74f2)
                #5  0x0000150ab2aac08a _ZN8librados7v14_2_05IoCtx6watch2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPmPNS0_9WatchCtx2E (librados.so.2 + 0x8008a)
                #6  0x000055803b6c22a5 _ZN10RGWWatcher13C_ReinitWatch6finishEi (radosgw-admin + 0x9132a5)
                #7  0x000055803b1bba5d _ZN7Context8completeEi (radosgw-admin + 0x40ca5d)
                #8  0x0000150ab2258325 _ZN8Finisher21finisher_thread_entryEv (libceph-common.so.2 + 0x1f0325)
                #9  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #10 0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 17:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe)
                #3  0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 19:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab23972d2 _ZN13DispatchQueue5entryEv (libceph-common.so.2 + 0x32f2d2)
                #4  0x0000150ab2422c61 _ZN13DispatchQueue14DispatchThread5entryEv (libceph-common.so.2 + 0x3bac61)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 20:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab2396d49 _ZN13DispatchQueue18run_local_deliveryEv (libceph-common.so.2 + 0x32ed49)
                #4  0x0000150ab2422c81 _ZN13DispatchQueue19LocalDeliveryThread5entryEv (libceph-common.so.2 + 0x3bac81)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 5:
                #0  0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe)
                #1  0x0000150ab246eb18 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (libceph-common.so.2 + 0x406b18)
                #2  0x0000150ab246d176 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405176)
                #3  0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 59:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe)
                #3  0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 60:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe)
                #3  0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 3:
                #0  0x0000150ab113a560 __GI___lll_lock_wait (libc.so.6 + 0x9c560)
                #1  0x0000150ab1140c22 __pthread_mutex_lock.5 (libc.so.6 + 0xa2c22)
                #2  0x0000150ab24905fd _ZN4ceph7logging3Log5flushEv (libceph-common.so.2 + 0x4285fd)
                #3  0x0000150ab24907d9 _ZN4ceph7logging3Log5entryEv (libceph-common.so.2 + 0x4287d9)
                #4  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #5  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 6:
                #0  0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe)
                #1  0x0000150ab246eb18 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (libceph-common.so.2 + 0x406b18)
                #2  0x0000150ab246d176 _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (libceph-common.so.2 + 0x405176)
                #3  0x0000150ab246dce6 _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEP6WorkerEUlvE_E9_M_invokeERKSt9_Any_data (libceph-common.so.2 + 0x405ce6)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 21:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172)
                #2  0x0000150ab229444b _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c44b)
                #3  0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1)
                #4  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #5  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 16:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab2b548fe _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (librados.so.2 + 0x1288fe)
                #3  0x0000150ab2aec89a _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv (librados.so.2 + 0xc089a)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 11:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172)
                #2  0x0000150ab22ad480 _ZN4ceph6common24CephContextServiceThread5entryEv (libceph-common.so.2 + 0x245480)
                #3  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #4  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 2:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab2ad133f _ZN8librados9IoCtxImpl7unwatchEm (librados.so.2 + 0xa533f)
                #4  0x000055803b6bff9d _ZN12RGWSI_Notify7unwatchERN11RGWSI_RADOS3ObjEm (radosgw-admin + 0x910f9d)
                #5  0x000055803b6c2700 _ZN12RGWSI_Notify14finalize_watchEv (radosgw-admin + 0x913700)
                #6  0x000055803b6c277e _ZN12RGWSI_Notify8shutdownEv.part.0 (radosgw-admin + 0x91377e)
                #7  0x000055803b21cf6c _ZN15RGWServices_Def8shutdownEv.part.0 (radosgw-admin + 0x46df6c)
                #8  0x000055803b4995a6 _ZN8RGWRados8finalizeEv (radosgw-admin + 0x6ea5a6)
                #9  0x000055803b5a0a16 _ZN12StoreManager13close_storageEPN3rgw3sal5StoreE (radosgw-admin + 0x7f1a16)
                #10 0x000055803b0f4c0d main (radosgw-admin + 0x345c0d)
                #11 0x0000150ab10ddeb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #12 0x0000150ab10ddf60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #13 0x000055803b14e1d5 _start (radosgw-admin + 0x39f1d5)
                
                Stack trace of thread 18:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172)
                #2  0x0000150ab2b22f3b _ZN4ceph5timerINS_17coarse_mono_clockEE12timer_threadEv (librados.so.2 + 0xf6f3b)
                #3  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #4  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #5  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 62:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab23972d2 _ZN13DispatchQueue5entryEv (libceph-common.so.2 + 0x32f2d2)
                #4  0x0000150ab2422c61 _ZN13DispatchQueue14DispatchThread5entryEv (libceph-common.so.2 + 0x3bac61)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 22:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab2294170 _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c170)
                #4  0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 64:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172)
                #2  0x0000150ab229444b _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c44b)
                #3  0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1)
                #4  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #5  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 55:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113d172 pthread_cond_clockwait (libc.so.6 + 0x9f172)
                #2  0x000055803b66874f _ZN17RGWDataChangesLog9renew_runEv (radosgw-admin + 0x8b974f)
                #3  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #4  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #5  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 65:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x0000150ab2294170 _ZN15CommonSafeTimerISt5mutexE12timer_threadEv (libceph-common.so.2 + 0x22c170)
                #4  0x0000150ab22948c1 _ZN21CommonSafeTimerThreadISt5mutexE5entryEv (libceph-common.so.2 + 0x22c8c1)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 70:
                #0  0x0000150ab113a39a __futex_abstimed_wait_common (libc.so.6 + 0x9c39a)
                #1  0x0000150ab113cba0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x9eba0)
                #2  0x0000150ab13975b0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6 + 0xd55b0)
                #3  0x000055803b49925d _ZN25RGWIndexCompletionManager7processEv (radosgw-admin + 0x6ea25d)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 71:
                #0  0x0000150ab11ecabe epoll_wait (libc.so.6 + 0x14eabe)
                #1  0x000055803b19ec09 _ZN5boost4asio6detail13epoll_reactor3runElRNS1_8op_queueINS1_19scheduler_operationEEE (radosgw-admin + 0x3efc09)
                #2  0x000055803bb26f4a _ZN5boost4asio6detail9scheduler3runERNS_6system10error_codeE.constprop.0.isra.0 (radosgw-admin + 0xd77f4a)
                #3  0x000055803b3d51dd _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN3rgw6notify7ManagerC4EPN4ceph6common11CephContextEjjjjjjjjPNS3_3sal10RadosStoreEEUlvE0_EEEEE6_M_runEv (radosgw-admin + 0x6261dd)
                #4  0x0000150ab139d9d4 execute_native_thread_routine (libstdc++.so.6 + 0xdb9d4)
                #5  0x0000150ab113d802 start_thread (libc.so.6 + 0x9f802)
                #6  0x0000150ab10dd450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 72:
                #0  0x0000150ab11e096f __poll (libc.so.6 + 0x14296f)
                #1  0x0000150ab2c1492a n/a (/usr/lib64/libcurl.so.4.7.0 + 0x3d92a)
                ELF object binary architecture: AMD x86-64

Core dump file - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump/core.radosgw-admin.0.20fba749b4c54d64a29e5e722e68b161.1754618.1689598278000000.zst

Consolidated location of all relevant files - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/rgw-core-dump
Comment 21 Harsh Kumar 2023-10-05 05:28:25 UTC 
Thanks Adam for providing the fix.

QE estimate for verification is 2 days with the same machine setup used previously.
Comment 25 errata-xmlrpc 2023-12-13 15:20:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780
Comment 26 Red Hat Bugzilla 2024-06-02 04:25:05 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days