Description of problem: The c++ broker very rarely crash in unknown location during long term failover soak test. There seems to be much more function/methods calls in the Thread 1 call stack than expected, see below. The ?? calls from Thread 1 are neither from openais nor from kernel. (both debuginfo packages installed) The issue has been visible on RHEL 5.3 x86_64. Version-Release number of selected component (if applicable): [root@mrg-qe-02 qpid_ptest_cluster_failover_soak]# rpm -qa | egrep '(qpid|rhm|openais)' | sort -u openais-0.80.3-22.el5_3.8 openais-debuginfo-0.80.3-22.el5_3.8 python-qpid-0.5.752581-3.el5 qpidc-0.5.752581-22.el5 qpidc-debuginfo-0.5.752581-22.el5 qpidc-devel-0.5.752581-22.el5 qpidc-perftest-0.5.752581-22.el5 qpidc-rdma-0.5.752581-22.el5 qpidc-ssl-0.5.752581-22.el5 qpidd-0.5.752581-22.el5 qpidd-acl-0.5.752581-22.el5 qpidd-cluster-0.5.752581-22.el5 qpidd-devel-0.5.752581-22.el5 qpid-dotnet-0.4.738274-2.el5 qpidd-rdma-0.5.752581-22.el5 qpidd-ssl-0.5.752581-22.el5 qpidd-xml-0.5.752581-22.el5 qpid-java-client-0.5.751061-8.el5 qpid-java-common-0.5.751061-8.el5 rhm-0.5.3206-5.el5 rhm-docs-0.5.756148-1.el5 How reproducible: very hard to reproduce (~1%) Steps to Reproduce: 1. run the qpid_ptest_cluster_failover_soak test in the loop export TEST_PARAM_RHTS_TEST_MRG_QPID_PTEST_CLUSTER_FAILOVER_SOAK_LOOPS=222 2. wait for core files & examine them Actual results: Crash one a few hundrets of runs Expected results: No crash Additional info: (backtrace) GNU gdb Fedora (6.8-27.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done. done. ... Reading symbols from /usr/lib64/libaio.so.1...done. Loaded symbols for /usr/lib64/libaio.so.1 Core was generated by `qpidd --cluster-name soakTestCluster_d80a8d16-1503-45c7-a8a0-2742e501defd --aut'. Program terminated with signal 11, Segmentation fault. [New process 14312] [New process 14317] [New process 14316] [New process 14315] [New process 14314] [New process 14313] [New process 14311] [New process 14310] [New process 14309] [New process 14307] [New process 14306] [New process 14305] #0 0x00000036e047b6f0 in memcpy () from /lib64/libc.so.6 (gdb) Thread 12 (process 14305): #0 0x00000036e04c56db in write () from /lib64/libc.so.6 #1 0x00002b2a8f4ad1a0 in mrg::journal::fcntl::clean_file () from /usr/lib64/qpid/daemon/msgstore.so #2 0x00002b2a8f4ae63c in mrg::journal::fcntl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #3 0x00002b2a8f4ae73f in mrg::journal::fcntl::fcntl () from /usr/lib64/qpid/daemon/msgstore.so #4 0x00002b2a8f4b4f37 in mrg::journal::jcntl::new_fcntl () from /usr/lib64/qpid/daemon/msgstore.so #5 0x00002b2a8f4c7f6f in mrg::journal::lpmgr::append () from /usr/lib64/qpid/daemon/msgstore.so #6 0x00002b2a8f4c8b7a in mrg::journal::lpmgr::initialize () from /usr/lib64/qpid/daemon/msgstore.so #7 0x00002b2a8f4b4dd8 in mrg::journal::jcntl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #8 0x00002b2a8f46473c in mrg::msgstore::JournalImpl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #9 0x00002b2a8f4833c0 in mrg::msgstore::MessageStoreImpl::create () from /usr/lib64/qpid/daemon/msgstore.so #10 0x00000030f0548308 in qpid::broker::MessageStoreModule::create ( this=<value optimized out>, queue=<value optimized out>, args=<value optimized out>) at qpid/broker/MessageStoreModule.cpp:46 #11 0x00000030f04e7825 in qpid::broker::Queue::create ( this=<value optimized out>, _settings=<value optimized out>) at qpid/broker/Queue.cpp:757 #12 0x00000030f057c0cf in qpid::broker::SessionAdapter::QueueHandlerImpl::declare (this=<value optimized out>, name=<value optimized out>, alternateExchange=<value optimized out>, passive=<value optimized out>, durable=<value optimized out>, exclusive=<value optimized out>, autoDelete=<value optimized out>, arguments=Could not find the frame base for "qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&)". ) at qpid/broker/SessionAdapter.cpp:359 #13 0x00000030eff2cf06 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at gen/qpid/framing/QueueDeclareBody.h:95 #14 0x00000030eff328d7 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at gen/qpid/framing/ServerInvoker.cpp:368 #15 0x00000030f058b33e in qpid::framing::invoke<qpid::broker::SessionAdapter> ( target=<value optimized out>, body=<value optimized out>) at qpid/framing/Invoker.h:67 #16 0x00000030f0588adc in qpid::broker::SessionState::handleCommand ( this=<value optimized out>, method=<value optimized out>, id=<value optimized out>) at qpid/broker/SessionState.cpp:194 #17 0x00000030f058a1a7 in qpid::broker::SessionState::handleIn ( this=<value optimized out>, frame=<value optimized out>) at qpid/broker/SessionState.cpp:328 #18 0x00000030eff9b53e in qpid::amqp_0_10::SessionHandler::handleIn ( this=<value optimized out>, f=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:91 #19 0x00000030f04faf79 in qpid::broker::Connection::received ( this=<value optimized out>, frame=<value optimized out>) at qpid/framing/Handler.h:42 #20 0x00002b2a8cfbd2b6 in qpid::cluster::Connection::received ( this=0x2aaaac007730, f=@0x2aaaac007810) at qpid/cluster/Connection.cpp:123 #21 0x00002b2a8cfba59d in qpid::cluster::Connection::decode ( this=0x2aaaac007730, buffer=0x2aaaac00b760 "\017", size=18563) at qpid/cluster/Connection.cpp:218 #22 0x00000030effc43c7 in qpid::sys::AsynchIOHandler::readbuff ( this=<value optimized out>, buff=<value optimized out>) at qpid/sys/AsynchIOHandler.cpp:113 #23 0x00000030f05b3098 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke ( function_obj_ptr=<value optimized out>, a0=<value optimized out>, a1=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:252 #24 0x00000030eff707ba in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=<value optimized out>, a0=<value optimized out>, a1=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #25 0x00000030eff6a5b3 in qpid::sys::posix::AsynchIO::readable ( this=<value optimized out>, h=<value optimized out>) at qpid/sys/posix/AsynchIO.cpp:447 #26 0x00000030effc9d07 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #27 0x00000030effc79ec in qpid::sys::DispatchHandle::processEvent ( this=<value optimized out>, type=<value optimized out>) at qpid/sys/DispatchHandle.cpp:438 #28 0x00000030eff7dcb3 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/Poller.h:122 #29 0x00000030f04c941e in qpid::broker::Broker::run ( this=<value optimized out>) at qpid/broker/Broker.cpp:319 #30 0x00000000004069b8 in QpiddBroker::execute (this=<value optimized out>, options=0x189a2d30) at posix/QpiddBroker.cpp:166 #31 0x00000000004054a8 in main (argc=17, argv=0x7fff1e5d5c58) at qpidd.cpp:77 Thread 11 (process 14306): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000030f0591379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 10 (process 14307): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000030f0591379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 9 (process 14309): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000030f0591379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 8 (process 14310): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 7 (process 14311): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 6 (process 14313): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 5 (process 14314): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 4 (process 14315): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 3 (process 14316): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 2 (process 14317): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00000030eff7d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00000030eff7dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00000030eff73cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 1 (process 14312): #0 0x00000036e047b6f0 in memcpy () from /lib64/libc.so.6 #1 0x00002b2a8d219585 in memcpy_swrap (dest=<value optimized out>, src=<value optimized out>, len=<value optimized out>, read=<value optimized out>) at /usr/include/bits/string3.h:51 #2 0x00002b2a8d219dbe in openais_dispatch_recv ( ipc_ctx=<value optimized out>, data=<value optimized out>, timeout=<value optimized out>) at util.c:554 #3 0x00002b2a8d21ab8a in cpg_dispatch (handle=<value optimized out>, dispatch_types=<value optimized out>) at cpg.c:266 #4 0x000c000000050000 in ?? () #5 0x00216e7302010000 in ?? () #6 0x0413000000a4f600 in ?? () #7 0x73656d0d02021001 in ?? () #8 0x6575715f65676173 in ?? () #9 0x0100110003076575 in ?? () #10 0x3336000000000200 in ?? () #11 0x001000000f303431 in ?? () #12 0x0200000000020000 in ?? () #13 0x001500010b00040c in ?? () #14 0x0400000000020001 in ?? () #15 0x0001000007000101 in ?? () #16 0x0200010043000203 in ?? () #17 0x1c00000000000000 in ?? () #18 0x0000000001010304 in ?? () #19 0x0c00000005000000 in ?? () #20 0x216e730201000000 in ?? () #21 0x13000000a5f60000 in ?? () #22 0x656d0d0202100104 in ?? () #23 0x75715f6567617373 in ?? () #24 0x0011000307657565 in ?? () #25 0x3600000000020001 in ?? () #26 0x1a00000f31343133 in ?? () #27 0x0000000002000000 in ?? () #28 0x0000080000010b02 in ?? () #29 0x000f0ca40000c1a3 in ?? () #30 0x0000020000001a00 in ?? () #31 0x080000010b020000 in ?? () #32 0x0ca40000c1a30000 in ?? () #33 0x020000001a00000f in ?? () #34 0x00010b0200000000 in ?? () #35 0x0000c1a300000800 in ?? () #36 0x00001a00000f0ca4 in ?? () #37 0x0b02000000000200 in ?? () #38 0xc1a3000008000001 in ?? () #39 0x1a00000f0ca40000 in ?? () #40 0x0000000002000000 in ?? () #41 0x0000080000010b02 in ?? () #42 0x000f0ca40000c1a3 in ?? () #43 0x0000020000001200 in ?? () #44 0x000000010b020000 in ?? () #45 0x020000001200000f in ?? () #46 0x00010b0200000000 in ?? () #47 0x00001200000f0000 in ?? () #48 0x0b02000000000200 in ?? () #49 0x1200000f00000001 in ?? () #50 0x0000000002000000 in ?? () #51 0x00e9000000010b02 in ?? () #52 0x00050037ae000000 in ?? () #53 0x003721220a310000 in ?? () #54 0x0034000fc0000000 in ?? () #55 0x6f7300007fff0000 in ?? () #56 0x6c43747365546b61 in ?? () #57 0x38645f7265747375 in ?? () #58 0x312d363164386130 in ?? () #59 0x376335342d333035 in ?? () #60 0x37322d306138612d in ?? () #61 0x6564313035653234 in ?? () #62 0x0030f051e99e6466 in ?? () #63 0x00001e3c38900000 in ?? () #64 0x0036eca9c9010000 in ?? () #65 0x00001e3ae6680000 in ?? () #66 0x00001e3c37780000 in ?? () #67 0x00001e3c2d400000 in ?? () #68 0x0036e047402a0000 in ?? () #69 0x00001e3c38900000 in ?? () #70 0x2b7bdcf458010000 in ?? () #71 0x00311e3c15400000 in ?? () #72 0x220a000000020000 in ?? () #73 0x37ae0103041c3721 in ?? () #74 0x0000000000000000 in ?? () (gdb) quit
There is one more coredump detected, which is much more steep (also unidentified by gdb): Doing the analysis of memory addresses in Thread 1 I found that calls from Thread 1 are duplicated, so it might indicate bad core file: [freznice@dhcp-lab-200 ~]$ gawk 'BEGIN{addr="";calls=""}{if ($1 != addr){printf("%s %s\n",addr,calls);addr=$1;calls=$2;}else{calls=sprintf("%s %s",calls, $2)}}' gdb_fail.sort.txt | awk '{if (NF>2){print}}' 0x0000000000000001 #156 #217 #278 #339 #34 #400 #461 #95 0x0000000000000002 #124 #185 #246 #307 #368 #429 #490 #63 0x0000000000010103 #110 #171 #232 #293 #354 #415 #476 #49 0x0000000000020001 #139 #17 #200 #261 #322 #383 #444 #78 0x0000000002000000 #119 #180 #241 #302 #363 #424 #485 #58 0x0000000002000100 #10 #121 #132 #182 #193 #243 #254 #304 #315 #365 #376 #426 #437 #487 #498 #60 #71 0x000000010103041c #125 #186 #247 #308 #369 #430 #491 #64 0x0000000200010043 #154 #215 #276 #32 #337 #398 #459 #93 0x0000000600000000 #126 #187 #248 #309 #370 #4 #431 #492 #65 0x0000000c00000006 #157 #218 #279 #340 #35 #401 #462 #96 0x0000020000001000 #12 #134 #195 #256 #317 #378 #439 #500 #73 0x0000020001001200 #147 #208 #25 #269 #330 #391 #452 #86 0x0000020001001500 #136 #14 #197 #258 #319 #380 #441 #75 0x0001004300020300 #123 #184 #245 #306 #367 #428 #489 #62 0x00010103041c0000 #140 #18 #201 #262 #323 #384 #445 #79 0x0001010400000000 #152 #213 #274 #30 #335 #396 #457 #91 0x0002000100430002 #108 #169 #230 #291 #352 #413 #47 #474 0x0002030001000007 #153 #214 #275 #31 #336 #397 #458 #92 0x00040c0200000000 #150 #211 #272 #28 #333 #394 #455 #89 0x0006000000000000 #141 #19 #202 #263 #324 #385 #446 #80 0x0007000101040000 #137 #15 #198 #259 #320 #381 #442 #76 0x000c000000060000 #111 #172 #233 #294 #355 #416 #477 #50 0x00216e7302010000 #112 #173 #234 #295 #356 #417 #478 #51 0x0043000203000100 #138 #16 #199 #260 #321 #382 #443 #77 0x0100000700010104 #122 #183 #244 #305 #366 #427 #488 #61 0x0100120003076575 #116 #177 #238 #299 #360 #421 #482 #55 0x01001500010b0004 #105 #166 #227 #288 #349 #410 #44 #471 0x0103041c00000000 #155 #216 #277 #33 #338 #399 #460 #94 0x0104000000000200 #106 #167 #228 #289 #350 #411 #45 #472 0x010b00040c020000 #13 #135 #196 #257 #318 #379 #440 #501 #74 0x020000001000000f #149 #210 #27 #271 #332 #393 #454 #88 0x0200010012000307 #101 #162 #223 #284 #345 #40 #406 #467 0x020001001500010b #151 #212 #273 #29 #334 #395 #456 #90 0x02010000000c0000 #142 #20 #203 #264 #325 #386 #447 #81 0x0202100104130000 #144 #205 #22 #266 #327 #388 #449 #83 0x0300010000070001 #107 #168 #229 #290 #351 #412 #46 #473 0x030765756575715f #146 #207 #24 #268 #329 #390 #451 #85 0x041c000000000000 #109 #170 #231 #292 #353 #414 #475 #48 0x0c02000000000200 #104 #165 #226 #287 #348 #409 #43 #470 0x1200030765756575 #131 #192 #253 #314 #375 #436 #497 #70 #9 0x1500010b00040c02 #120 #181 #242 #303 #364 #425 #486 #59 0x3432000000000200 #117 #178 #239 #300 #361 #422 #483 #56 0x3535343200000000 #102 #163 #224 #285 #346 #41 0x3635343200000000 #407 #468 0x617373656d0d0202 #160 #221 #282 #343 #38 #404 #465 #99 0x6567617373656d0d #145 #206 #23 #267 #328 #389 #450 #84 0x65756575715f6567 #100 #161 #222 #283 #344 #39 #405 #466 0x6575715f65676173 #115 #176 #237 #298 #359 #420 #481 #54 0x6d0d020210010413 #129 #190 #251 #312 #373 #434 #495 #68 #7 0x6e7302010000000c #127 #188 #249 #310 #371 #432 #493 #5 #66 0x715f656761737365 #130 #191 #252 #313 #374 #435 #496 #69 #8 0x73656d0d02021001 #114 #175 #236 #297 #358 #419 #480 #53 0xbf0300216e730201 #158 #219 #280 #341 #36 #402 #463 #97 Original coredump: GNU gdb Fedora (6.8-27.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... warning: .dynamic section for "/usr/lib64/libqpidbroker.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libqpidcommon.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libqpidclient.so.0" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libxerces-c.so.28" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib64/libxqilla.so.3" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidbroker.so.0 Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidcommon.so.0 Reading symbols from /usr/lib64/libboost_program_options.so.2...done. Loaded symbols for /usr/lib64/libboost_program_options.so.2 Reading symbols from /usr/lib64/libboost_filesystem.so.2...done. Loaded symbols for /usr/lib64/libboost_filesystem.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/librt.so.1...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libpthread.so.0...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so Reading symbols from /usr/lib64/qpid/daemon/rdma.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/rdma.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/rdma.so Reading symbols from /usr/lib64/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib64/librdmawrap.so.0.1.0.debug...done. done. Loaded symbols for /usr/lib64/librdmawrap.so.0 Reading symbols from /usr/lib64/librdmacm.so.1...done. Loaded symbols for /usr/lib64/librdmacm.so.1 Reading symbols from /usr/lib64/libibverbs.so.1...done. Loaded symbols for /usr/lib64/libibverbs.so.1 Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/cluster.so Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/openais/libcpg.so.2 Reading symbols from /usr/lib64/libcman.so.2...done. Loaded symbols for /usr/lib64/libcman.so.2 Reading symbols from /usr/lib64/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.0.1.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidclient.so.0 Reading symbols from /usr/lib64/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/rdmaconnector.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/client/rdmaconnector.so Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/client/sslconnector.so Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done. done. Loaded symbols for /usr/lib64/libsslcommon.so.0 Reading symbols from /usr/lib64/libnss3.so...done. Loaded symbols for /usr/lib64/libnss3.so Reading symbols from /usr/lib64/libssl3.so...done. Loaded symbols for /usr/lib64/libssl3.so Reading symbols from /usr/lib64/libnspr4.so...done. Loaded symbols for /usr/lib64/libnspr4.so Reading symbols from /usr/lib64/libnssutil3.so...done. Loaded symbols for /usr/lib64/libnssutil3.so Reading symbols from /usr/lib64/libplc4.so...done. Loaded symbols for /usr/lib64/libplc4.so Reading symbols from /usr/lib64/libplds4.so...done. Loaded symbols for /usr/lib64/libplds4.so Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/xml.so Reading symbols from /usr/lib64/libxerces-c.so.28...done. Loaded symbols for /usr/lib64/libxerces-c.so.28 Reading symbols from /usr/lib64/libxqilla.so.3...done. Loaded symbols for /usr/lib64/libxqilla.so.3 Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/acl.so Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/ssl.so Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...done. Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so Reading symbols from /usr/lib64/libdb_cxx-4.3.so...done. Loaded symbols for /usr/lib64/libdb_cxx-4.3.so Reading symbols from /usr/lib64/libaio.so.1...done. Loaded symbols for /usr/lib64/libaio.so.1 Core was generated by `qpidd --cluster-name soakTestCluster_b78a0df7-65aa-49f3-8f82-5de9c70fcce3 --aut'. Program terminated with signal 11, Segmentation fault. [New process 7047] [New process 7052] [New process 7051] [New process 7050] [New process 7049] [New process 7048] [New process 7046] [New process 7045] [New process 7044] [New process 7042] [New process 7041] [New process 7040] #0 0x00000036e047b703 in memcpy () from /lib64/libc.so.6 (gdb) Thread 12 (process 7040): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b074318841e in qpid::broker::Broker::run ( this=<value optimized out>) at qpid/broker/Broker.cpp:319 #4 0x00000000004069b8 in QpiddBroker::execute (this=<value optimized out>, options=0x8d26d40) at posix/QpiddBroker.cpp:166 #5 0x00000000004054a8 in main (argc=16, argv=0x7fff67a00f48) at qpidd.cpp:77 Thread 11 (process 7041): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b0743250379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 10 (process 7042): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b0743250379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 9 (process 7044): #0 0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b0743250379 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69 #2 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 8 (process 7045): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 7 (process 7046): #0 0x00000036e04c56db in write () from /lib64/libc.so.6 #1 0x00002b07468de1a0 in mrg::journal::fcntl::clean_file () from /usr/lib64/qpid/daemon/msgstore.so #2 0x00002b07468df63c in mrg::journal::fcntl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #3 0x00002b07468df73f in mrg::journal::fcntl::fcntl () from /usr/lib64/qpid/daemon/msgstore.so #4 0x00002b07468e5f37 in mrg::journal::jcntl::new_fcntl () from /usr/lib64/qpid/daemon/msgstore.so #5 0x00002b07468f8f6f in mrg::journal::lpmgr::append () from /usr/lib64/qpid/daemon/msgstore.so #6 0x00002b07468f9b7a in mrg::journal::lpmgr::initialize () from /usr/lib64/qpid/daemon/msgstore.so #7 0x00002b07468e5dd8 in mrg::journal::jcntl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #8 0x00002b074689573c in mrg::msgstore::JournalImpl::initialize () from /usr/lib64/qpid/daemon/msgstore.so #9 0x00002b07468b43c0 in mrg::msgstore::MessageStoreImpl::create () from /usr/lib64/qpid/daemon/msgstore.so #10 0x00002b0743207308 in qpid::broker::MessageStoreModule::create ( this=<value optimized out>, queue=<value optimized out>, args=<value optimized out>) at qpid/broker/MessageStoreModule.cpp:46 #11 0x00002b07431a6825 in qpid::broker::Queue::create ( this=<value optimized out>, _settings=<value optimized out>) at qpid/broker/Queue.cpp:757 #12 0x00002b074323b0cf in qpid::broker::SessionAdapter::QueueHandlerImpl::declare (this=<value optimized out>, name=<value optimized out>, alternateExchange=<value optimized out>, passive=<value optimized out>, durable=<value optimized out>, exclusive=<value optimized out>, autoDelete=<value optimized out>, arguments=Could not find the frame base for "qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&)". ) at qpid/broker/SessionAdapter.cpp:359 #13 0x00002b07435fdf06 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at gen/qpid/framing/QueueDeclareBody.h:95 #14 0x00002b07436038d7 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at gen/qpid/framing/ServerInvoker.cpp:368 #15 0x00002b074324a33e in qpid::framing::invoke<qpid::broker::SessionAdapter> ( target=<value optimized out>, body=<value optimized out>) at qpid/framing/Invoker.h:67 #16 0x00002b0743247adc in qpid::broker::SessionState::handleCommand ( this=<value optimized out>, method=<value optimized out>, id=<value optimized out>) at qpid/broker/SessionState.cpp:194 #17 0x00002b07432491a7 in qpid::broker::SessionState::handleIn ( this=<value optimized out>, frame=<value optimized out>) at qpid/broker/SessionState.cpp:328 #18 0x00002b074366c53e in qpid::amqp_0_10::SessionHandler::handleIn ( this=<value optimized out>, f=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:91 #19 0x00002b07431b9f79 in qpid::broker::Connection::received ( this=<value optimized out>, frame=<value optimized out>) at qpid/framing/Handler.h:42 #20 0x00002b07443ee2b6 in qpid::cluster::Connection::received (this=0x8d52a20, f=@0x8d52b00) at qpid/cluster/Connection.cpp:123 #21 0x00002b07443eb59d in qpid::cluster::Connection::decode (this=0x8d52a20, buffer=0x2aaab0001af0 "\017", size=12893) at qpid/cluster/Connection.cpp:218 #22 0x00002b07436953c7 in qpid::sys::AsynchIOHandler::readbuff ( this=<value optimized out>, buff=<value optimized out>) at qpid/sys/AsynchIOHandler.cpp:113 #23 0x00002b0743272098 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke ( function_obj_ptr=<value optimized out>, a0=<value optimized out>, a1=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:252 #24 0x00002b07436417ba in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=<value optimized out>, a0=<value optimized out>, a1=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #25 0x00002b074363b5b3 in qpid::sys::posix::AsynchIO::readable ( this=<value optimized out>, h=<value optimized out>) at qpid/sys/posix/AsynchIO.cpp:447 #26 0x00002b074369ad07 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #27 0x00002b0743698a0b in qpid::sys::DispatchHandle::processEvent ( this=<value optimized out>, type=<value optimized out>) at qpid/sys/DispatchHandle.cpp:432 #28 0x00002b074364ecb3 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/Poller.h:122 #29 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #30 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #31 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 6 (process 7048): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 5 (process 7049): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 4 (process 7050): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 3 (process 7051): #0 0x00000036e100cfb5 in __lll_unlock_wake () from /lib64/libpthread.so.0 #1 0x00000036e1009a89 in _L_unlock_63 () from /lib64/libpthread.so.0 #2 0x00000036e1009727 in __pthread_mutex_unlock_usercnt () from /lib64/libpthread.so.0 #3 0x00002b074314b2ea in qpid::sys::Mutex::unlock (this=<value optimized out>) at qpid/sys/posix/Mutex.h:120 #4 0x00002b07443dacb8 in qpid::sys::PollableQueue<qpid::cluster::Event>::dispatch (this=0x8d45730, cond=@0x8d457a0) at qpid/sys/Mutex.h:34 #5 0x00002b074364c10f in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() ( this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #6 0x00002b074369ad07 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:576 #7 0x00002b0743698a0b in qpid::sys::DispatchHandle::processEvent ( this=<value optimized out>, type=<value optimized out>) at qpid/sys/DispatchHandle.cpp:432 #8 0x00002b074364ecb3 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/Poller.h:122 #9 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #10 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #11 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 2 (process 7052): #0 0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b074364e0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:439 #2 0x00002b074364ec87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405 #3 0x00002b0743644cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0 #5 0x00000036e04d30ad in clone () from /lib64/libc.so.6 Thread 1 (process 7047): #0 0x00000036e047b703 in memcpy () from /lib64/libc.so.6 #1 0x00002b074464a585 in memcpy_swrap (dest=<value optimized out>, src=<value optimized out>, len=<value optimized out>, read=<value optimized out>) at /usr/include/bits/string3.h:51 #2 0x00002b074464adbe in openais_dispatch_recv ( ipc_ctx=<value optimized out>, data=<value optimized out>, timeout=<value optimized out>) at util.c:554 #3 0x00002b074464bb8a in cpg_dispatch (handle=<value optimized out>, dispatch_types=<value optimized out>) at cpg.c:266 #4 0x0000000600000000 in ?? () #5 0x6e7302010000000c in ?? () #6 0x00000047bf030021 in ?? () #7 0x6d0d020210010413 in ?? () #8 0x715f656761737365 in ?? () #9 0x1200030765756575 in ?? () #10 0x0000000002000100 in ?? () #11 0x000f353735353432 in ?? () #12 0x0000020000001000 in ?? () #13 0x010b00040c020000 in ?? () #14 0x0000020001001500 in ?? () #15 0x0007000101040000 in ?? () #16 0x0043000203000100 in ?? () #17 0x0000000000020001 in ?? () #18 0x00010103041c0000 in ?? () #19 0x0006000000000000 in ?? () #20 0x02010000000c0000 in ?? () #21 0x0048bf0300216e73 in ?? () #22 0x0202100104130000 in ?? () #23 0x6567617373656d0d in ?? () #24 0x030765756575715f in ?? () #25 0x0000020001001200 in ?? () #26 0x3637353534320000 in ?? () #27 0x020000001000000f in ?? () #28 0x00040c0200000000 in ?? () #29 0x020001001500010b in ?? () #30 0x0001010400000000 in ?? () #31 0x0002030001000007 in ?? () #32 0x0000000200010043 in ?? () #33 0x0103041c00000000 in ?? () #34 0x0000000000000001 in ?? () #35 0x0000000c00000006 in ?? () #36 0xbf0300216e730201 in ?? () #37 0x1001041300000049 in ?? () #38 0x617373656d0d0202 in ?? () #39 0x65756575715f6567 in ?? () #40 0x0200010012000307 in ?? () #41 0x3535343200000000 in ?? () #42 0x00001000000f3737 in ?? () #43 0x0c02000000000200 in ?? () #44 0x01001500010b0004 in ?? () #45 0x0104000000000200 in ?? () #46 0x0300010000070001 in ?? () #47 0x0002000100430002 in ?? () #48 0x041c000000000000 in ?? () #49 0x0000000000010103 in ?? () #50 0x000c000000060000 in ?? () #51 0x00216e7302010000 in ?? () #52 0x04130000004abf03 in ?? () #53 0x73656d0d02021001 in ?? () #54 0x6575715f65676173 in ?? () #55 0x0100120003076575 in ?? () #56 0x3432000000000200 in ?? () #57 0x1000000f38373535 in ?? () #58 0x0000000002000000 in ?? () #59 0x1500010b00040c02 in ?? () #60 0x0000000002000100 in ?? () #61 0x0100000700010104 in ?? () #62 0x0001004300020300 in ?? () #63 0x0000000000000002 in ?? () #64 0x000000010103041c in ?? () #65 0x0000000600000000 in ?? () #66 0x6e7302010000000c in ?? () #67 0x0000004bbf030021 in ?? () #68 0x6d0d020210010413 in ?? () #69 0x715f656761737365 in ?? () #70 0x1200030765756575 in ?? () #71 0x0000000002000100 in ?? () #72 0x000f393735353432 in ?? () #73 0x0000020000001000 in ?? () #74 0x010b00040c020000 in ?? () #75 0x0000020001001500 in ?? () #76 0x0007000101040000 in ?? () #77 0x0043000203000100 in ?? () #78 0x0000000000020001 in ?? () #79 0x00010103041c0000 in ?? () #80 0x0006000000000000 in ?? () #81 0x02010000000c0000 in ?? () #82 0x004cbf0300216e73 in ?? () #83 0x0202100104130000 in ?? () #84 0x6567617373656d0d in ?? () #85 0x030765756575715f in ?? () #86 0x0000020001001200 in ?? () #87 0x3038353534320000 in ?? () #88 0x020000001000000f in ?? () #89 0x00040c0200000000 in ?? () #90 0x020001001500010b in ?? () #91 0x0001010400000000 in ?? () #92 0x0002030001000007 in ?? () #93 0x0000000200010043 in ?? () #94 0x0103041c00000000 in ?? () #95 0x0000000000000001 in ?? () #96 0x0000000c00000006 in ?? () #97 0xbf0300216e730201 in ?? () #98 0x100104130000004d in ?? () #99 0x617373656d0d0202 in ?? () #100 0x65756575715f6567 in ?? () #101 0x0200010012000307 in ?? () #102 0x3535343200000000 in ?? () #103 0x00001000000f3138 in ?? () #104 0x0c02000000000200 in ?? () #105 0x01001500010b0004 in ?? () #106 0x0104000000000200 in ?? () #107 0x0300010000070001 in ?? () #108 0x0002000100430002 in ?? () #109 0x041c000000000000 in ?? () #110 0x0000000000010103 in ?? () #111 0x000c000000060000 in ?? () #112 0x00216e7302010000 in ?? () #113 0x04130000004ebf03 in ?? () #114 0x73656d0d02021001 in ?? () #115 0x6575715f65676173 in ?? () #116 0x0100120003076575 in ?? () #117 0x3432000000000200 in ?? () #118 0x1000000f32383535 in ?? () #119 0x0000000002000000 in ?? () #120 0x1500010b00040c02 in ?? () #121 0x0000000002000100 in ?? () #122 0x0100000700010104 in ?? () #123 0x0001004300020300 in ?? () #124 0x0000000000000002 in ?? () #125 0x000000010103041c in ?? () #126 0x0000000600000000 in ?? () #127 0x6e7302010000000c in ?? () #128 0x0000004fbf030021 in ?? () #129 0x6d0d020210010413 in ?? () #130 0x715f656761737365 in ?? () #131 0x1200030765756575 in ?? () #132 0x0000000002000100 in ?? () #133 0x000f333835353432 in ?? () #134 0x0000020000001000 in ?? () #135 0x010b00040c020000 in ?? () #136 0x0000020001001500 in ?? () #137 0x0007000101040000 in ?? () #138 0x0043000203000100 in ?? () #139 0x0000000000020001 in ?? () #140 0x00010103041c0000 in ?? () #141 0x0006000000000000 in ?? () #142 0x02010000000c0000 in ?? () #143 0x0050bf0300216e73 in ?? () #144 0x0202100104130000 in ?? () #145 0x6567617373656d0d in ?? () #146 0x030765756575715f in ?? () #147 0x0000020001001200 in ?? () #148 0x3438353534320000 in ?? () #149 0x020000001000000f in ?? () #150 0x00040c0200000000 in ?? () #151 0x020001001500010b in ?? () #152 0x0001010400000000 in ?? () #153 0x0002030001000007 in ?? () #154 0x0000000200010043 in ?? () #155 0x0103041c00000000 in ?? () #156 0x0000000000000001 in ?? () #157 0x0000000c00000006 in ?? () #158 0xbf0300216e730201 in ?? () #159 0x1001041300000051 in ?? () #160 0x617373656d0d0202 in ?? () #161 0x65756575715f6567 in ?? () #162 0x0200010012000307 in ?? () #163 0x3535343200000000 in ?? () #164 0x00001000000f3538 in ?? () #165 0x0c02000000000200 in ?? () #166 0x01001500010b0004 in ?? () #167 0x0104000000000200 in ?? () #168 0x0300010000070001 in ?? () #169 0x0002000100430002 in ?? () #170 0x041c000000000000 in ?? () #171 0x0000000000010103 in ?? () #172 0x000c000000060000 in ?? () #173 0x00216e7302010000 in ?? () #174 0x041300000052bf03 in ?? () #175 0x73656d0d02021001 in ?? () #176 0x6575715f65676173 in ?? () #177 0x0100120003076575 in ?? () #178 0x3432000000000200 in ?? () #179 0x1000000f36383535 in ?? () #180 0x0000000002000000 in ?? () #181 0x1500010b00040c02 in ?? () #182 0x0000000002000100 in ?? () #183 0x0100000700010104 in ?? () #184 0x0001004300020300 in ?? () #185 0x0000000000000002 in ?? () #186 0x000000010103041c in ?? () #187 0x0000000600000000 in ?? () #188 0x6e7302010000000c in ?? () #189 0x00000053bf030021 in ?? () #190 0x6d0d020210010413 in ?? () #191 0x715f656761737365 in ?? () #192 0x1200030765756575 in ?? () #193 0x0000000002000100 in ?? () #194 0x000f373835353432 in ?? () #195 0x0000020000001000 in ?? () #196 0x010b00040c020000 in ?? () #197 0x0000020001001500 in ?? () #198 0x0007000101040000 in ?? () #199 0x0043000203000100 in ?? () #200 0x0000000000020001 in ?? () #201 0x00010103041c0000 in ?? () #202 0x0006000000000000 in ?? () #203 0x02010000000c0000 in ?? () #204 0x0054bf0300216e73 in ?? () #205 0x0202100104130000 in ?? () #206 0x6567617373656d0d in ?? () #207 0x030765756575715f in ?? () #208 0x0000020001001200 in ?? () #209 0x3838353534320000 in ?? () #210 0x020000001000000f in ?? () #211 0x00040c0200000000 in ?? () #212 0x020001001500010b in ?? () #213 0x0001010400000000 in ?? () #214 0x0002030001000007 in ?? () #215 0x0000000200010043 in ?? () #216 0x0103041c00000000 in ?? () #217 0x0000000000000001 in ?? () #218 0x0000000c00000006 in ?? () #219 0xbf0300216e730201 in ?? () #220 0x1001041300000055 in ?? () #221 0x617373656d0d0202 in ?? () #222 0x65756575715f6567 in ?? () #223 0x0200010012000307 in ?? () #224 0x3535343200000000 in ?? () #225 0x00001000000f3938 in ?? () #226 0x0c02000000000200 in ?? () #227 0x01001500010b0004 in ?? () #228 0x0104000000000200 in ?? () #229 0x0300010000070001 in ?? () #230 0x0002000100430002 in ?? () #231 0x041c000000000000 in ?? () #232 0x0000000000010103 in ?? () #233 0x000c000000060000 in ?? () #234 0x00216e7302010000 in ?? () #235 0x041300000056bf03 in ?? () #236 0x73656d0d02021001 in ?? () #237 0x6575715f65676173 in ?? () #238 0x0100120003076575 in ?? () #239 0x3432000000000200 in ?? () #240 0x1000000f30393535 in ?? () #241 0x0000000002000000 in ?? () #242 0x1500010b00040c02 in ?? () #243 0x0000000002000100 in ?? () #244 0x0100000700010104 in ?? () #245 0x0001004300020300 in ?? () #246 0x0000000000000002 in ?? () #247 0x000000010103041c in ?? () #248 0x0000000600000000 in ?? () #249 0x6e7302010000000c in ?? () #250 0x00000057bf030021 in ?? () #251 0x6d0d020210010413 in ?? () #252 0x715f656761737365 in ?? () #253 0x1200030765756575 in ?? () #254 0x0000000002000100 in ?? () #255 0x000f313935353432 in ?? () #256 0x0000020000001000 in ?? () #257 0x010b00040c020000 in ?? () #258 0x0000020001001500 in ?? () #259 0x0007000101040000 in ?? () #260 0x0043000203000100 in ?? () #261 0x0000000000020001 in ?? () #262 0x00010103041c0000 in ?? () #263 0x0006000000000000 in ?? () #264 0x02010000000c0000 in ?? () #265 0x0058bf0300216e73 in ?? () #266 0x0202100104130000 in ?? () #267 0x6567617373656d0d in ?? () #268 0x030765756575715f in ?? () #269 0x0000020001001200 in ?? () #270 0x3239353534320000 in ?? () #271 0x020000001000000f in ?? () #272 0x00040c0200000000 in ?? () #273 0x020001001500010b in ?? () #274 0x0001010400000000 in ?? () #275 0x0002030001000007 in ?? () #276 0x0000000200010043 in ?? () #277 0x0103041c00000000 in ?? () #278 0x0000000000000001 in ?? () #279 0x0000000c00000006 in ?? () #280 0xbf0300216e730201 in ?? () #281 0x1001041300000059 in ?? () #282 0x617373656d0d0202 in ?? () #283 0x65756575715f6567 in ?? () #284 0x0200010012000307 in ?? () #285 0x3535343200000000 in ?? () #286 0x00001000000f3339 in ?? () #287 0x0c02000000000200 in ?? () #288 0x01001500010b0004 in ?? () #289 0x0104000000000200 in ?? () #290 0x0300010000070001 in ?? () #291 0x0002000100430002 in ?? () #292 0x041c000000000000 in ?? () #293 0x0000000000010103 in ?? () #294 0x000c000000060000 in ?? () #295 0x00216e7302010000 in ?? () #296 0x04130000005abf03 in ?? () #297 0x73656d0d02021001 in ?? () #298 0x6575715f65676173 in ?? () #299 0x0100120003076575 in ?? () #300 0x3432000000000200 in ?? () #301 0x1000000f34393535 in ?? () #302 0x0000000002000000 in ?? () #303 0x1500010b00040c02 in ?? () #304 0x0000000002000100 in ?? () #305 0x0100000700010104 in ?? () #306 0x0001004300020300 in ?? () #307 0x0000000000000002 in ?? () #308 0x000000010103041c in ?? () #309 0x0000000600000000 in ?? () #310 0x6e7302010000000c in ?? () #311 0x0000005bbf030021 in ?? () #312 0x6d0d020210010413 in ?? () #313 0x715f656761737365 in ?? () #314 0x1200030765756575 in ?? () #315 0x0000000002000100 in ?? () #316 0x000f353935353432 in ?? () #317 0x0000020000001000 in ?? () #318 0x010b00040c020000 in ?? () #319 0x0000020001001500 in ?? () #320 0x0007000101040000 in ?? () #321 0x0043000203000100 in ?? () #322 0x0000000000020001 in ?? () #323 0x00010103041c0000 in ?? () #324 0x0006000000000000 in ?? () #325 0x02010000000c0000 in ?? () #326 0x005cbf0300216e73 in ?? () #327 0x0202100104130000 in ?? () #328 0x6567617373656d0d in ?? () #329 0x030765756575715f in ?? () #330 0x0000020001001200 in ?? () #331 0x3639353534320000 in ?? () #332 0x020000001000000f in ?? () #333 0x00040c0200000000 in ?? () #334 0x020001001500010b in ?? () #335 0x0001010400000000 in ?? () #336 0x0002030001000007 in ?? () #337 0x0000000200010043 in ?? () #338 0x0103041c00000000 in ?? () #339 0x0000000000000001 in ?? () #340 0x0000000c00000006 in ?? () #341 0xbf0300216e730201 in ?? () #342 0x100104130000005d in ?? () #343 0x617373656d0d0202 in ?? () #344 0x65756575715f6567 in ?? () #345 0x0200010012000307 in ?? () #346 0x3535343200000000 in ?? () #347 0x00001000000f3739 in ?? () #348 0x0c02000000000200 in ?? () #349 0x01001500010b0004 in ?? () #350 0x0104000000000200 in ?? () #351 0x0300010000070001 in ?? () #352 0x0002000100430002 in ?? () #353 0x041c000000000000 in ?? () #354 0x0000000000010103 in ?? () #355 0x000c000000060000 in ?? () #356 0x00216e7302010000 in ?? () #357 0x04130000005ebf03 in ?? () #358 0x73656d0d02021001 in ?? () #359 0x6575715f65676173 in ?? () #360 0x0100120003076575 in ?? () #361 0x3432000000000200 in ?? () #362 0x1000000f38393535 in ?? () #363 0x0000000002000000 in ?? () #364 0x1500010b00040c02 in ?? () #365 0x0000000002000100 in ?? () #366 0x0100000700010104 in ?? () #367 0x0001004300020300 in ?? () #368 0x0000000000000002 in ?? () #369 0x000000010103041c in ?? () #370 0x0000000600000000 in ?? () #371 0x6e7302010000000c in ?? () #372 0x0000005fbf030021 in ?? () #373 0x6d0d020210010413 in ?? () #374 0x715f656761737365 in ?? () #375 0x1200030765756575 in ?? () #376 0x0000000002000100 in ?? () #377 0x000f393935353432 in ?? () #378 0x0000020000001000 in ?? () #379 0x010b00040c020000 in ?? () #380 0x0000020001001500 in ?? () #381 0x0007000101040000 in ?? () #382 0x0043000203000100 in ?? () #383 0x0000000000020001 in ?? () #384 0x00010103041c0000 in ?? () #385 0x0006000000000000 in ?? () #386 0x02010000000c0000 in ?? () #387 0x0060bf0300216e73 in ?? () #388 0x0202100104130000 in ?? () #389 0x6567617373656d0d in ?? () #390 0x030765756575715f in ?? () #391 0x0000020001001200 in ?? () #392 0x3030363534320000 in ?? () #393 0x020000001000000f in ?? () #394 0x00040c0200000000 in ?? () #395 0x020001001500010b in ?? () #396 0x0001010400000000 in ?? () #397 0x0002030001000007 in ?? () #398 0x0000000200010043 in ?? () #399 0x0103041c00000000 in ?? () #400 0x0000000000000001 in ?? () #401 0x0000000c00000006 in ?? () #402 0xbf0300216e730201 in ?? () #403 0x1001041300000061 in ?? () #404 0x617373656d0d0202 in ?? () #405 0x65756575715f6567 in ?? () #406 0x0200010012000307 in ?? () #407 0x3635343200000000 in ?? () #408 0x00001000000f3130 in ?? () #409 0x0c02000000000200 in ?? () #410 0x01001500010b0004 in ?? () #411 0x0104000000000200 in ?? () #412 0x0300010000070001 in ?? () #413 0x0002000100430002 in ?? () #414 0x041c000000000000 in ?? () #415 0x0000000000010103 in ?? () #416 0x000c000000060000 in ?? () #417 0x00216e7302010000 in ?? () #418 0x041300000062bf03 in ?? () #419 0x73656d0d02021001 in ?? () #420 0x6575715f65676173 in ?? () #421 0x0100120003076575 in ?? () #422 0x3432000000000200 in ?? () #423 0x1000000f32303635 in ?? () #424 0x0000000002000000 in ?? () #425 0x1500010b00040c02 in ?? () #426 0x0000000002000100 in ?? () #427 0x0100000700010104 in ?? () #428 0x0001004300020300 in ?? () #429 0x0000000000000002 in ?? () #430 0x000000010103041c in ?? () #431 0x0000000600000000 in ?? () #432 0x6e7302010000000c in ?? () #433 0x00000063bf030021 in ?? () #434 0x6d0d020210010413 in ?? () #435 0x715f656761737365 in ?? () #436 0x1200030765756575 in ?? () #437 0x0000000002000100 in ?? () #438 0x000f333036353432 in ?? () #439 0x0000020000001000 in ?? () #440 0x010b00040c020000 in ?? () #441 0x0000020001001500 in ?? () #442 0x0007000101040000 in ?? () #443 0x0043000203000100 in ?? () #444 0x0000000000020001 in ?? () #445 0x00010103041c0000 in ?? () #446 0x0006000000000000 in ?? () #447 0x02010000000c0000 in ?? () #448 0x0064bf0300216e73 in ?? () #449 0x0202100104130000 in ?? () #450 0x6567617373656d0d in ?? () #451 0x030765756575715f in ?? () #452 0x0000020001001200 in ?? () #453 0x3430363534320000 in ?? () #454 0x020000001000000f in ?? () #455 0x00040c0200000000 in ?? () #456 0x020001001500010b in ?? () #457 0x0001010400000000 in ?? () #458 0x0002030001000007 in ?? () #459 0x0000000200010043 in ?? () #460 0x0103041c00000000 in ?? () #461 0x0000000000000001 in ?? () #462 0x0000000c00000006 in ?? () #463 0xbf0300216e730201 in ?? () #464 0x1001041300000065 in ?? () #465 0x617373656d0d0202 in ?? () #466 0x65756575715f6567 in ?? () #467 0x0200010012000307 in ?? () #468 0x3635343200000000 in ?? () #469 0x00001000000f3530 in ?? () #470 0x0c02000000000200 in ?? () #471 0x01001500010b0004 in ?? () #472 0x0104000000000200 in ?? () #473 0x0300010000070001 in ?? () #474 0x0002000100430002 in ?? () #475 0x041c000000000000 in ?? () #476 0x0000000000010103 in ?? () #477 0x000c000000060000 in ?? () #478 0x00216e7302010000 in ?? () #479 0x041300000066bf03 in ?? () #480 0x73656d0d02021001 in ?? () #481 0x6575715f65676173 in ?? () #482 0x0100120003076575 in ?? () #483 0x3432000000000200 in ?? () #484 0x1000000f36303635 in ?? () #485 0x0000000002000000 in ?? () #486 0x1500010b00040c02 in ?? () #487 0x0000000002000100 in ?? () #488 0x0100000700010104 in ?? () #489 0x0001004300020300 in ?? () #490 0x0000000000000002 in ?? () #491 0x000000010103041c in ?? () #492 0x0000000600000000 in ?? () #493 0x6e7302010000000c in ?? () #494 0x00000067bf030021 in ?? () #495 0x6d0d020210010413 in ?? () #496 0x715f656761737365 in ?? () #497 0x1200030765756575 in ?? () #498 0x0000000002000100 in ?? () #499 0x000f373036353432 in ?? () #500 0x0000020000001000 in ?? () #501 0x010b00040c020000 in ?? () #502 0x00002aaa0000012d in ?? () #503 0x0000000000000005 in ?? () #504 0x00007fff50a31c48 in ?? () #505 0x00007fff00000034 in ?? () #506 0x747365546b616f73 in ?? () #507 0x5f72657473756c43 in ?? () #508 0x3766643061383762 in ?? () #509 0x39342d616135362d in ?? () #510 0x2d323866382d3366 in ?? () #511 0x6630376339656435 in ?? () #512 0x912b399e33656363 in ?? () #513 0x191858d000002b11 in ?? () #514 0xeca9c90100000000 in ?? () #515 0x1917068800000036 in ?? () #516 0x1918579800000000 in ?? () #517 0x191858b000000000 in ?? () #518 0xe047402a00000000 in ?? () #519 0x191858d000000036 in ?? () #520 0x9270980100000000 in ?? () #521 0x191833b000002b11 in ?? () #522 0x0413000000000075 in ?? () #523 0x73656d0d3721220a in ?? () #524 0x6575715f00001b46 in ?? () #525 0x0000000000000000 in ?? ()
I've been running failover_soak on the current trunk code for 8 hours without a crash. I believe this issue is fixed.