Red Hat Bugzilla – Bug 586997
qpid c++ client library crash in qpid::client::ConnectionImpl::release()
Last modified: 2015-11-15 20:12:16 EST
Created attachment 409894 [details] the test reproducer Description of problem: There is observed crash of qpic c++ client (perftest) during qpidd broker stress test in qpid::client::ConnectionImpl::release(): Thread 1 (process 18252): #0 0x00000018 in ?? () ---Type <return> to continue, or q <return> to quit--- #1 0x007997b1 in qpid::client::ConnectionImpl::release (this=0x83755a0) at /usr/include/boost/scoped_ptr.hpp:93 #2 0x007a7dcd in boost::detail::sp_counted_impl_pd<qpid::client::ConnectionImpl*, boost::_bi::bind_t<void, boost::_mfi::mf0<void, qpid::client::ConnectionImpl>, boost::_bi::list1<boost::arg<1> > > >::dispose (this=0x8579b20) at /usr/include/boost/bind/mem_fn_template.hpp:45 #3 0x00789f9e in ~Connection (this=0xbffa6268) at ./boost/detail/sp_counted_base_gcc_x86.hpp:145 #4 0x080537cd in ~Client (this=0xbffa6260) at ../../include/qpid/client/AsyncSession_0_10.h:553 #5 0x08050136 in main (argc=15, argv=0xbffa6444) at perftest.cpp:719 #6 0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6 #7 0x0804f2c1 in _start () Observed on RHEL 4.8 i386, one occurrence so far. Version-Release number of selected component (if applicable): How reproducible: rarely Steps to Reproduce: 1. start broker 2. ./run.sh 30 1 0; while true; do ./run.sh $(( $((${RANDOM}%6)) + 5)) 0 0 || break; done this line is tuned for 8 core machine, lower number of clients for less powerfull machines 3. wait for crash Actual results: qpid c++ client crashes. Expected results: qpid c++ client should not crash. Additional info: ./run.sh: line 30: 18252 Segmentation fault (core dumped) perftest --unique-data yes --base-name=perftest_${i_fmt}_ -s --mechanism PLAIN --user guest --password guest --count $((${RANDOM} * 10)) --durable $((${RANDOM} % 2)) >&./perftest_${i_fmt}.log 0|9|9 > 12:10:14 up 36 days, 7:33, 1 user, load average: 9.32, 9.87, 8.80< perftest: no process killed .done (err_cnt:1, dur:55) [root@mrg-qe-07 _]# file core.18252 core.18252: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'perftest' [root@mrg-qe-07 _]# gdb `which perftest` core.18252 GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Core was generated by `perftest --unique-data yes --base-name=perftest_004_ -s --mechanism PLAIN --use'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidclient.so.2.0.0.debug...done. done. ... Reading symbols from /usr/lib/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib/sasl2/libanonymous.so.2 #0 0x00000018 in ?? () (gdb) info threads 5 process 18253 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 4 process 18254 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 3 process 18256 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 2 process 18258 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 * 1 process 18252 0x00000018 in ?? () (gdb) thread apply all bt Thread 5 (process 18253): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x816c468, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x816c468) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 4 (process 18254): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x816c468, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x816c468) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 3 (process 18256): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x816c468, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x816c468) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 2 (process 18258): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x816c468, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x816c468) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 1 (process 18252): #0 0x00000018 in ?? () ---Type <return> to continue, or q <return> to quit--- #1 0x007997b1 in qpid::client::ConnectionImpl::release (this=0x83755a0) at /usr/include/boost/scoped_ptr.hpp:93 #2 0x007a7dcd in boost::detail::sp_counted_impl_pd<qpid::client::ConnectionImpl*, boost::_bi::bind_t<void, boost::_mfi::mf0<void, qpid::client::ConnectionImpl>, boost::_bi::list1<boost::arg<1> > > >::dispose (this=0x8579b20) at /usr/include/boost/bind/mem_fn_template.hpp:45 #3 0x00789f9e in ~Connection (this=0xbffa6268) at ./boost/detail/sp_counted_base_gcc_x86.hpp:145 #4 0x080537cd in ~Client (this=0xbffa6260) at ../../include/qpid/client/AsyncSession_0_10.h:553 #5 0x08050136 in main (argc=15, argv=0xbffa6444) at perftest.cpp:719 #6 0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6 #7 0x0804f2c1 in _start () (gdb) Quit
On RHEL 4.8 i386 observed on packages: [root@mrg-qe-07 _]# rpm -qa | grep qpid | sort python-qpid-0.7.934605-1.el4 qpid-cpp-client-0.7.935473-1.el4 qpid-cpp-client-devel-0.7.935473-1.el4 qpid-cpp-client-devel-docs-0.7.935473-1.el4 qpid-cpp-client-ssl-0.7.935473-1.el4 qpid-cpp-mrg-debuginfo-0.7.935473-1.el4 qpid-cpp-server-0.7.935473-1.el4 qpid-cpp-server-devel-0.7.935473-1.el4 qpid-cpp-server-ssl-0.7.935473-1.el4 qpid-cpp-server-store-0.7.935473-1.el4 qpid-cpp-server-xml-0.7.935473-1.el4 qpid-java-client-0.7.934605-1.el4 qpid-java-common-0.7.934605-1.el4 qpid-tests-0.7.930108-1.el4 qpid-tools-0.7.934605-2.el4
There was found another occurrence of the issue on rhel 4.8 x86_64. Reproducibility is still rare. [root@mrg-qe-03 _]# file core.14511 core.14511: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'perftest' [root@mrg-qe-03 _]# gdb `which perftest` core.14511 GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1". Core was generated by `perftest --unique-data yes --base-name=perftest_006_ -s --mechanism PLAIN --use'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib64/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.2.0.0.debug...done. done. ... Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 #0 0x0000003e436b853a in qpid::client::ConnectionImpl::release (this=0x96d720) at /usr/include/boost/scoped_ptr.hpp:93 93 { (gdb) info threads 5 process 14512 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 4 process 14513 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 3 process 14515 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 2 process 14517 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 * 1 process 14511 0x0000003e436b853a in qpid::client::ConnectionImpl::release (this=0x96d720) at /usr/include/boost/scoped_ptr.hpp:93 (gdb) thread apply all bt Thread 5 (process 14512): #0 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 #1 0x0000003e4336c9c9 in qpid::sys::Poller::wait (this=0x55e730, timeout=Variable "timeout" is not available. ) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003e4336d826 in qpid::sys::Poller::run (this=0x55e730) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003e4336197a in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f30d06317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x0000003f300c9fc3 in clone () from /lib64/tls/libc.so.6 Thread 4 (process 14513): #0 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 #1 0x0000003e4336c9c9 in qpid::sys::Poller::wait (this=0x55e730, timeout=Variable "timeout" is not available. ) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003e4336d826 in qpid::sys::Poller::run (this=0x55e730) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003e4336197a in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f30d06317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x0000003f300c9fc3 in clone () from /lib64/tls/libc.so.6 Thread 3 (process 14515): #0 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 #1 0x0000003e4336c9c9 in qpid::sys::Poller::wait (this=0x55e730, timeout=Variable "timeout" is not available. ) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003e4336d826 in qpid::sys::Poller::run (this=0x55e730) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003e4336197a in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f30d06317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x0000003f300c9fc3 in clone () from /lib64/tls/libc.so.6 Thread 2 (process 14517): #0 0x0000003f300ca39c in epoll_wait () from /lib64/tls/libc.so.6 #1 0x0000003e4336c9c9 in qpid::sys::Poller::wait (this=0x55e730, timeout=Variable "timeout" is not available. ) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003e4336d826 in qpid::sys::Poller::run (this=0x55e730) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003e4336197a in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f30d06317 in start_thread () from /lib64/tls/libpthread.so.0 #5 0x0000003f300c9fc3 in clone () from /lib64/tls/libc.so.6 Thread 1 (process 14511): #0 0x0000003e436b853a in qpid::client::ConnectionImpl::release (this=0x96d720) at /usr/include/boost/scoped_ptr.hpp:93 #1 0x0000003e436c5aed in boost::detail::sp_counted_impl_pd<qpid::client::ConnectionImpl*, boost::_bi::bind_t<void, boost::_mfi::mf0<void, qpid::client::ConnectionImpl>, boost::_bi::list1<boost::arg<1> > > >::dispose (this=Variable "this" is not available. ) at /usr/include/boost/bind/mem_fn_template.hpp:45 #2 0x0000003e436aa4bc in ~Connection (this=0x7fbffff650) at ./boost/detail/sp_counted_base_gcc_x86.hpp:145 #3 0x000000000040d26d in ~Client (this=0x7fbffff640) at ../../include/qpid/client/AsyncSession_0_10.h:553 #4 0x0000000000409d56 in main (argc=Variable "argc" is not available. ) at perftest.cpp:719 #5 0x0000003f3001c4cb in __libc_start_main () from /lib64/tls/libc.so.6 #6 0x0000000000408a5a in _start () #7 0x0000007fbffff878 in ?? () #8 0x000000000000001c in ?? () #9 0x000000000000000f in ?? () #10 0x0000007fbffffaea in ?? () #11 0x0000007fbffffaf3 in ?? () #12 0x0000007fbffffb01 in ?? () #13 0x0000007fbffffb05 in ?? () #14 0x0000007fbffffb1f in ?? () #15 0x0000007fbffffb22 in ?? () #16 0x0000007fbffffb2e in ?? () #17 0x0000007fbffffb34 in ?? () #18 0x0000007fbffffb3b in ?? () #19 0x0000007fbffffb41 in ?? () #20 0x0000007fbffffb4c in ?? () #21 0x0000007fbffffb52 in ?? () #22 0x0000007fbffffb5a in ?? () #23 0x0000007fbffffb60 in ?? () #24 0x0000007fbffffb6a in ?? () #25 0x0000000000000000 in ?? ()
I'm having a hard time reproducing this issue. The reproducer has been running on mrg9 and mrg11 for some time now (3days and 9days respectively).
*** Bug 590595 has been marked as a duplicate of this bug. ***
I believe this is a result of a race between ConnectionImpl::shutdown() and ConnectionImpl::release(). From a ptolemy failure: ==1516== Invalid read of size 4 ==1516== at 0xC30ABA: pthread_mutex_lock (in /lib/libpthread-2.5.so) ==1516== by 0x4876535: pthread_mutex_lock (in /lib/libc-2.5.so) ==1516== by 0x450395A: qpid::client::TCPConnector::close() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x44B64EA: qpid::client::ConnectionImpl::release() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x44BCDA9: boost::detail::sp_counted_impl_pd<qpid::client::ConnectionImpl*, boost::_bi::bind_t<void, boost::_mfi::mf0<void, qpid::client::ConnectionImpl>, boost::_bi::list1<boost::arg<1> > > >::dispose() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x44A963A: qpid::client::Connection::~Connection() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x80EB584: qpid::tests::SessionFixtureT<qpid::tests::LocalConnection, qpid::client::Session_0_10>::~SessionFixtureT() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/tests/.libs/lt-unit_test) ==1516== by 0x80E83A8: qpid::tests::testWaitTillComplete() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/tests/.libs/lt-unit_test) ==1516== by 0x8083D4B: boost::unit_test::ut_detail::callback0_impl_t<boost::unit_test::ut_detail::unused, void (*)()>::invoke() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/tests/.libs/lt-unit_test) ==1516== by 0x403948C: (within /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x4029F34: boost::execution_monitor::catch_signals(boost::unit_test::callback0<int> const&, bool, int) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x402A2C5: boost::execution_monitor::execute(boost::unit_test::callback0<int> const&, bool, int) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x4039598: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::unit_test::test_case const&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x402D193: boost::unit_test::framework_impl::visit(boost::unit_test::test_case const&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x403FEF6: boost::unit_test::traverse_test_tree(boost::unit_test::test_case const&, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x404069F: boost::unit_test::traverse_test_tree(unsigned long, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x40404D7: boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x40406D4: boost::unit_test::traverse_test_tree(unsigned long, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x40404D7: boost::unit_test::traverse_test_tree(boost::unit_test::test_suite const&, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x40406D4: boost::unit_test::traverse_test_tree(unsigned long, boost::unit_test::test_tree_visitor&) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x402C168: boost::unit_test::framework::run(unsigned long, bool) (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== by 0x4039248: main (in /usr/lib/libboost_unit_test_framework.so.1.33.1) ==1516== Address 0x7D33E48 is 24 bytes inside a block of size 152 free'd ==1516== at 0x4004CF1: operator delete(void*) (vg_replace_malloc.c:244) ==1516== by 0x45047D3: qpid::client::TCPConnector::~TCPConnector() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x44BA246: qpid::client::ConnectionImpl::~ConnectionImpl() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x44B65CC: qpid::client::ConnectionImpl::shutdown() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x4503038: qpid::client::TCPConnector::socketClosed(qpid::sys::AsynchIO&, qpid::sys::Socket const&) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x45081A3: boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::client::TCPConnector, qpid::sys::AsynchIO&, qpid::sys::Socket const&>, boost::_bi::list3<boost::_bi::value<qpid::client::TCPConnector*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::Socket const&>::invoke(boost::detail::function::any_pointer, qpid::sys::AsynchIO&, qpid::sys::Socket const&) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidclient.so.2.0.0) ==1516== by 0x460FA3D: boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::Socket const&, std::allocator<boost::function_base> >::operator()(qpid::sys::AsynchIO&, qpid::sys::Socket const&) const (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x460D162: qpid::sys::posix::AsynchIO::close(qpid::sys::DispatchHandle&) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x460D401: qpid::sys::posix::AsynchIO::writeable(qpid::sys::DispatchHandle&) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x460E86C: boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke(boost::detail::function::any_pointer, qpid::sys::DispatchHandle&) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x47000D6: boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator()(qpid::sys::DispatchHandle&) const (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x46FEAE7: qpid::sys::DispatchHandle::processEvent(qpid::sys::Poller::EventType) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x461B69F: qpid::sys::Poller::run() (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0x4612780: qpid::sys::(anonymous namespace)::runRunnable(void*) (in /var/lib/ptolemy/sources/qpid/cpp/qpidc-0.7/_build/src/.libs/libqpidcommon.so.2.0.0) ==1516== by 0xC2E73A: start_thread (in /lib/libpthread-2.5.so) ==1516== by 0x4869CFD: clone (in /lib/libc-2.5.so)
I have checked in a fix for a race that would lead to these symptoms as r946106. I believe this fixes the issue, but as reproducing is very difficult and time consuming it is hard to verify. I'll mark this modified and we can re-open if it re-occurs.
The issue has been fixed, tested on rhel4.8 / 5.5 i 386 / x86_64 on packages: python-qmf-0.7.946106-3.el5 python-qpid-0.7.946106-1.el5 qmf-0.7.946106-1.el5 qmf-devel-0.7.946106-1.el5 qpid-cpp-client-0.7.946106-1.el5 qpid-cpp-client-devel-0.7.946106-1.el5 qpid-cpp-client-devel-docs-0.7.946106-1.el5 qpid-cpp-client-ssl-0.7.946106-1.el5 qpid-cpp-mrg-debuginfo-0.7.946106-1.el5 qpid-cpp-server-0.7.946106-1.el5 qpid-cpp-server-cluster-0.7.946106-1.el5 qpid-cpp-server-devel-0.7.946106-1.el5 qpid-cpp-server-ssl-0.7.946106-1.el5 qpid-cpp-server-store-0.7.946106-1.el5 qpid-cpp-server-xml-0.7.946106-1.el5 qpid-java-client-0.7.946106-3.el5 qpid-java-common-0.7.946106-3.el5 qpid-tests-0.7.946106-1.el5 qpid-tools-0.7.946106-4.el5 -> VERIFIED