Bug 602198

Summary: qpidd crashes when testing heartbeats
Product: Red Hat Enterprise MRG Reporter: Gordon Sim <gsim>
Component: qpid-cppAssignee: Alan Conway <aconway>
Status: CLOSED ERRATA QA Contact: ppecka <ppecka>
Severity: urgent Docs Contact:
Priority: urgent    
Version: DevelopmentCC: aconway, freznice, ppecka
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gordon Sim 2010-06-09 11:50:17 UTC
Description of problem:

When testing for bug 591291 (using r952694 on qpid trunk) qpidd segfaulted. 

Version-Release number of selected component (if applicable):

r952694 from qpid trunk

How reproducible:

Quite rare

Steps to Reproduce:

See bug 591291 - need to run for many hours before seeing crash

Additional info:

Core was generated by `/home/gordon/qpid/cpp/build/src/.libs/lt-qpidd --load-module /home/gordon/qpid/'.
Program terminated with signal 11, Segmentation fault.
[New process 9483]
[New process 9516]
[New process 9490]
[New process 9489]
[New process 9488]
[New process 9487]
[New process 9486]
[New process 9484]
#0  0x00002aaefd5e999d in qpid::sys::AsynchIOHandler::disconnect (this=0x1a4f0880) at ../../src/qpid/sys/AsynchIOHandler.cpp:194
194	    if (codec) codec->closed();
(gdb) bt
#0  0x00002aaefd5e999d in qpid::sys::AsynchIOHandler::disconnect (this=0x1a4f0880) at ../../src/qpid/sys/AsynchIOHandler.cpp:194
#1  0x00002aaefd5e9c29 in qpid::sys::AsynchIOHandler::eof (this=0x2aaab1b330c0, a=@0x1a8778a0) at ../../src/qpid/sys/AsynchIOHandler.cpp:177
#2  0x00002aaefd51ccbf in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x2aaefd5e9c20, 
    a0=@0x1a8778a0) at /usr/include/boost/function/function_template.hpp:576
#3  0x00002aaefd51c873 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > >, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::_bi::value<boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > > > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:149
#4  0x00002aaefd5ee49f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2aaefd5e9c20, 
    a0=@0x1a8778a0) at /usr/include/boost/function/function_template.hpp:576
#5  0x00002aaefd5ed2b6 in qpid::sys::DispatchHandle::processEvent (this=0x1a8778a8, type=qpid::sys::Poller::READ_WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:309
#6  0x00002aaefd526e98 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/Poller.h:125
#7  0x00002aaefd0844e4 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:339
#8  0x0000000000409014 in QpiddDaemon::child (this=0x7fff989850d0) at ../../src/posix/QpiddBroker.cpp:130
#9  0x00002aaefd0a482f in qpid::broker::Daemon::fork (this=0x7fff989850d0) at ../../src/qpid/broker/Daemon.cpp:91
#10 0x0000000000407585 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at ../../src/posix/QpiddBroker.cpp:168
#11 0x00000000004057ff in main (argc=12, argv=0x7fff989856b8) at ../../src/qpidd.cpp:80
(gdb) print codec
$1 = (class qpid::sys::ConnectionCodec *) 0x2aaab1b330c0
(gdb) thread apply all bt

Thread 8 (process 9484):
#0  0x00000038f9c92149 in strftime_l () from /lib64/libc.so.6
#1  0x00002aaefd51f111 in qpid::sys::outputFormattedNow (o=@0x413214b0) at ../../src/qpid/sys/posix/Time.cpp:89
#2  0x00002aaefd5da5a1 in qpid::log::Logger::log (this=0x2aaefd895680, s=@0x2aaefd894d80, msg=@0x41321770) at ../../src/qpid/log/Logger.cpp:77
#3  0x00002aaefd5dfa9c in qpid::log::Statement::log (this=0x2aaefd894d80, message=@0x41321f90) at ../../src/qpid/log/Statement.cpp:57
#4  0x00002aaefd5f3b4b in qpid::sys::Timer::run (this=0x19b59b40) at ../../src/qpid/sys/Timer.cpp:129
#5  0x00002aaefd51f14a in runRunnable (p=0x413213f0) at ../../src/qpid/sys/posix/Thread.cpp:35
#6  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#7  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 7 (process 9486):
#0  0x00000038fa40ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002aaefd5f3b73 in qpid::sys::Timer::run (this=0x19b5f490) at ../../include/qpid/sys/posix/Condition.h:63
#2  0x00002aaefd51f14a in runRunnable (p=0x19b5f4c4) at ../../src/qpid/sys/posix/Thread.cpp:35
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 6 (process 9487):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaefd526343 in qpid::sys::Poller::wait (this=0x19b4ee00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002aaefd526ea7 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002aaefd51f14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 5 (process 9488):
#0  0x00000038f9cd3f9a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002aaefd525fd7 in qpid::sys::PollerPrivate::resetMode (this=0x19b593a0, eh=@0x2aaab4090920) at ../../src/qpid/sys/epoll/EpollPoller.cpp:389
#2  0x00002aaefd52624d in qpid::sys::Poller::wait (this=0x19b4ee00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:558
#3  0x00002aaefd526ea7 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#4  0x00002aaefd51f14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#5  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#6  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 9489):
#0  0x00000038f9cd447b in accept () from /lib64/libc.so.6
#1  0x00002aaefd511ca4 in qpid::sys::Socket::accept (this=<value optimized out>) at ../../src/qpid/sys/posix/Socket.cpp:215
#2  0x00002aaefd51a085 in qpid::sys::posix::AsynchAcceptor::readable (this=0x19b634b0, h=@0x19b634d0) at ../../src/qpid/sys/posix/AsynchIO.cpp:121
#3  0x00002aaefd5ee49f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x157, a0=@0x0)
    at /usr/include/boost/function/function_template.hpp:576
---Type <return> to continue, or q <return> to quit---
#4  0x00002aaefd5ed1f9 in qpid::sys::DispatchHandle::processEvent (this=0x19b634d0, type=qpid::sys::Poller::READABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:278
#5  0x00002aaefd526e98 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/Poller.h:125
#6  0x00002aaefd51f14a in runRunnable (p=0x13) at ../../src/qpid/sys/posix/Thread.cpp:35
#7  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#8  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 9490):
#0  0x00000038f9cd3f9a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002aaefd525fd7 in qpid::sys::PollerPrivate::resetMode (this=0x19b593a0, eh=@0x2aaab40dc910) at ../../src/qpid/sys/epoll/EpollPoller.cpp:389
#2  0x00002aaefd52624d in qpid::sys::Poller::wait (this=0x19b4ee00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:558
#3  0x00002aaefd526ea7 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#4  0x00002aaefd51f14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#5  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#6  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 9516):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002aaefd526343 in qpid::sys::Poller::wait (this=0x19b65e60, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002aaefd526ea7 in qpid::sys::Poller::run (this=0x19b65e60) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002aaefd51f14a in runRunnable (p=0x14) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 9483):
#0  0x00002aaefd5e999d in qpid::sys::AsynchIOHandler::disconnect (this=0x1a4f0880) at ../../src/qpid/sys/AsynchIOHandler.cpp:194
#1  0x00002aaefd5e9c29 in qpid::sys::AsynchIOHandler::eof (this=0x2aaab1b330c0, a=@0x1a8778a0) at ../../src/qpid/sys/AsynchIOHandler.cpp:177
#2  0x00002aaefd51ccbf in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x2aaefd5e9c20, 
    a0=@0x1a8778a0) at /usr/include/boost/function/function_template.hpp:576
#3  0x00002aaefd51c873 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > >, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::_bi::value<boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > > > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:149
#4  0x00002aaefd5ee49f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2aaefd5e9c20, 
    a0=@0x1a8778a0) at /usr/include/boost/function/function_template.hpp:576
#5  0x00002aaefd5ed2b6 in qpid::sys::DispatchHandle::processEvent (this=0x1a8778a8, type=qpid::sys::Poller::READ_WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:309
#6  0x00002aaefd526e98 in qpid::sys::Poller::run (this=0x19b4ee00) at ../../src/qpid/sys/Poller.h:125
#7  0x00002aaefd0844e4 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:339
#8  0x0000000000409014 in QpiddDaemon::child (this=0x7fff989850d0) at ../../src/posix/QpiddBroker.cpp:130
#9  0x00002aaefd0a482f in qpid::broker::Daemon::fork (this=0x7fff989850d0) at ../../src/qpid/broker/Daemon.cpp:91
---Type <return> to continue, or q <return> to quit---
#10 0x0000000000407585 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at ../../src/posix/QpiddBroker.cpp:168
#11 0x00000000004057ff in main (argc=12, argv=0x7fff989856b8) at ../../src/qpidd.cpp:80
(gdb)

Comment 1 Gordon Sim 2010-06-09 11:51:49 UTC
Another core from same test:

Core was generated by `/home/gordon/qpid/cpp/build/src/.libs/lt-qpidd --load-module /home/gordon/qpid/'.
Program terminated with signal 6, Aborted.
[New process 17767]
[New process 17795]
[New process 17769]
[New process 17768]
[New process 17766]
[New process 17765]
[New process 17763]
[New process 17762]
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038f9c296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00002b2a4e697466 in qpid::sys::DispatchHandle::processEvent (this=0x2aaab436aab8, type=qpid::sys::Poller::INTERRUPTED)
    at ../../src/qpid/sys/DispatchHandle.cpp:295
#4  0x00002b2a4e5d0e98 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/Poller.h:125
#5  0x00002b2a4e5c914a in runRunnable (p=0x4562) at ../../src/qpid/sys/posix/Thread.cpp:35
#6  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#7  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 8 (process 17762):
#0  0x00000038f9c729b5 in _int_malloc () from /lib64/libc.so.6
#1  0x00000038f9c74bee in malloc () from /lib64/libc.so.6
#2  0x00000038fccbd17d in operator new () from /usr/lib64/libstdc++.so.6
#3  0x00002b2a4e5d3ecf in qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::markForDeletion (handle=0x42a8610)
    at /usr/include/boost/detail/shared_count.hpp:82
#4  0x00002b2a4e5c0ba7 in ~AsynchIO (this=0x3c79880) at ../../src/qpid/sys/posix/AsynchIO.cpp:299
#5  0x00002b2a4e6971e1 in qpid::sys::DispatchHandle::processEvent (this=0x3c79888, type=qpid::sys::Poller::WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:334
#6  0x00002b2a4e5d0e98 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/Poller.h:125
#7  0x00002b2a4e12e4e4 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:339
#8  0x0000000000409014 in QpiddDaemon::child (this=0x7fffd0091e40) at ../../src/posix/QpiddBroker.cpp:130
#9  0x00002b2a4e14e82f in qpid::broker::Daemon::fork (this=0x7fffd0091e40) at ../../src/qpid/broker/Daemon.cpp:91
#10 0x0000000000407585 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at ../../src/posix/QpiddBroker.cpp:168
#11 0x00000000004057ff in main (argc=12, argv=0x7fffd0092428) at ../../src/qpidd.cpp:80

Thread 7 (process 17763):
#0  0x00000038f9cd48c1 in send () from /lib64/libc.so.6
#1  0x00000038f9cd01c0 in __vsyslog_chk () from /lib64/libc.so.6
#2  0x00000038f9cd0580 in syslog () from /lib64/libc.so.6
#3  0x00002b2a4e686010 in std::for_each<boost::void_ptr_iterator<__gnu_cxx::__normal_iterator<void**, std::vector<void*, std::allocator<void*> > >, qpid::log::Logger::Output>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list3<boost::arg<1>, boost::_bi::value<qpid::log::Statement>, boost::_bi::value<std::string> > > > (__first=<value optimized out>, __last=
        {iter_ = {_M_current = 0x2187ac8}}, __f=
              {f_ = {f_ = 0x40e8a490, this adjustment 1088987712}, l_ = {a1_ = {<No data fields>}, a2_ = {t_ = {enabled = false, file = 0x4000 <Address 0x4000 out of bounds>, line = 582, function = 0x40e8a4d0 "h�ƴ�*", level = 1088988328}}, a3_ = {t_ = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x2187ac8 "LOG_LOCAp\a"}}}}})
    at /usr/include/boost/bind/mem_fn_template.hpp:262
#4  0x00002b2a4e684410 in qpid::log::Logger::log (this=0x2b2a4e93f680, s=@0x2b2a4e499300, msg=@0x40e8a5a0) at ../../src/qpid/log/Logger.cpp:95
#5  0x00002b2a4e689a9c in qpid::log::Statement::log (this=0x2b2a4e499300, message=@0x40e8a770) at ../../src/qpid/log/Statement.cpp:57
#6  0x00002b2a4e148037 in qpid::broker::ConnectionTimeoutTask::fire (this=0x2aaab49da9e0) at ../../src/qpid/broker/Connection.cpp:73
#7  0x00002b2a4e69d9d5 in qpid::sys::Timer::run (this=0x219fb40) at ../../src/qpid/sys/Timer.cpp:119
#8  0x00002b2a4e5c914a in runRunnable (p=0xb) at ../../src/qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 6 (process 17765):
#0  0x00000038fa40ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b2a4e69db73 in qpid::sys::Timer::run (this=0x21a5490) at ../../include/qpid/sys/posix/Condition.h:63
#2  0x00002b2a4e5c914a in runRunnable (p=0x21a54c4) at ../../src/qpid/sys/posix/Thread.cpp:35
---Type <return> to continue, or q <return> to quit---
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 5 (process 17766):
#0  0x00000038f9cd46b1 in recv () from /lib64/libc.so.6
#1  0x00000038fa801ca7 in openais_dispatch_recv (ipc_ctx=<value optimized out>, data=<value optimized out>, timeout=<value optimized out>)
    at /usr/include/bits/socket2.h:35
#2  0x00000038fa802b8a in cpg_dispatch (handle=<value optimized out>, dispatch_types=<value optimized out>) at cpg.c:266
#3  0x00002b2a4e9db524 in qpid::cluster::Cpg::dispatchAll (this=0x21a21e0) at ../../src/qpid/cluster/Cpg.cpp:194
#4  0x00002b2a4e9feaf3 in qpid::cluster::PollerDispatch::dispatch (this=0x21a2500, h=@0x425526df) at ../../src/qpid/cluster/PollerDispatch.cpp:53
#5  0x00002b2a4e69849f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x1, 
    a0=@0x425526df) at /usr/include/boost/function/function_template.hpp:576
#6  0x00002b2a4e6971f9 in qpid::sys::DispatchHandle::processEvent (this=0x21a3b40, type=qpid::sys::Poller::READABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:278
#7  0x00002b2a4e5d0e98 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/Poller.h:125
#8  0x00002b2a4e5c914a in runRunnable (p=0xc) at ../../src/qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 17768):
#0  0x00000038f9cd3f9a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b2a4e5cffd7 in qpid::sys::PollerPrivate::resetMode (this=0x219f3a0, eh=@0x2aaab6193320) at ../../src/qpid/sys/epoll/EpollPoller.cpp:389
#2  0x00002b2a4e5d024d in qpid::sys::Poller::wait (this=0x2194e00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:558
#3  0x00002b2a4e5d0ea7 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#4  0x00002b2a4e5c914a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#5  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#6  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 17769):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b2a4e5d0343 in qpid::sys::Poller::wait (this=0x2194e00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002b2a4e5d0ea7 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002b2a4e5c914a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 17795):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b2a4e5d0343 in qpid::sys::Poller::wait (this=0x21abf10, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002b2a4e5d0ea7 in qpid::sys::Poller::run (this=0x21abf10) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002b2a4e5c914a in runRunnable (p=0x14) at ../../src/qpid/sys/posix/Thread.cpp:35
---Type <return> to continue, or q <return> to quit---
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 17767):
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038f9c296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00002b2a4e697466 in qpid::sys::DispatchHandle::processEvent (this=0x2aaab436aab8, type=qpid::sys::Poller::INTERRUPTED)
    at ../../src/qpid/sys/DispatchHandle.cpp:295
#4  0x00002b2a4e5d0e98 in qpid::sys::Poller::run (this=0x2194e00) at ../../src/qpid/sys/Poller.h:125
#5  0x00002b2a4e5c914a in runRunnable (p=0x4562) at ../../src/qpid/sys/posix/Thread.cpp:35
#6  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#7  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb)

Comment 2 Gordon Sim 2010-06-09 13:43:13 UTC
Sorry, typo in description, see bug 591292!

Comment 3 Gordon Sim 2010-06-09 19:05:56 UTC
Another instance:

Core was generated by `/home/gordon/qpid/cpp/build/src/.libs/lt-qpidd --load-module /home/gordon/qpid/'.
Program terminated with signal 11, Segmentation fault.
[New process 13733]
[New process 13762]
[New process 13736]
[New process 13735]
[New process 13734]
[New process 13732]
[New process 13730]
[New process 13729]
#0  0x00002ae7f8dddfb0 in vtable for qpid::sys::OutputTask () from /home/gordon/qpid/cpp/build/src/.libs/libqpidbroker.so.2
(gdb) bt
#0  0x00002ae7f8dddfb0 in vtable for qpid::sys::OutputTask () from /home/gordon/qpid/cpp/build/src/.libs/libqpidbroker.so.2
#1  0x00002ae7f8fe49a0 in qpid::sys::AsynchIOHandler::disconnect (this=0xdcede80) at ../../src/qpid/sys/AsynchIOHandler.cpp:194
#2  0x00002ae7f8fe4c29 in qpid::sys::AsynchIOHandler::eof (this=0xdccf1d0, a=@0xdc8f1b0) at ../../src/qpid/sys/AsynchIOHandler.cpp:177
#3  0x00002ae7f8f17cbf in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xd41c410, 
    a0=@0xdc8f1b0) at /usr/include/boost/function/function_template.hpp:576
#4  0x00002ae7f8f17873 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > >, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::_bi::value<boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > > > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:149
#5  0x00002ae7f8fe949f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xd41c410, 
    a0=@0xdc8f1b0) at /usr/include/boost/function/function_template.hpp:576
#6  0x00002ae7f8fe82b6 in qpid::sys::DispatchHandle::processEvent (this=0xdc8f1b8, type=qpid::sys::Poller::READ_WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:309
#7  0x00002ae7f8f21e98 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/Poller.h:125
#8  0x00002ae7f8f1a14a in runRunnable (p=0xdccf1d0) at ../../src/qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 8 (process 13729):
#0  0x00000038f9cd3f9a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002ae7f8f2356f in qpid::sys::Poller::monitorHandle (this=0xb98ae00, handle=<value optimized out>, dir=qpid::sys::Poller::INOUT)
    at ../../src/qpid/sys/epoll/EpollPoller.cpp:427
#2  0x00002ae7f8fe75ec in qpid::sys::DispatchHandle::startWatch (this=0xd049408, poller0={px = 0x7fffe6bde990, pn = {pi_ = 0x7fffe6bde930}})
    at ../../src/qpid/sys/DispatchHandle.cpp:59
#3  0x00002ae7f8f10498 in qpid::sys::posix::AsynchIO::start (this=<value optimized out>, poller=<value optimized out>)
    at ../../src/qpid/sys/posix/AsynchIO.cpp:306
#4  0x00002ae7f8b77a98 in qpid::sys::AsynchIOProtocolFactory::established (this=<value optimized out>, poller=
        {px = 0x7fffe6bdef00, pn = {pi_ = 0x2ae7f92fdb8a}}, s=@0xbaa2cb0, f=<value optimized out>, isClient=<value optimized out>)
    at ../../src/qpid/sys/TCPIOPlugin.cpp:103
#5  0x00002ae7f8b79329 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf4<void, qpid::sys::AsynchIOProtocolFactory, boost::shared_ptr<qpid::sys::Poller>, qpid::sys::Socket const&, qpid::sys::ConnectionCodec::Factory*, bool>, boost::_bi::list5<boost::_bi::value<qpid::sys::AsynchIOProtocolFactory*>, boost::_bi::value<boost::shared_ptr<qpid::sys::Poller> >, boost::arg<1>, boost::_bi::value<qpid::sys::ConnectionCodec::Factory*>, boost::_bi::value<bool> > >, void, qpid::sys::Socket const&>::invoke (function_obj_ptr=<value optimized out>, a0=@0x206)
    at /usr/include/boost/bind/mem_fn_template.hpp:454
#6  0x00002ae7f8f1813f in boost::function1<void, qpid::sys::Socket const&, std::allocator<boost::function_base> >::operator() (this=0x0, a0=@0x3)
    at /usr/include/boost/function/function_template.hpp:576
#7  0x00002ae7f8f15071 in qpid::sys::posix::AsynchAcceptor::readable (this=0xb99f4b0, h=@0xb99f4d0) at ../../src/qpid/sys/posix/AsynchIO.cpp:123
#8  0x00002ae7f8fe949f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x0, a0=@0x3)
    at /usr/include/boost/function/function_template.hpp:576
#9  0x00002ae7f8fe81f9 in qpid::sys::DispatchHandle::processEvent (this=0xb99f4d0, type=qpid::sys::Poller::READABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:278
#10 0x00002ae7f8f21e98 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/Poller.h:125
#11 0x00002ae7f8a7f4e4 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:339
#12 0x0000000000409014 in QpiddDaemon::child (this=0x7fffe6bdfd60) at ../../src/posix/QpiddBroker.cpp:130
#13 0x00002ae7f8a9f82f in qpid::broker::Daemon::fork (this=0x7fffe6bdfd60) at ../../src/qpid/broker/Daemon.cpp:91
#14 0x0000000000407585 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at ../../src/posix/QpiddBroker.cpp:168
#15 0x00000000004057ff in main (argc=12, argv=0x7fffe6be0348) at ../../src/qpidd.cpp:80

Thread 7 (process 13730):
#0  0x00000038f9cd48c1 in send () from /lib64/libc.so.6
#1  0x00000038f9cd01c0 in __vsyslog_chk () from /lib64/libc.so.6
#2  0x00000038f9cd0580 in syslog () from /lib64/libc.so.6
#3  0x00002ae7f8fd7010 in std::for_each<boost::void_ptr_iterator<__gnu_cxx::__normal_iterator<void**, std::vector<void*, std::allocator<void*> > >, qpid::log::Logger::Output>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list3<boost::arg<1>, boost::_bi::value<qpid::log::Statement>, boost::_bi::value<std::string> > > > (__first=<value optimized out>, __last=
        {iter_ = {_M_current = 0xb97dac8}}, __f=
              {f_ = {f_ = 0x425b8490, this adjustment 1113293376}, l_ = {a1_ = {<No data fields>}, a2_ = {t_ = {enabled = false, file = 0x4000 <Address 0x4000 out of bounds>, line = 582, function = 0x425b84d0 "HEi��*", level = 1113293992}}, a3_ = {t_ = {static npos = 18446744073709551615, _M_dataplus ---Type <return> to continue, or q <return> to quit---
= {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xb97dac8 "LOG_LOCAp\a"}}}}})
    at /usr/include/boost/bind/mem_fn_template.hpp:262
#4  0x00002ae7f8fd5410 in qpid::log::Logger::log (this=0x2ae7f9290680, s=@0x2ae7f8dea300, msg=@0x425b85a0) at ../../src/qpid/log/Logger.cpp:95
#5  0x00002ae7f8fdaa9c in qpid::log::Statement::log (this=0x2ae7f8dea300, message=@0x425b8770) at ../../src/qpid/log/Statement.cpp:57
#6  0x00002ae7f8a99037 in qpid::broker::ConnectionTimeoutTask::fire (this=0xd3aa900) at ../../src/qpid/broker/Connection.cpp:73
#7  0x00002ae7f8fee9d5 in qpid::sys::Timer::run (this=0xb995b40) at ../../src/qpid/sys/Timer.cpp:119
#8  0x00002ae7f8f1a14a in runRunnable (p=0xa) at ../../src/qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 6 (process 13732):
#0  0x00000038fa40ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002ae7f8feeb73 in qpid::sys::Timer::run (this=0xb99b490) at ../../include/qpid/sys/posix/Condition.h:63
#2  0x00002ae7f8f1a14a in runRunnable (p=0xb99b4c4) at ../../src/qpid/sys/posix/Thread.cpp:35
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 5 (process 13734):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002ae7f8f21343 in qpid::sys::Poller::wait (this=0xb98ae00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002ae7f8f21ea7 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002ae7f8f1a14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 13735):
#0  0x00000038f9cd3f9a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002ae7f8f20fd7 in qpid::sys::PollerPrivate::resetMode (this=0xb9953a0, eh=@0x2aaab47973a0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:389
#2  0x00002ae7f8f2124d in qpid::sys::Poller::wait (this=0xb98ae00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:558
#3  0x00002ae7f8f21ea7 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#4  0x00002ae7f8f1a14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#5  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#6  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 13736):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002ae7f8f21343 in qpid::sys::Poller::wait (this=0xb98ae00, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002ae7f8f21ea7 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002ae7f8f1a14a in runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 2 (process 13762):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00002ae7f8f21343 in qpid::sys::Poller::wait (this=0xb9a1e50, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:570
#2  0x00002ae7f8f21ea7 in qpid::sys::Poller::run (this=0xb9a1e50) at ../../src/qpid/sys/epoll/EpollPoller.cpp:517
#3  0x00002ae7f8f1a14a in runRunnable (p=0x14) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 13733):
#0  0x00002ae7f8dddfb0 in vtable for qpid::sys::OutputTask () from /home/gordon/qpid/cpp/build/src/.libs/libqpidbroker.so.2
#1  0x00002ae7f8fe49a0 in qpid::sys::AsynchIOHandler::disconnect (this=0xdcede80) at ../../src/qpid/sys/AsynchIOHandler.cpp:194
#2  0x00002ae7f8fe4c29 in qpid::sys::AsynchIOHandler::eof (this=0xdccf1d0, a=@0xdc8f1b0) at ../../src/qpid/sys/AsynchIOHandler.cpp:177
#3  0x00002ae7f8f17cbf in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xd41c410, 
    a0=@0xdc8f1b0) at /usr/include/boost/function/function_template.hpp:576
#4  0x00002ae7f8f17873 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > >, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::_bi::value<boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> > > > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:149
#5  0x00002ae7f8fe949f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xd41c410, 
    a0=@0xdc8f1b0) at /usr/include/boost/function/function_template.hpp:576
#6  0x00002ae7f8fe82b6 in qpid::sys::DispatchHandle::processEvent (this=0xdc8f1b8, type=qpid::sys::Poller::READ_WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:309
#7  0x00002ae7f8f21e98 in qpid::sys::Poller::run (this=0xb98ae00) at ../../src/qpid/sys/Poller.h:125
#8  0x00002ae7f8f1a14a in runRunnable (p=0xdccf1d0) at ../../src/qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb)

Comment 5 Gordon Sim 2010-06-21 09:07:21 UTC
*** Bug 605656 has been marked as a duplicate of this bug. ***

Comment 6 ppecka 2010-09-15 12:49:28 UTC
running this test 4th day.
VERIFIED on RHEL 5.5 i386 / x86_64
#rpm -qa | grep -P '(qpid|qmf|ais)' | sort -u
openais-0.80.6-16.el5_5.7
openais-devel-0.80.6-16.el5_5.7
python-qmf-0.7.946106-13.el5
python-qpid-0.7.946106-14.el5
qmf-0.7.946106-14.el5
qmf-devel-0.7.946106-14.el5
qpid-cpp-client-0.7.946106-14.el5
qpid-cpp-client-devel-0.7.946106-14.el5
qpid-cpp-client-devel-docs-0.7.946106-14.el5
qpid-cpp-client-ssl-0.7.946106-14.el5
qpid-cpp-mrg-debuginfo-0.7.946106-14.el5
qpid-cpp-server-0.7.946106-14.el5
qpid-cpp-server-0.7.946106-3.el5
qpid-cpp-server-cluster-0.7.946106-14.el5
qpid-cpp-server-devel-0.7.946106-14.el5
qpid-cpp-server-ssl-0.7.946106-14.el5
qpid-cpp-server-store-0.7.946106-14.el5
qpid-cpp-server-xml-0.7.946106-14.el5
qpid-java-client-0.7.946106-9.el5
qpid-java-common-0.7.946106-9.el5
qpid-tools-0.7.946106-10.el5


-> VERIFIED