Bug 669529

Summary: Segmentation Fault on Sender::Send
Product: Red Hat Enterprise MRG Reporter: Chris Pitman <christopher.pitman>
Component: qpid-cppAssignee: Gordon Sim <gsim>
Status: CLOSED INSUFFICIENT_DATA QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 1.3CC: gsim, jross
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-25 02:43:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Chris Pitman 2011-01-13 21:40:55 UTC 
Description of problem:

When attempting to send a message with a qpid::messaging::Sender, the call sometimes triggers a seg fault.  The core dump shows that the Session is in the DETACHING state and that SenderImpl::checkPendingSends attempts to dereference a CompletionImpl which is null.

Following is the back trace with select frames:


#0  isComplete (this=<value optimized out>) at qpid/client/CompletionImpl.h:39
#1  qpid::client::Completion::isComplete (this=<value optimized out>) at qpid/client/Completion.cpp:37
#2  0x0000003e0027a604 in qpid::client::amqp0_10::SenderImpl::checkPendingSends (this=0x2aaab40090b0, flush=128) at qpid/client/amqp0_10/SenderImpl.cpp:155
#3  0x0000003e0027a7e9 in qpid::client::amqp0_10::SenderImpl::waitForCapacity (this=0x2aaab40090b0) at qpid/client/amqp0_10/SenderImpl.cpp:111
#4  0x0000003e0027dbdc in operator() (this=0x2aaab0a42310, f=...) at qpid/client/amqp0_10/SenderImpl.h:108
#5  qpid::client::amqp0_10::SessionImpl::execute<qpid::client::amqp0_10::SenderImpl::Send> (this=0x2aaab0a42310, f=...) at qpid/client/amqp0_10/SessionImpl.h:98
#6  0x0000003e0027c1c1 in qpid::client::amqp0_10::SenderImpl::send (this=0x2aaab40090b0, message=<value optimized out>, sync=true) at qpid/client/amqp0_10/SenderImpl.cpp:44
#7  0x000000000048d191 in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Send (this=0x2aaab0004620, address=..., message=...) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:172
#8  0x000000000043ff04 in ITCM::TransmitSpecialHandling::TransmitMessage (this=0x2aaab00061e0, message=<value optimized out>)
    at /usr/src/debug/itc-connection-manager-0.9/src/amqp/TransmitSpecialHandling.cpp:46
#9  0x000000000049e207 in ITCM::Listener::Process (this=<value optimized out>, message=...) at /usr/src/debug/itc-connection-manager-0.9/src/Listener.cpp:40
#10 0x00000000004878d8 in ITCM::AmqpListener<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread> >::Received (this=0x2aaab00093c0, data=...) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpListener-inl.h:54
#11 0x000000352505601c in ITCM::Common::SubscriptionManagerImpl<qpid::messaging::Connection, qpid::messaging::Session, qpid::messaging::Receiver>::Run (this=0x2aaab400b2f0)
    at /usr/src/debug/itc-common-library-0.9/src/Subscription/SubscriptionManagerImpl.h:238
#12 0x00000000004688cc in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Run (this=0x2aaab0001330) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:264
#13 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0x2aaab4210020) at /usr/include/boost/function/function_template.hpp:576
#14 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0001300) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#15 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0x2aaab4210020) at /usr/include/boost/function/function_template.hpp:576
#16 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#17 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#18 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6

===============================================================================

(gdb) frame 1
#1  qpid::client::Completion::isComplete (this=<value optimized out>) at qpid/client/Completion.cpp:37
37      bool Completion::isComplete() { return impl->isComplete(); }
(gdb) p this->impl
Cannot access memory at address 0x0

===============================================================================

(gdb) frame
#5  qpid::client::amqp0_10::SessionImpl::execute<qpid::client::amqp0_10::SenderImpl::Send> (this=0x2aaab0a42310, f=...) at qpid/client/amqp0_10/SessionImpl.h:98
98                  f();
(gdb) list
93          void setSession(qpid::client::Session);
94
95          template <class T> bool execute(T& f)
96          {
97              try {
98                  f();
99                  return true;
100             } catch (const qpid::TransportFailure&) {
101                 reconnect();
102                 return false;
(gdb) p this->session.impl->px->state.state
$7 = DETACHING


How reproducible:
Intermittent error which is not reproduced in a concise test case

Additional info:
Comment 1 Gordon Sim 2011-01-14 15:10:19 UTC 
Can you paste in the backtraces of any other threads (`thread apply all bt` to gdb)?
Comment 2 Chris Pitman 2011-01-14 16:09:28 UTC 
Here are traces for all remaining threads:


(gdb) thread 1
[Switching to thread 1 (Thread 17976)]#0  isComplete (this=<value optimized out>) at qpid/client/CompletionImpl.h:39
39          bool isComplete() { return future.isComplete(*session); }
(gdb) bt
#0  isComplete (this=<value optimized out>) at qpid/client/CompletionImpl.h:39
#1  qpid::client::Completion::isComplete (this=<value optimized out>) at qpid/client/Completion.cpp:37
#2  0x0000003e0027a604 in qpid::client::amqp0_10::SenderImpl::checkPendingSends (this=0x2aaab40090b0, flush=128) at qpid/client/amqp0_10/SenderImpl.cpp:155
#3  0x0000003e0027a7e9 in qpid::client::amqp0_10::SenderImpl::waitForCapacity (this=0x2aaab40090b0) at qpid/client/amqp0_10/SenderImpl.cpp:111
#4  0x0000003e0027dbdc in operator() (this=0x2aaab0a42310, f=...) at qpid/client/amqp0_10/SenderImpl.h:108
#5  qpid::client::amqp0_10::SessionImpl::execute<qpid::client::amqp0_10::SenderImpl::Send> (this=0x2aaab0a42310, f=...) at qpid/client/amqp0_10/SessionImpl.h:98
#6  0x0000003e0027c1c1 in qpid::client::amqp0_10::SenderImpl::send (this=0x2aaab40090b0, message=<value optimized out>, sync=true) at qpid/client/amqp0_10/SenderImpl.cpp:44
#7  0x000000000048d191 in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Send (this=0x2aaab0004620, address=..., message=...) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:172
#8  0x000000000043ff04 in ITCM::TransmitSpecialHandling::TransmitMessage (this=0x2aaab00061e0, message=<value optimized out>)
    at /usr/src/debug/itc-connection-manager-0.9/src/amqp/TransmitSpecialHandling.cpp:46
#9  0x000000000049e207 in ITCM::Listener::Process (this=<value optimized out>, message=...) at /usr/src/debug/itc-connection-manager-0.9/src/Listener.cpp:40
#10 0x00000000004878d8 in ITCM::AmqpListener<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread> >::Received (this=0x2aaab00093c0, data=...) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpListener-inl.h:54
#11 0x000000352505601c in ITCM::Common::SubscriptionManagerImpl<qpid::messaging::Connection, qpid::messaging::Session, qpid::messaging::Receiver>::Run (this=0x2aaab400b2f0)
    at /usr/src/debug/itc-common-library-0.9/src/Subscription/SubscriptionManagerImpl.h:238
#12 0x00000000004688cc in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Run (this=0x2aaab0001330) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:264
#13 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0x2aaab4210020) at /usr/include/boost/function/function_template.hpp:576
#14 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0001300) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#15 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0x2aaab4210020) at /usr/include/boost/function/function_template.hpp:576
#16 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#17 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#18 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 2
[Switching to thread 2 (Thread 18017)]#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003e00263029 in wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/Monitor.h:45
#3  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/Waitable.h:94
#4  qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/BlockingQueue.h:58
#5  0x0000003e0026039a in qpid::client::amqp0_10::IncomingMessages::wait (this=0x4de6130, duration=<value optimized out>) at qpid/client/amqp0_10/IncomingMessages.cpp:222
#6  0x0000003e00262403 in qpid::client::amqp0_10::IncomingMessages::getNextDestination (this=0x4de6130, destination=..., timeout=...) at qpid/client/amqp0_10/IncomingMessages.cpp:135
#7  0x0000003e00270491 in qpid::client::amqp0_10::SessionImpl::nextReceiver (this=0x4de60e0, receiver=..., timeout=<value optimized out>) at qpid/client/amqp0_10/SessionImpl.cpp:338
#8  0x0000003e0023fc5d in qpid::messaging::Session::nextReceiver (this=<value optimized out>, receiver=..., timeout=...) at qpid/messaging/Session.cpp:72
#9  0x0000003525055d4a in ITCM::Common::SubscriptionManagerImpl<qpid::messaging::Connection, qpid::messaging::Session, qpid::messaging::Receiver>::Run (this=0x4de5cb0)
    at /usr/src/debug/itc-common-library-0.9/src/Subscription/SubscriptionManagerImpl.h:217
#10 0x00000000004688cc in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Run (this=0x2aaab0216dc0) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:264
#11 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#12 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0215790) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#13 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#14 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#15 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 3
[Switching to thread 3 (Thread 17946)]#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003dffdfbc6f in qpid::sys::Timer::run (this=0x3dffae0e60) at ../include/qpid/sys/posix/Condition.h:69
#2  0x0000003dffd2195a in qpid::sys::(anonymous namespace)::runRunnable (p=0x3dffae0e94) at qpid/sys/posix/Thread.cpp:35
#3  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#4  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 4
[Switching to thread 4 (Thread 17163)]#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003e00263029 in wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/Monitor.h:45
#3  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/Waitable.h:94
#4  qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/BlockingQueue.h:58
#5  0x0000003e0026039a in qpid::client::amqp0_10::IncomingMessages::wait (this=0x4deb530, duration=<value optimized out>) at qpid/client/amqp0_10/IncomingMessages.cpp:222
#6  0x0000003e00262403 in qpid::client::amqp0_10::IncomingMessages::getNextDestination (this=0x4deb530, destination=..., timeout=...) at qpid/client/amqp0_10/IncomingMessages.cpp:135
#7  0x0000003e00270491 in qpid::client::amqp0_10::SessionImpl::nextReceiver (this=0x4deb4e0, receiver=..., timeout=<value optimized out>) at qpid/client/amqp0_10/SessionImpl.cpp:338
#8  0x0000003e0023fc5d in qpid::messaging::Session::nextReceiver (this=<value optimized out>, receiver=..., timeout=...) at qpid/messaging/Session.cpp:72
#9  0x0000003525055d4a in ITCM::Common::SubscriptionManagerImpl<qpid::messaging::Connection, qpid::messaging::Session, qpid::messaging::Receiver>::Run (this=0x4de8ec0)
    at /usr/src/debug/itc-common-library-0.9/src/Subscription/SubscriptionManagerImpl.h:217
#10 0x00000000004688cc in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Run (this=0x2aaab062a800) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:264
#11 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#12 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab062a5a0) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#13 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#14 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#15 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 5
[Switching to thread 5 (Thread 17107)]#0  0x0000003dfbccb375 in ppoll () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003dfbccb375 in ppoll () from /lib64/libc.so.6
#1  0x0000003525c66558 in ClassD::IOMultiplexerImpl::Poll (this=0x4de4600, timeout=...) at /usr/src/debug/itc-class-d-0.9/src/networkio/IOMultiplexerImpl.cpp:282
#2  0x0000003525c70d33 in ClassD::IOMultiplexerImpl::Run (this=0x4de4600) at /usr/src/debug/itc-class-d-0.9/src/networkio/IOMultiplexerImpl.cpp:227
#3  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfe) at /usr/include/boost/function/function_template.hpp:576
#4  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#5  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 6
[Switching to thread 6 (Thread 17105)]#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000003e00a072e2 in boost::thread::sleep(boost::xtime const&) () from /usr/lib64/libboost_thread.so.2
#2  0x00000000004a1f0b in ITCM::Thread::Sleep (timeToSleep=1) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:86
#3  0x0000000000486554 in ITCM::HrxTransportAdapter<ITCM::HrxWorkerAdapter>::Run (this=0x2aaab0837890) at /usr/src/debug/itc-connection-manager-0.9/src/hrx/HrxTransportAdapter-inl.h:157
#4  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#5  0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab08379a8) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#6  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#7  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#8  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#9  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 7
[Switching to thread 7 (Thread 17097)]#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000003e00a072e2 in boost::thread::sleep(boost::xtime const&) () from /usr/lib64/libboost_thread.so.2
#2  0x00000000004a1f0b in ITCM::Thread::Sleep (timeToSleep=1) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:86
#3  0x000000000048ba2f in ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter>::Execute (this=0x2aaab062d1a0, brokerUrls=..., 
    transportAdapter=..., retryTime=1) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/FailoverManager-inl.h:74
#4  0x00000000004857d6 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::Run (this=0x2aaab062b840) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:213
#5  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#6  0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab062b958) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#7  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#8  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#9  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 8
[Switching to thread 8 (Thread 17074)]#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003e00263029 in wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at ../include/qpid/sys/Monitor.h:45
#3  wait (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/Waitable.h:94
#4  qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0xffffffffffffff92, result=..., timeout=<value optimized out>) at qpid/sys/BlockingQueue.h:58
#5  0x0000003e0026039a in qpid::client::amqp0_10::IncomingMessages::wait (this=0x2aaab4003370, duration=<value optimized out>) at qpid/client/amqp0_10/IncomingMessages.cpp:222
#6  0x0000003e00262403 in qpid::client::amqp0_10::IncomingMessages::getNextDestination (this=0x2aaab4003370, destination=..., timeout=...) at qpid/client/amqp0_10/IncomingMessages.cpp:135
#7  0x0000003e00270491 in qpid::client::amqp0_10::SessionImpl::nextReceiver (this=0x2aaab4003320, receiver=..., timeout=<value optimized out>) at qpid/client/amqp0_10/SessionImpl.cpp:338
#8  0x0000003e0023fc5d in qpid::messaging::Session::nextReceiver (this=<value optimized out>, receiver=..., timeout=...) at qpid/messaging/Session.cpp:72
#9  0x0000003525055d4a in ITCM::Common::SubscriptionManagerImpl<qpid::messaging::Connection, qpid::messaging::Session, qpid::messaging::Receiver>::Run (this=0x2aaab4007740)
    at /usr/src/debug/itc-common-library-0.9/src/Subscription/SubscriptionManagerImpl.h:217
#10 0x00000000004688cc in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::Run (this=0x2aaab041dd20) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:264
#11 0x0000000000486f98 in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::StartSubscription (this=0x2aaab041dd20, listenAddress=..., pListener=..., blockCall=true)
    at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:246
#12 0x00000000004873ab in ITCM::AmqpListener<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread> >::Start (this=<value optimized out>) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpListener-inl.h:32
#13 0x000000000048ecc1 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::PostHandleConnect (this=0x2aaab0423540) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:115
#14 0x00000000004929c7 in ITCM::TransportAdapter::HandleConnect (this=0x2aaab0423540) at /usr/src/debug/itc-connection-manager-0.9/src/TransportAdapter.cpp:125
#15 0x000000000048b9da in ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter>::Execute (this=0x2aaab0420350, brokerUrls=..., 
---Type <return> to continue, or q <return> to quit---
    transportAdapter=..., retryTime=1) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/FailoverManager-inl.h:42
#16 0x00000000004857d6 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::Run (this=0x2aaab0423540) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:213
#17 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#18 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0423658) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#19 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#20 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#21 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#22 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 9
[Switching to thread 9 (Thread 17043)]#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000003e00a072e2 in boost::thread::sleep(boost::xtime const&) () from /usr/lib64/libboost_thread.so.2
#2  0x00000000004a1f0b in ITCM::Thread::Sleep (timeToSleep=1) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:86
#3  0x000000000048ba2f in ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter>::Execute (this=0x2aaab0218d20, brokerUrls=..., 
    transportAdapter=..., retryTime=1) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/FailoverManager-inl.h:74
#4  0x00000000004857d6 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::Run (this=0x2aaab0217140) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:213
#5  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#6  0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0217258) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#7  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#8  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#9  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 10
[Switching to thread 10 (Thread 17017)]#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000003e00a072e2 in boost::thread::sleep(boost::xtime const&) () from /usr/lib64/libboost_thread.so.2
#2  0x00000000004a1f0b in ITCM::Thread::Sleep (timeToSleep=1) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:86
#3  0x000000000048ba2f in ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter>::Execute (this=0x2aaab020dab0, brokerUrls=..., 
    transportAdapter=..., retryTime=1) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/FailoverManager-inl.h:74
#4  0x00000000004857d6 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::Run (this=0x2aaab0211d10) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:213
#5  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#6  0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0211e28) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#7  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffdfc) at /usr/include/boost/function/function_template.hpp:576
#8  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#9  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 11
[Switching to thread 11 (Thread 16998)]#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003dffd2ac71 in qpid::sys::Poller::wait (this=0x47ca3c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x0000003dffd2b707 in qpid::sys::Poller::run (this=0x47ca3c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x0000003dffd2195a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 12
[Switching to thread 12 (Thread 16997)]#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003dffd2ac71 in qpid::sys::Poller::wait (this=0x47ca3c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x0000003dffd2b707 in qpid::sys::Poller::run (this=0x47ca3c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x0000003dffd2195a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 13
[Switching to thread 13 (Thread 16996)]#0  0x0000003dfc40aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003dff89d28a in wait (this=0x2aaab0a42d30) at ../include/qpid/sys/posix/Condition.h:63
#2  wait (this=0x2aaab0a42d30) at ../include/qpid/sys/Monitor.h:41
#3  wait (this=0x2aaab0a42d30) at qpid/sys/Waitable.h:88
#4  waitFor (this=0x2aaab0a42d30) at qpid/sys/StateMonitor.h:63
#5  waitFor (this=0x2aaab0a42d30) at qpid/client/SessionImpl.cpp:768
#6  qpid::client::SessionImpl::close (this=0x2aaab0a42d30) at qpid/client/SessionImpl.cpp:127
#7  0x0000003e0026dc2d in qpid::client::amqp0_10::SessionImpl::close (this=0x2aaab0a42310) at qpid/client/amqp0_10/SessionImpl.cpp:151
#8  0x0000003e00257b14 in qpid::client::amqp0_10::ConnectionImpl::close (this=0x2aaab0a44cc0) at qpid/client/amqp0_10/ConnectionImpl.cpp:158
#9  0x000000000046427c in ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>::CloseConnection (this=<value optimized out>) at /usr/src/debug/itc-connection-manager-0.9/src/broker/BrokerClient-inl.h:83
#10 0x000000000048b9ff in ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter>::Execute (this=0x2aaab0005420, brokerUrls=..., 
    transportAdapter=..., retryTime=1) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/FailoverManager-inl.h:82
#11 0x00000000004857d6 in ITCM::AmqpTransportAdapter<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverManager<ITCM::BrokerClient<qpid::messaging::Connection, ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>, qpid::messaging::Session, qpid::messaging::Receiver, ITCM::Common::SubscriptionManager, ITCM::Thread>, ITCM::FailoverMessageReader, qpid::messaging::Connection, ITCM::TransportAdapter> >::Run (this=0x2aaab0005600) at /usr/src/debug/itc-connection-manager-0.9/src/amqp/AmqpTransportAdapter-inl.h:213
#12 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#13 0x000000000049fd7d in ITCM::Thread::RunWithJoin (this=0x2aaab0005718) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:60
#14 0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#15 0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#16 0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#17 0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 14
[Switching to thread 14 (Thread 16958)]#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003dffd2ac71 in qpid::sys::Poller::wait (this=0x47ca3c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x0000003dffd2b707 in qpid::sys::Poller::run (this=0x47ca3c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x0000003dffd2195a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 15
[Switching to thread 15 (Thread 16942)]#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003dfbcd4358 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003dffd2ac71 in qpid::sys::Poller::wait (this=0x47ca3c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x0000003dffd2b707 in qpid::sys::Poller::run (this=0x47ca3c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x0000003dffd2195a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 16
[Switching to thread 16 (Thread 16931)]#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000003e0220e30b in tbb::internal::concurrent_queue_base_v3::internal_pop(void*) () from /usr/lib64/libtbb.so.2
#2  0x0000003525089401 in pop (this=0x47c1dd0) at /usr/include/tbb/concurrent_queue.h:269
#3  ITCM::Common::TaskPool::ThreadWorker::operator() (this=0x47c1dd0) at /usr/src/debug/itc-common-library-0.9/src/TaskPool.cpp:177
#4  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#5  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#6  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#7  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 17
[Switching to thread 17 (Thread 16930)]#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000003e0220e30b in tbb::internal::concurrent_queue_base_v3::internal_pop(void*) () from /usr/lib64/libtbb.so.2
#2  0x0000003525089401 in pop (this=0x47c5920) at /usr/include/tbb/concurrent_queue.h:269
#3  ITCM::Common::TaskPool::ThreadWorker::operator() (this=0x47c5920) at /usr/src/debug/itc-common-library-0.9/src/TaskPool.cpp:177
#4  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#5  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#6  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#7  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 18
[Switching to thread 18 (Thread 16929)]#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000003e0220e30b in tbb::internal::concurrent_queue_base_v3::internal_pop(void*) () from /usr/lib64/libtbb.so.2
#2  0x0000003525089401 in pop (this=0x47c5900) at /usr/include/tbb/concurrent_queue.h:269
#3  ITCM::Common::TaskPool::ThreadWorker::operator() (this=0x47c5900) at /usr/src/debug/itc-common-library-0.9/src/TaskPool.cpp:177
#4  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#5  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#6  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#7  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 19
[Switching to thread 19 (Thread 16927)]#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40cd01 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000003e0220e30b in tbb::internal::concurrent_queue_base_v3::internal_pop(void*) () from /usr/lib64/libtbb.so.2
#2  0x0000003525089401 in pop (this=0x47d2690) at /usr/include/tbb/concurrent_queue.h:269
#3  ITCM::Common::TaskPool::ThreadWorker::operator() (this=0x47d2690) at /usr/src/debug/itc-common-library-0.9/src/TaskPool.cpp:177
#4  0x00000000004a2ff7 in boost::function0<void, std::allocator<boost::function_base> >::operator() (this=0xfffffffffffffffc) at /usr/include/boost/function/function_template.hpp:576
#5  0x0000003e00a076bf in ?? () from /usr/lib64/libboost_thread.so.2
#6  0x0000003dfc40673d in start_thread () from /lib64/libpthread.so.0
#7  0x0000003dfbcd3f6d in clone () from /lib64/libc.so.6



(gdb) thread 20
[Switching to thread 20 (Thread 16910)]#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003dfc40e1c1 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000003e00a072e2 in boost::thread::sleep(boost::xtime const&) () from /usr/lib64/libboost_thread.so.2
#2  0x00000000004a1f0b in ITCM::Thread::Sleep (timeToSleep=1) at /usr/src/debug/itc-connection-manager-0.9/src/util/Thread.cpp:86
#3  0x0000000000457463 in WaitForStopSignal (transportAdapters=...) at /usr/src/debug/itc-connection-manager-0.9/src/Main.cpp:149
#4  0x000000000045b8bb in Run (vm=...) at /usr/src/debug/itc-connection-manager-0.9/src/Main.cpp:185
#5  0x000000000045bffb in main (argc=<value optimized out>, argv=0x7fff473404b8) at /usr/src/debug/itc-connection-manager-0.9/src/Main.cpp:222
Comment 3 Gordon Sim 2011-03-09 17:28:12 UTC 
In ITCM::BrokerClient<qpid::messaging::Connection,
ITCM::Common::ConnectionBuilder<qpid::messaging::Connection>,
qpid::messaging::Session, qpid::messaging::Receiver,
ITCM::Common::SubscriptionManager, ITCM::Thread>::Send() does the code use an explicit handle to the sender? Is the sender created in that method or does it already exist?
Comment 4 Justin Ross 2012-10-31 17:50:27 UTC 
Chris, do you know if this is still an issue?  If so, see Gordon's question above.  This one's pretty stale now, so I'll close it if I don't get further feedback.
Comment 5 Chris Pitman 2012-11-02 20:04:44 UTC 
This was a one off issue that I have not seen since then. Here is the chunk of code that was involved:

try
{
    m_Session.getSender(address.getName()).send(message, true);
}
catch (const qpid::messaging::KeyError&)
{
    try
    {
        LOG(CreatingQpidEndpointLog, m_identifier, address, CreatingQpidEndpointLog::SENDER);
        m_Session.createSender(address).send(message, true);
    }
    catch (std::exception& e)
    {
        LOG(GeneralDebugLog, e);
        throw;
    }
}

The line numbers are not matching up, so it is one of those two calls to send.