Bug 877081 - Broker crash re-routing messages through a header exchange
Broker crash re-routing messages through a header exchange
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
high Severity unspecified
: 2.3
: ---
Assigned To: Chuck Rolke
Frantisek Reznicek
: Patch
Depends On:
Blocks: 698367
  Show dependency treegraph
 
Reported: 2012-11-15 11:34 EST by Jason Dillaman
Modified: 2013-03-06 13:52 EST (History)
5 users (show)

See Also:
Fixed In Version: qpid-cpp-0.18-11
Doc Type: Bug Fix
Doc Text:
Cause: Improper message header record locking may cause a broker crash. Consequence: When messages are shared among many queues and subsequently rerouted then header processing by one thread may invalidate header processing by another thread. Fix: Properly lock message header records to prevent concurrent changes. Result: Message header records may be processed by any number of queues without causing a broker crash.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-06 13:52:51 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Quick patch to prevent this crash (8.01 KB, patch)
2012-11-20 13:10 EST, Jason Dillaman
no flags Details | Diff

  None (edit)
Description Jason Dillaman 2012-11-15 11:34:29 EST
Description of problem:
When a single message is delivered to multiple queues, a pointer to that original message is stored in each queue.  If those queues are then concurrently re-routed through a header exchange (i.e. delete the queues, each with an alt-exchange), there is a risk of a broker crash because the application headers on the shared copy of the message are being manipulated concurrently in multiple threads in order to clear the trace id.  

Version-Release number of selected component (if applicable):
Qpid 0.18-9

How reproducible:
Occasionally -- race condition

Steps to Reproduce:
1. Declare multiple queues, each with an alt-exchange through a header exchange
2. Bind the queues to an exchange such that a single message is delivered to all the queues
3. Concurrently re-route the queues through the header exchange (i.e. delete the queues or issue QMF re-route commands)
  
Actual results:
Thread A is attempting to evaluate the application headers on message 1 during a re-route from Queue A.  Thread B is concurrently clearing the trace id on message 1 while re-routing from Queue B.  Therefore, Thread B invalidated the application headers while Thread A was attempting to access them.

Expected results:
Since a single message can be delivered to multiple queues, individual queues should not manipulate shared message properties.  

Additional info:
Comment 1 Jason Dillaman 2012-11-15 11:48:13 EST
Stack Backtrace:

#0  0x0000000100000001 in ?? ()
#1  0x00000034db2bb3ed in __dynamic_cast () from /usr/lib64/libstdc++.so.6
#2  0x00007f537d197d41 in qpid::framing::FixedWidthValue<2>::operator== (this=0x7f53385c4060, d=<value optimized out>) at ../include/qpid/framing/FieldValue.h:162
#3  0x00007f537d62ce25 in match_values (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:267
#4  qpid::broker::HeadersExchange::match (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:285
#5  0x00007f537d62d100 in qpid::broker::HeadersExchange::route (this=0x7f535c1c13e0, msg=...) at qpid/broker/HeadersExchange.cpp:220
#6  0x00007f537d61b8da in qpid::broker::Exchange::routeWithAlternate (this=0x7f535c1c1448, msg=...) at qpid/broker/Exchange.cpp:409
#7  0x00007f537d66ab4e in qpid::broker::Queue::destroyed (this=0x329ca20) at qpid/broker/Queue.cpp:1371
#8  0x00007f537d66b4d7 in qpid::broker::tryAutoDeleteImpl (broker=..., queue=..., connectionId="192.168.1.105:10002-192.168.1.16:42056", userId="anonymous") at qpid/broker/Queue.cpp:1516
#9  0x00007f537d66b768 in qpid::broker::Queue::tryAutoDelete (broker=..., queue=<value optimized out>, connectionId="192.168.1.105:10002-192.168.1.16:42056", userId="anonymous")
    at qpid/broker/Queue.cpp:1548
#10 0x00007f537d6aacd0 in qpid::broker::SessionAdapter::QueueHandlerImpl::destroyExclusiveQueues (this=0x32b1478) at qpid/broker/SessionAdapter.cpp:228
#11 0x00007f537d6ab585 in qpid::broker::SessionAdapter::QueueHandlerImpl::~QueueHandlerImpl (this=0x32b1478, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.cpp:216
#12 0x00007f537d6bb0b1 in ~SessionAdapter (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.h:55
#13 qpid::broker::SessionState::~SessionState (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:90
#14 0x00007f537d6bb409 in qpid::broker::SessionState::~SessionState (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:90
#15 0x00007f537d6b55b1 in ~auto_ptr (this=0x2fcf770) at /usr/include/c++/4.4.6/backward/auto_ptr.h:168
#16 qpid::broker::SessionHandler::handleDetach (this=0x2fcf770) at qpid/broker/SessionHandler.cpp:81
#17 0x00007f537d5f4dcc in qpid::broker::Connection::closed (this=0x7f533cac4f60) at qpid/broker/Connection.cpp:358
#18 0x00007f537d1b254d in qpid::sys::AsynchIOHandler::disconnect (this=0x7f5354001d20) at qpid/sys/AsynchIOHandler.cpp:231
#19 0x00007f537d1b2a89 in qpid::sys::AsynchIOHandler::eof (this=0x7f5354001d20, a=<value optimized out>) at qpid/sys/AsynchIOHandler.cpp:214
#20 0x00007f537d0d55f8 in boost::function1<void, qpid::sys::AsynchIO&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
#21 0x00007f537d0d323d in qpid::sys::posix::AsynchIO::readable (this=0x7f535400f540, h=...) at qpid/sys/posix/AsynchIO.cpp:485
#22 0x00007f537d1b7da3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#23 0x00007f537d1b4fa6 in qpid::sys::DispatchHandle::processEvent (this=0x7f535400f548, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#24 0x00007f537d0dfa4d in process (this=0x1da9c80) at qpid/sys/Poller.h:131
#25 qpid::sys::Poller::run (this=0x1da9c80) at qpid/sys/epoll/EpollPoller.cpp:524
#26 0x00007f537d0d724a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#27 0x00000034d9e077f1 in start_thread () from /lib64/libpthread.so.0
#28 0x00000034d96e592d in clone () from /lib64/libc.so.6
Comment 2 Justin Ross 2012-11-15 16:49:49 EST
Andrew, please do a quick assessment.
Comment 3 Jason Dillaman 2012-11-20 13:10:14 EST
Created attachment 648711 [details]
Quick patch to prevent this crash
Comment 4 Andrew Stitcher 2012-11-26 14:25:26 EST
I think the attached patch only closes the race window but doesn't eliminate it, since it doesn't seem to do anything to avoid the concurrent modification itself. It just reduces the period where concurrent access is a problem.

Jason - is that correct or did I misread the patch somehow?
Comment 6 Jason Dillaman 2012-11-26 15:27:41 EST
(In reply to comment #4)
Swapping out 'getApplicationHeaders', which returns a pointer to a FieldTable, for multiple calls to 'getApplicationHeader(key)' eliminates the concurrent access to the FieldTable since 'getApplicationHeader(key)' uses the same lock as the commands which manipulate the field table (i.e. addTraceId, clearTrace, etc). Since the previous implementation just returned a pointer to a FieldTable, there was no way to lock the FieldTable from modification.  While it is true that FieldTable values might change, it would not happen concurrently and would only affect certain fields (like the trace id).
Comment 7 Chuck Rolke 2012-11-26 20:02:04 EST
I basically approve of the patch. However, there are two issues.

1. In file HeadersExchange.cpp I don't quite understand why function HeadersExchange::equal() is not also affected.

On first glance ::equal() is almost covered by FieldTable::operator== but not quite. 

* operator== does a few extra realdecode steps related to performance buffering that are missing from ::equal and should be done before the comparison.

* ::equal includes a type comparison to magic number 0xf0 before comparing the FieldValues. <magic number rant> What's 0xf0 mean? Must I figure this out? </magic number rant>. Is this significant and should it also be included in operator==?

2. Are either of HeadersExchange::equal() and FieldTable::operator== protected by the multi-thread warning in Message.h around the getApplicationHeaders function warning?

The proposed patch properly protects the FieldTable values. But ::equal() and operator== both look like another thread could clobber part of the comparison.

Am I missing something?
Comment 9 Jason Dillaman 2012-11-27 08:54:47 EST
(In reply to comment #7)
As far as I can tell, HeadersExchange::equal() is only utilized by HeadersExchange::isBound(). When that method is executed, it is not passed application headers from a message (which is the unsafe operation since multiple queues can be mutating a given message's application headers concurrently) but rather bind arguments (which are constant).
Comment 16 Frantisek Reznicek 2013-02-07 07:59:03 EST
Retest started, evaluation of backtrace indicate that concurrent queue delete was done using qpidd's queue autodelete feature.
Comment 18 Frantisek Reznicek 2013-02-09 17:43:06 EST
Reproduced on qpid-cpp-server-0.18-9.el5 and qpid-cpp-server-0.18-10.el[56].
Full backtrace:
Core was generated by `qpidd --no-data-dir --store-dir /root/bz/bz877081/qpidd.5672 --port 5672 --auth'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003f7eebb3b9 in __dynamic_cast () from /usr/lib64/libstdc++.so.6
(gdb) info threads
  9 Thread 0x7fd9b1125700 (LWP 22196)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  8 Thread 0x7fd9b1b26700 (LWP 22195)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  7 Thread 0x7fd9a7d23700 (LWP 22198)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  6 Thread 0x7fd9b2527700 (LWP 22194)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  5 Thread 0x7fd9b0724700 (LWP 22197)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  4 Thread 0x7fd9b30bf700 (LWP 22193)  0x0000003f7d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 0x7fd98ffff700 (LWP 22200)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
  2 Thread 0x7fd9b70ed7a0 (LWP 22191)  0x0000003f7cee8f03 in epoll_wait ()
   from /lib64/libc.so.6
* 1 Thread 0x7fd9afd23700 (LWP 22199)  0x0000003f7eebb3b9 in __dynamic_cast ()
   from /usr/lib64/libstdc++.so.6
(gdb) t a a bt

Thread 9 (Thread 0x7fd9b1125700 (LWP 22196)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fd9b1b26700 (LWP 22195)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fd9a7d23700 (LWP 22198)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fd9b2527700 (LWP 22194)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fd9b0724700 (LWP 22197)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fd9b30bf700 (LWP 22193)):
#0  0x0000003f7d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fd9b7539a9a in wait (this=0x17b0bf0)
    at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0x17b0bf0) at ../include/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0x17b0bf0) at qpid/sys/Timer.cpp:196
#4  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fd98ffff700 (LWP 22200)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f7cee890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fd9b70ed7a0 (LWP 22191)):
#0  0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...)
    at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680)
    at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x00007fd9b795aff2 in qpid::broker::Broker::run (this=<value optimized out>)
    at qpid/broker/Broker.cpp:431
#4  0x000000000040deb1 in qpid::broker::QpiddBroker::execute (
    this=<value optimized out>, options=<value optimized out>)
    at posix/QpiddBroker.cpp:206
#5  0x000000000040aa46 in qpid::broker::run_broker (argc=11, argv=0x7fffd218ee98, 
    hidden=<value optimized out>) at qpidd.cpp:106
#6  0x0000003f7ce1ecdd in __libc_start_main () from /lib64/libc.so.6
#7  0x000000000040a2b9 in _start ()
---Type <return> to continue, or q <return> to quit---

Thread 1 (Thread 0x7fd9afd23700 (LWP 22199)):
#0  0x0000003f7eebb3b9 in __dynamic_cast () from /usr/lib64/libstdc++.so.6
#1  0x00007fd9b7514eb1 in qpid::framing::VariableWidthValue<2>::operator== (
    this=0x7fd99c049b80, d=<value optimized out>)
    at ../include/qpid/framing/FieldValue.h:290
#2  0x00007fd9b79aa155 in match_values (bind=..., msg=...)
    at qpid/broker/HeadersExchange.cpp:269
#3  qpid::broker::HeadersExchange::match (bind=..., msg=...)
    at qpid/broker/HeadersExchange.cpp:287
#4  0x00007fd9b79aa588 in qpid::broker::HeadersExchange::route (this=0x7fd9a8042420, 
    msg=...) at qpid/broker/HeadersExchange.cpp:219
#5  0x00007fd9b799922a in qpid::broker::Exchange::routeWithAlternate (
    this=0x7fd9a8042488, msg=...) at qpid/broker/Exchange.cpp:409
#6  0x00007fd9b79e5b0e in qpid::broker::Queue::destroyed (this=0x7fd9a0025150)
    at qpid/broker/Queue.cpp:1371
#7  0x00007fd9b79e6497 in qpid::broker::tryAutoDeleteImpl (broker=..., queue=..., 
    connectionId="127.0.0.1:5672-127.0.0.1:38572", userId="anonymous")
    at qpid/broker/Queue.cpp:1516
#8  0x00007fd9b79e6728 in qpid::broker::Queue::tryAutoDelete (broker=..., 
    queue=<value optimized out>, connectionId="127.0.0.1:5672-127.0.0.1:38572", 
    userId="anonymous") at qpid/broker/Queue.cpp:1548
#9  0x00007fd9b7a256b0 in qpid::broker::SessionAdapter::QueueHandlerImpl::destroyExclusiveQueues (this=0x7fd99c0aa4a8) at qpid/broker/SessionAdapter.cpp:228
#10 0x00007fd9b7a25b25 in qpid::broker::SessionAdapter::QueueHandlerImpl::~QueueHandlerImpl (this=0x7fd99c0aa4a8, __in_chrg=<value optimized out>)
    at qpid/broker/SessionAdapter.cpp:216
#11 0x00007fd9b7a33f1e in ~SessionAdapter (this=0x7fd99c0aa110, 
    __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.h:55
#12 qpid::broker::SessionState::~SessionState (this=0x7fd99c0aa110, 
    __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:88
#13 0x00007fd9b7a34219 in qpid::broker::SessionState::~SessionState (
    this=0x7fd99c0aa110, __in_chrg=<value optimized out>)
    at qpid/broker/SessionState.cpp:88
#14 0x00007fd9b7a2fd51 in ~auto_ptr (this=0x7fd99c0a9340)
    at /usr/include/c++/4.4.6/backward/auto_ptr.h:168
#15 qpid::broker::SessionHandler::handleDetach (this=0x7fd99c0a9340)
    at qpid/broker/SessionHandler.cpp:81
#16 0x00007fd9b74ff54a in qpid::amqp_0_10::SessionHandler::detach (
    this=0x7fd99c0a9340, name="3e3b5df2-51e9-4a48-93a8-7b0e51f15a93")
    at qpid/amqp_0_10/SessionHandler.cpp:185
#17 0x00007fd9b74744a3 in invoke<qpid::framing::AMQP_AllOperations::SessionHandler> (
    this=<value optimized out>, body=<value optimized out>)
    at qpid/framing/SessionDetachBody.h:65
#18 qpid::framing::AMQP_AllOperations::SessionHandler::Invoker::visit (
    this=<value optimized out>, body=<value optimized out>)
    at qpid/framing/AllInvoker.cpp:840
---Type <return> to continue, or q <return> to quit---
#19 0x00007fd9b74fcd0c in invoke<qpid::amqp_0_10::SessionHandler> (
    this=<value optimized out>, m=<value optimized out>) at qpid/framing/Invoker.h:67
#20 qpid::amqp_0_10::SessionHandler::invoke (this=<value optimized out>, 
    m=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:67
#21 0x00007fd9b74fe12f in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0x7fd99c0a9340, f=...) at qpid/amqp_0_10/SessionHandler.cpp:82
#22 0x00007fd9b797ad23 in operator() (this=0x7fd99c09ebf0, frame=...)
    at qpid/framing/Handler.h:42
#23 qpid::broker::ConnectionHandler::handle (this=0x7fd99c09ebf0, frame=...)
    at qpid/broker/ConnectionHandler.cpp:90
#24 0x00007fd9b7973728 in qpid::broker::Connection::received (this=0x7fd99c09e9e0, 
    frame=...) at qpid/broker/Connection.cpp:166
#25 0x00007fd9b793f48d in qpid::amqp_0_10::Connection::decode (this=0x7fd99c0a5280, 
    buffer=<value optimized out>, size=<value optimized out>)
    at qpid/amqp_0_10/Connection.cpp:58
#26 0x00007fd9b752e90d in qpid::sys::AsynchIOHandler::readbuff (this=0x7fd99c044f80, 
    buff=0x7fd99c0a6e00) at qpid/sys/AsynchIOHandler.cpp:164
#27 0x00007fd9b7450541 in operator() (this=0x7fd99c09ae30, h=...)
    at /usr/include/boost/function/function_template.hpp:1013
#28 qpid::sys::posix::AsynchIO::readable (this=0x7fd99c09ae30, h=...)
    at qpid/sys/posix/AsynchIO.cpp:448
#29 0x00007fd9b7534773 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#30 0x00007fd9b7531976 in qpid::sys::DispatchHandle::processEvent (
    this=0x7fd99c09ae38, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:280
#31 0x00007fd9b745cadd in process (this=0x16ec680) at qpid/sys/Poller.h:131
#32 qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:524
#33 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable (
    p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#34 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0
#35 0x0000003f7cee890d in clone () from /lib64/libc.so.6
(gdb) quit
Comment 19 Frantisek Reznicek 2013-02-11 03:36:59 EST
Extensive testing proved that issue has been fixed. Tested on 10 machines with  total test time over 270 hours on packages:

  python-qpid-0.18-4.el5
  python-qpid-qmf-0.18-14.el5
  qpid-cpp-client-0.18-14.el5
  qpid-cpp-client-devel-0.18-14.el5
  qpid-cpp-client-devel-docs-0.18-14.el5
  qpid-cpp-client-rdma-0.18-14.el5
  qpid-cpp-client-ssl-0.18-14.el5
  qpid-cpp-mrg-debuginfo-0.18-14.el5
  qpid-cpp-server-0.18-14.el5
  qpid-cpp-server-cluster-0.18-14.el5
  qpid-cpp-server-devel-0.18-14.el5
  qpid-cpp-server-ha-0.18-14.el5
  qpid-cpp-server-rdma-0.18-14.el5
  qpid-cpp-server-ssl-0.18-14.el5
  qpid-cpp-server-store-0.18-14.el5
  qpid-cpp-server-xml-0.18-14.el5
  qpid-java-client-0.18-7.el5
  qpid-java-common-0.18-7.el5
  qpid-java-example-0.18-7.el5
  qpid-jca-0.18-8.el5
  qpid-jca-xarecovery-0.18-8.el5
  qpid-jca-zip-0.18-8.el5
  qpid-qmf-0.18-14.el5
  qpid-qmf-debuginfo-0.18-14.el5
  qpid-qmf-devel-0.18-14.el5
  qpid-tests-0.18-2.el5
  qpid-tools-0.18-8.el5
  rh-qpid-cpp-tests-0.18-14.el5
  ruby-qpid-qmf-0.18-14.el5

-> VERIFIED
Comment 21 errata-xmlrpc 2013-03-06 13:52:51 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0561.html

Note You need to log in before you can comment on or make changes to this bug.