Description of problem: When a single message is delivered to multiple queues, a pointer to that original message is stored in each queue. If those queues are then concurrently re-routed through a header exchange (i.e. delete the queues, each with an alt-exchange), there is a risk of a broker crash because the application headers on the shared copy of the message are being manipulated concurrently in multiple threads in order to clear the trace id. Version-Release number of selected component (if applicable): Qpid 0.18-9 How reproducible: Occasionally -- race condition Steps to Reproduce: 1. Declare multiple queues, each with an alt-exchange through a header exchange 2. Bind the queues to an exchange such that a single message is delivered to all the queues 3. Concurrently re-route the queues through the header exchange (i.e. delete the queues or issue QMF re-route commands) Actual results: Thread A is attempting to evaluate the application headers on message 1 during a re-route from Queue A. Thread B is concurrently clearing the trace id on message 1 while re-routing from Queue B. Therefore, Thread B invalidated the application headers while Thread A was attempting to access them. Expected results: Since a single message can be delivered to multiple queues, individual queues should not manipulate shared message properties. Additional info:
Stack Backtrace: #0 0x0000000100000001 in ?? () #1 0x00000034db2bb3ed in __dynamic_cast () from /usr/lib64/libstdc++.so.6 #2 0x00007f537d197d41 in qpid::framing::FixedWidthValue<2>::operator== (this=0x7f53385c4060, d=<value optimized out>) at ../include/qpid/framing/FieldValue.h:162 #3 0x00007f537d62ce25 in match_values (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:267 #4 qpid::broker::HeadersExchange::match (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:285 #5 0x00007f537d62d100 in qpid::broker::HeadersExchange::route (this=0x7f535c1c13e0, msg=...) at qpid/broker/HeadersExchange.cpp:220 #6 0x00007f537d61b8da in qpid::broker::Exchange::routeWithAlternate (this=0x7f535c1c1448, msg=...) at qpid/broker/Exchange.cpp:409 #7 0x00007f537d66ab4e in qpid::broker::Queue::destroyed (this=0x329ca20) at qpid/broker/Queue.cpp:1371 #8 0x00007f537d66b4d7 in qpid::broker::tryAutoDeleteImpl (broker=..., queue=..., connectionId="192.168.1.105:10002-192.168.1.16:42056", userId="anonymous") at qpid/broker/Queue.cpp:1516 #9 0x00007f537d66b768 in qpid::broker::Queue::tryAutoDelete (broker=..., queue=<value optimized out>, connectionId="192.168.1.105:10002-192.168.1.16:42056", userId="anonymous") at qpid/broker/Queue.cpp:1548 #10 0x00007f537d6aacd0 in qpid::broker::SessionAdapter::QueueHandlerImpl::destroyExclusiveQueues (this=0x32b1478) at qpid/broker/SessionAdapter.cpp:228 #11 0x00007f537d6ab585 in qpid::broker::SessionAdapter::QueueHandlerImpl::~QueueHandlerImpl (this=0x32b1478, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.cpp:216 #12 0x00007f537d6bb0b1 in ~SessionAdapter (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.h:55 #13 qpid::broker::SessionState::~SessionState (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:90 #14 0x00007f537d6bb409 in qpid::broker::SessionState::~SessionState (this=0x32b10e0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:90 #15 0x00007f537d6b55b1 in ~auto_ptr (this=0x2fcf770) at /usr/include/c++/4.4.6/backward/auto_ptr.h:168 #16 qpid::broker::SessionHandler::handleDetach (this=0x2fcf770) at qpid/broker/SessionHandler.cpp:81 #17 0x00007f537d5f4dcc in qpid::broker::Connection::closed (this=0x7f533cac4f60) at qpid/broker/Connection.cpp:358 #18 0x00007f537d1b254d in qpid::sys::AsynchIOHandler::disconnect (this=0x7f5354001d20) at qpid/sys/AsynchIOHandler.cpp:231 #19 0x00007f537d1b2a89 in qpid::sys::AsynchIOHandler::eof (this=0x7f5354001d20, a=<value optimized out>) at qpid/sys/AsynchIOHandler.cpp:214 #20 0x00007f537d0d55f8 in boost::function1<void, qpid::sys::AsynchIO&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #21 0x00007f537d0d323d in qpid::sys::posix::AsynchIO::readable (this=0x7f535400f540, h=...) at qpid/sys/posix/AsynchIO.cpp:485 #22 0x00007f537d1b7da3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #23 0x00007f537d1b4fa6 in qpid::sys::DispatchHandle::processEvent (this=0x7f535400f548, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #24 0x00007f537d0dfa4d in process (this=0x1da9c80) at qpid/sys/Poller.h:131 #25 qpid::sys::Poller::run (this=0x1da9c80) at qpid/sys/epoll/EpollPoller.cpp:524 #26 0x00007f537d0d724a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #27 0x00000034d9e077f1 in start_thread () from /lib64/libpthread.so.0 #28 0x00000034d96e592d in clone () from /lib64/libc.so.6
Andrew, please do a quick assessment.
Created attachment 648711 [details] Quick patch to prevent this crash
I think the attached patch only closes the race window but doesn't eliminate it, since it doesn't seem to do anything to avoid the concurrent modification itself. It just reduces the period where concurrent access is a problem. Jason - is that correct or did I misread the patch somehow?
(In reply to comment #4) Swapping out 'getApplicationHeaders', which returns a pointer to a FieldTable, for multiple calls to 'getApplicationHeader(key)' eliminates the concurrent access to the FieldTable since 'getApplicationHeader(key)' uses the same lock as the commands which manipulate the field table (i.e. addTraceId, clearTrace, etc). Since the previous implementation just returned a pointer to a FieldTable, there was no way to lock the FieldTable from modification. While it is true that FieldTable values might change, it would not happen concurrently and would only affect certain fields (like the trace id).
I basically approve of the patch. However, there are two issues. 1. In file HeadersExchange.cpp I don't quite understand why function HeadersExchange::equal() is not also affected. On first glance ::equal() is almost covered by FieldTable::operator== but not quite. * operator== does a few extra realdecode steps related to performance buffering that are missing from ::equal and should be done before the comparison. * ::equal includes a type comparison to magic number 0xf0 before comparing the FieldValues. <magic number rant> What's 0xf0 mean? Must I figure this out? </magic number rant>. Is this significant and should it also be included in operator==? 2. Are either of HeadersExchange::equal() and FieldTable::operator== protected by the multi-thread warning in Message.h around the getApplicationHeaders function warning? The proposed patch properly protects the FieldTable values. But ::equal() and operator== both look like another thread could clobber part of the comparison. Am I missing something?
(In reply to comment #7) As far as I can tell, HeadersExchange::equal() is only utilized by HeadersExchange::isBound(). When that method is executed, it is not passed application headers from a message (which is the unsafe operation since multiple queues can be mutating a given message's application headers concurrently) but rather bind arguments (which are constant).
Retest started, evaluation of backtrace indicate that concurrent queue delete was done using qpidd's queue autodelete feature.
Reproduced on qpid-cpp-server-0.18-9.el5 and qpid-cpp-server-0.18-10.el[56]. Full backtrace: Core was generated by `qpidd --no-data-dir --store-dir /root/bz/bz877081/qpidd.5672 --port 5672 --auth'. Program terminated with signal 11, Segmentation fault. #0 0x0000003f7eebb3b9 in __dynamic_cast () from /usr/lib64/libstdc++.so.6 (gdb) info threads 9 Thread 0x7fd9b1125700 (LWP 22196) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 8 Thread 0x7fd9b1b26700 (LWP 22195) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 7 Thread 0x7fd9a7d23700 (LWP 22198) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 6 Thread 0x7fd9b2527700 (LWP 22194) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 5 Thread 0x7fd9b0724700 (LWP 22197) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 4 Thread 0x7fd9b30bf700 (LWP 22193) 0x0000003f7d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7fd98ffff700 (LWP 22200) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 2 Thread 0x7fd9b70ed7a0 (LWP 22191) 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 0x7fd9afd23700 (LWP 22199) 0x0000003f7eebb3b9 in __dynamic_cast () from /usr/lib64/libstdc++.so.6 (gdb) t a a bt Thread 9 (Thread 0x7fd9b1125700 (LWP 22196)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd9b1b26700 (LWP 22195)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd9a7d23700 (LWP 22198)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd9b2527700 (LWP 22194)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd9b0724700 (LWP 22197)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd9b30bf700 (LWP 22193)): #0 0x0000003f7d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd9b7539a9a in wait (this=0x17b0bf0) at ../include/qpid/sys/posix/Condition.h:69 #2 wait (this=0x17b0bf0) at ../include/qpid/sys/Monitor.h:45 #3 qpid::sys::Timer::run (this=0x17b0bf0) at qpid/sys/Timer.cpp:196 #4 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #5 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd98ffff700 (LWP 22200)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003f7cee890d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd9b70ed7a0 (LWP 22191)): #0 0x0000003f7cee8f03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd9b745c55a in qpid::sys::Poller::wait (this=0x16ec680, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00007fd9b745cab7 in qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00007fd9b795aff2 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:431 #4 0x000000000040deb1 in qpid::broker::QpiddBroker::execute ( this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:206 #5 0x000000000040aa46 in qpid::broker::run_broker (argc=11, argv=0x7fffd218ee98, hidden=<value optimized out>) at qpidd.cpp:106 #6 0x0000003f7ce1ecdd in __libc_start_main () from /lib64/libc.so.6 #7 0x000000000040a2b9 in _start () ---Type <return> to continue, or q <return> to quit--- Thread 1 (Thread 0x7fd9afd23700 (LWP 22199)): #0 0x0000003f7eebb3b9 in __dynamic_cast () from /usr/lib64/libstdc++.so.6 #1 0x00007fd9b7514eb1 in qpid::framing::VariableWidthValue<2>::operator== ( this=0x7fd99c049b80, d=<value optimized out>) at ../include/qpid/framing/FieldValue.h:290 #2 0x00007fd9b79aa155 in match_values (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:269 #3 qpid::broker::HeadersExchange::match (bind=..., msg=...) at qpid/broker/HeadersExchange.cpp:287 #4 0x00007fd9b79aa588 in qpid::broker::HeadersExchange::route (this=0x7fd9a8042420, msg=...) at qpid/broker/HeadersExchange.cpp:219 #5 0x00007fd9b799922a in qpid::broker::Exchange::routeWithAlternate ( this=0x7fd9a8042488, msg=...) at qpid/broker/Exchange.cpp:409 #6 0x00007fd9b79e5b0e in qpid::broker::Queue::destroyed (this=0x7fd9a0025150) at qpid/broker/Queue.cpp:1371 #7 0x00007fd9b79e6497 in qpid::broker::tryAutoDeleteImpl (broker=..., queue=..., connectionId="127.0.0.1:5672-127.0.0.1:38572", userId="anonymous") at qpid/broker/Queue.cpp:1516 #8 0x00007fd9b79e6728 in qpid::broker::Queue::tryAutoDelete (broker=..., queue=<value optimized out>, connectionId="127.0.0.1:5672-127.0.0.1:38572", userId="anonymous") at qpid/broker/Queue.cpp:1548 #9 0x00007fd9b7a256b0 in qpid::broker::SessionAdapter::QueueHandlerImpl::destroyExclusiveQueues (this=0x7fd99c0aa4a8) at qpid/broker/SessionAdapter.cpp:228 #10 0x00007fd9b7a25b25 in qpid::broker::SessionAdapter::QueueHandlerImpl::~QueueHandlerImpl (this=0x7fd99c0aa4a8, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.cpp:216 #11 0x00007fd9b7a33f1e in ~SessionAdapter (this=0x7fd99c0aa110, __in_chrg=<value optimized out>) at qpid/broker/SessionAdapter.h:55 #12 qpid::broker::SessionState::~SessionState (this=0x7fd99c0aa110, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:88 #13 0x00007fd9b7a34219 in qpid::broker::SessionState::~SessionState ( this=0x7fd99c0aa110, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:88 #14 0x00007fd9b7a2fd51 in ~auto_ptr (this=0x7fd99c0a9340) at /usr/include/c++/4.4.6/backward/auto_ptr.h:168 #15 qpid::broker::SessionHandler::handleDetach (this=0x7fd99c0a9340) at qpid/broker/SessionHandler.cpp:81 #16 0x00007fd9b74ff54a in qpid::amqp_0_10::SessionHandler::detach ( this=0x7fd99c0a9340, name="3e3b5df2-51e9-4a48-93a8-7b0e51f15a93") at qpid/amqp_0_10/SessionHandler.cpp:185 #17 0x00007fd9b74744a3 in invoke<qpid::framing::AMQP_AllOperations::SessionHandler> ( this=<value optimized out>, body=<value optimized out>) at qpid/framing/SessionDetachBody.h:65 #18 qpid::framing::AMQP_AllOperations::SessionHandler::Invoker::visit ( this=<value optimized out>, body=<value optimized out>) at qpid/framing/AllInvoker.cpp:840 ---Type <return> to continue, or q <return> to quit--- #19 0x00007fd9b74fcd0c in invoke<qpid::amqp_0_10::SessionHandler> ( this=<value optimized out>, m=<value optimized out>) at qpid/framing/Invoker.h:67 #20 qpid::amqp_0_10::SessionHandler::invoke (this=<value optimized out>, m=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:67 #21 0x00007fd9b74fe12f in qpid::amqp_0_10::SessionHandler::handleIn ( this=0x7fd99c0a9340, f=...) at qpid/amqp_0_10/SessionHandler.cpp:82 #22 0x00007fd9b797ad23 in operator() (this=0x7fd99c09ebf0, frame=...) at qpid/framing/Handler.h:42 #23 qpid::broker::ConnectionHandler::handle (this=0x7fd99c09ebf0, frame=...) at qpid/broker/ConnectionHandler.cpp:90 #24 0x00007fd9b7973728 in qpid::broker::Connection::received (this=0x7fd99c09e9e0, frame=...) at qpid/broker/Connection.cpp:166 #25 0x00007fd9b793f48d in qpid::amqp_0_10::Connection::decode (this=0x7fd99c0a5280, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #26 0x00007fd9b752e90d in qpid::sys::AsynchIOHandler::readbuff (this=0x7fd99c044f80, buff=0x7fd99c0a6e00) at qpid/sys/AsynchIOHandler.cpp:164 #27 0x00007fd9b7450541 in operator() (this=0x7fd99c09ae30, h=...) at /usr/include/boost/function/function_template.hpp:1013 #28 qpid::sys::posix::AsynchIO::readable (this=0x7fd99c09ae30, h=...) at qpid/sys/posix/AsynchIO.cpp:448 #29 0x00007fd9b7534773 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #30 0x00007fd9b7531976 in qpid::sys::DispatchHandle::processEvent ( this=0x7fd99c09ae38, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #31 0x00007fd9b745cadd in process (this=0x16ec680) at qpid/sys/Poller.h:131 #32 qpid::sys::Poller::run (this=0x16ec680) at qpid/sys/epoll/EpollPoller.cpp:524 #33 0x00007fd9b7454b7a in qpid::sys::(anonymous namespace)::runRunnable ( p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #34 0x0000003f7d207851 in start_thread () from /lib64/libpthread.so.0 #35 0x0000003f7cee890d in clone () from /lib64/libc.so.6 (gdb) quit
Extensive testing proved that issue has been fixed. Tested on 10 machines with total test time over 270 hours on packages: python-qpid-0.18-4.el5 python-qpid-qmf-0.18-14.el5 qpid-cpp-client-0.18-14.el5 qpid-cpp-client-devel-0.18-14.el5 qpid-cpp-client-devel-docs-0.18-14.el5 qpid-cpp-client-rdma-0.18-14.el5 qpid-cpp-client-ssl-0.18-14.el5 qpid-cpp-mrg-debuginfo-0.18-14.el5 qpid-cpp-server-0.18-14.el5 qpid-cpp-server-cluster-0.18-14.el5 qpid-cpp-server-devel-0.18-14.el5 qpid-cpp-server-ha-0.18-14.el5 qpid-cpp-server-rdma-0.18-14.el5 qpid-cpp-server-ssl-0.18-14.el5 qpid-cpp-server-store-0.18-14.el5 qpid-cpp-server-xml-0.18-14.el5 qpid-java-client-0.18-7.el5 qpid-java-common-0.18-7.el5 qpid-java-example-0.18-7.el5 qpid-jca-0.18-8.el5 qpid-jca-xarecovery-0.18-8.el5 qpid-jca-zip-0.18-8.el5 qpid-qmf-0.18-14.el5 qpid-qmf-debuginfo-0.18-14.el5 qpid-qmf-devel-0.18-14.el5 qpid-tests-0.18-2.el5 qpid-tools-0.18-8.el5 rh-qpid-cpp-tests-0.18-14.el5 ruby-qpid-qmf-0.18-14.el5 -> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0561.html