| Summary: | qpidd crashes while qpid-perftest --mode fanout / topic in qpid::broker::Message::encode() -> map_if<> -> operator() | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Frantisek Reznicek <freznice> | ||||
| Component: | qpid-cpp | Assignee: | Ken Giusti <kgiusti> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Frantisek Reznicek <freznice> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | Development | CC: | esammons, gsim, iboverma, jross | ||||
| Target Milestone: | 2.1.2 | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | qpid-cpp-0.14-12 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2012-12-07 17:40:50 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Bug Depends On: | |||||||
| Bug Blocks: | 791249 | ||||||
| Attachments: |
|
||||||
Looks like dup of 791249 The 791249 crash was caused by accessing a message header outside of the lock. This problem seems identical, but the access was to the message body. Looks like we have to lock the message body as well - will try to repo to be sure. -K Created attachment 568990 [details]
Potential fix - needs testing.
Retested on rhel5.7/5.8/6.2 i/x on packages: qpid-cpp-*0.14-12.el5 + qpid-qmf-*0.14-3.el5 qpid-cpp-*0.14-12.el6 + qpid-qmf-devel-0.14-5.el6 Issue is reliably fixed, no other crashes detected. Waiting for installable set & retest Reopened upstream jira: https://issues.apache.org/jira/browse/QPID-3877 Submitted fix to upstream trunk. Two patches applied, in order: http://svn.apache.org/viewvc?view=rev&rev=1296230 http://svn.apache.org/viewvc?view=rev&rev=1303068 Retested on rhel5.7/5.8/6.2 i/x on packages: qpid-cpp-*0.14-14.el5 + qpid-qmf-*0.14-4.el5 qpid-cpp-*0.14-12.el6 + qpid-qmf-*0.14-6.el6 Issue is reliably fixed, no other crashes detected. Upstream patch to qpid-0.16 (comment 12) looks ok, contains bug 791249 + bug 801310 patches. -> VERIFIED |
Description of problem: qpidd crashes while qpid-perftest --mode fanout or topic in qpid::broker::Message::encode() -> map_if<> -> operator(): (gdb) 6 Thread 0x2b6379e02040 (LWP 16744) 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 5 Thread 16745 0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 16746 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 3 Thread 16748 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 2 Thread 16749 0x00000036088c6c2b in write () from /lib64/libc.so.6 * 1 Thread 0x42e13940 (LWP 16747) 0x00002aaab40158b0 in ?? () ... Thread 1 (Thread 0x42e13940 (LWP 16747)): #0 0x00002aaab40158b0 in ?? () #1 0x000000360bbaa358 in operator() (this=<value optimized out>, buffer=...) at qpid/framing/TypeFilter.h:38 #2 map_if<qpid::framing::EncodeBody, qpid::framing::TypeFilter<3u> > ( this=<value optimized out>, buffer=...) at qpid/framing/FrameSet.h:110 #3 qpid::broker::Message::encode (this=<value optimized out>, buffer=...) at qpid/broker/Message.cpp:143 #4 0x00002b637c262583 in mrg::msgstore::MessageStoreImpl::msgEncode ( this=<value optimized out>, buff=std::vector of length 160, capacity 160 = {...}, message=...) at MessageStoreImpl.cpp:1321 #5 0x00002b637c262bf7 in mrg::msgstore::MessageStoreImpl::store ( qpid-perftest is ran with multiple publishers and subscribers in durable mode. This defect is similar to bug 791249 and is likely caused by bug 791249's fix. Version-Release number of selected component (if applicable): qpid-cpp*-0.14-9.el5 or qpid-cpp*-0.14-10.el5 qpid-java-*-0.14-3.el5 qpid-qmf-*0.14-3.el5 qpid-tests-0.14-1.el5 qpid-tools-0.14-1.el5 How reproducible: 80% detected on updated rhel5.7 i386 / x86_64 detected on updated rhel5.8 i386 / x86_64 not detected on rhel6.2 Steps to Reproduce: see bug 791249 steps Actual results: qpidd crashes while qpid-perftest is running. Expected results: qpidd should not crash. Additional info: Bug 791249 fix moved the original crash to other place: -rw------- 1 root root 77475840 Mar 7 14:33 /root/.qpidd/core.16744 /root/.qpidd/core.16744: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd' GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5) ... Core was generated by `/usr/sbin/qpidd --auth no --daemon --port 0 --log-enable info+ --log-to-file qp'. Program terminated with signal 11, Segmentation fault. (gdb) Stack level 0, frame at 0x42e0e7e0: rip = 0x2aaab40158b0; saved rip 0x360bbaa358 called by frame at 0x42e0e810 Arglist at 0x42e0e7d0, args: Locals at 0x42e0e7d0, Previous frame's sp is 0x42e0e7e0 Saved registers: rip at 0x42e0e7d8 (*): Shared library is missing debugging information. (gdb) 6 Thread 0x2b6379e02040 (LWP 16744) 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 5 Thread 16745 0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 16746 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 3 Thread 16748 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 2 Thread 16749 0x00000036088c6c2b in write () from /lib64/libc.so.6 * 1 Thread 0x42e13940 (LWP 16747) 0x00002aaab40158b0 in ?? () Thread 6 (Thread 0x2b6379e02040 (LWP 16744)): #0 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 #1 0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000360bb31d46 in qpid::broker::Broker::run ( this=<value optimized out>) at qpid/broker/Broker.cpp:400 #4 0x0000000000409b8c in QpiddDaemon::child (this=0x141fcbd0) at posix/QpiddBroker.cpp:144 #5 0x000000360bb5f60e in qpid::broker::Daemon::fork (this=0x7fff7f47ddb0) at qpid/broker/Daemon.cpp:91 #6 0x0000000000407085 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:182 #7 0x0000000000405822 in run_broker (argc=16, argv=0x7fff7f47e3a8, hidden=<value optimized out>) at qpidd.cpp:83 #8 0x000000360881d994 in __libc_start_main () from /lib64/libc.so.6 #9 0x0000000000405279 in _start () Thread 5 (Thread 16745): #0 0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000360b61253a in qpid::sys::Timer::run (this=0xffffffffffffff92) at ../include/qpid/sys/posix/Condition.h:69 #2 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable ( p=0x141e9c24) at qpid/sys/posix/Thread.cpp:35 #3 0x000000360900677d in start_thread () from /lib64/libpthread.so.0 #4 0x00000036088d49ad in clone () from /lib64/libc.so.6 Thread 4 (Thread 16746): #0 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 #1 0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable ( p=0x6) at qpid/sys/posix/Thread.cpp:35 #4 0x000000360900677d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036088d49ad in clone () from /lib64/libc.so.6 Thread 3 (Thread 16748): #0 0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6 #1 0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable ( p=0x6) at qpid/sys/posix/Thread.cpp:35 #4 0x000000360900677d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036088d49ad in clone () from /lib64/libc.so.6 Thread 2 (Thread 16749): #0 0x00000036088c6c2b in write () from /lib64/libc.so.6 #1 0x000000360b51c27f in qpid::sys::Socket::write ( this=<value optimized out>, buf=0x142998f0, count=205) at qpid/sys/posix/Socket.cpp:232 #2 0x000000360b52446b in qpid::sys::posix::AsynchIO::writeable ( this=0x14297700, h=...) at qpid/sys/posix/AsynchIO.cpp:516 #3 0x000000360b60b17a in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xcd, a0=...) at /usr/include/boost/function/function_template.hpp:576 #4 0x000000360b60a801 in qpid::sys::DispatchHandle::processEvent ( this=0x14297708, type=WRITABLE) at qpid/sys/DispatchHandle.cpp:283 #5 0x000000360b534ed4 in process (this=0x1417bca0) at qpid/sys/Poller.h:131 #6 qpid::sys::Poller::run (this=0x1417bca0) at qpid/sys/epoll/EpollPoller.cpp:524 #7 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable ( p=0x17) at qpid/sys/posix/Thread.cpp:35 #8 0x000000360900677d in start_thread () from /lib64/libpthread.so.0 #9 0x00000036088d49ad in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x42e13940 (LWP 16747)): #0 0x00002aaab40158b0 in ?? () #1 0x000000360bbaa358 in operator() (this=<value optimized out>, buffer=...) at qpid/framing/TypeFilter.h:38 #2 map_if<qpid::framing::EncodeBody, qpid::framing::TypeFilter<3u> > ( this=<value optimized out>, buffer=...) at qpid/framing/FrameSet.h:110 #3 qpid::broker::Message::encode (this=<value optimized out>, buffer=...) at qpid/broker/Message.cpp:143 #4 0x00002b637c262583 in mrg::msgstore::MessageStoreImpl::msgEncode ( this=<value optimized out>, buff=std::vector of length 160, capacity 160 = {...}, message=...) at MessageStoreImpl.cpp:1321 #5 0x00002b637c262bf7 in mrg::msgstore::MessageStoreImpl::store ( this=0x2aaab40158c0, queue=0x2aaaac5b52e0, txn=0x42e0edf0, message=...) at MessageStoreImpl.cpp:1331 #6 0x00002b637c27b7db in mrg::msgstore::MessageStoreImpl::enqueue ( this=0x141fd240, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1303 #7 0x000000360bbb783b in qpid::broker::MessageStoreModule::enqueue ( this=<value optimized out>, ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:125 #8 0x000000360bbc9f80 in qpid::broker::Queue::enqueue (this=0x2aaaac5b52e0, ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>) at qpid/broker/Queue.cpp:811 #9 0x000000360bbcb908 in qpid::broker::Queue::deliver (this=0x2aaaac5b52e0, msg=...) at qpid/broker/Queue.cpp:171 #10 0x000000360bb620c2 in qpid::broker::DeliverableMessage::deliverTo ( this=0x42e10870, queue=...) at qpid/broker/DeliverableMessage.cpp:33 #11 0x000000360bb7fbc2 in qpid::broker::Exchange::doRoute (this=0x14205a88, msg=..., b=...) at qpid/broker/Exchange.cpp:119 #12 0x000000360bc3ba18 in qpid::broker::TopicExchange::route ( this=0x14205a88, msg=..., routingKey="qpid-perftest0") at qpid/broker/TopicExchange.cpp:375 #13 0x000000360bc06a1a in qpid::broker::SemanticState::route ( this=<value optimized out>, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:495 #14 0x000000360bc0744d in qpid::broker::SemanticState::handle ( this=0x14254b88, msg=...) at qpid/broker/SemanticState.cpp:449 #15 0x000000360bc31448 in qpid::broker::SessionState::handleContent ( this=0x142549b0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:266 #16 0x000000360bc31bd0 in qpid::broker::SessionState::handleIn ( this=0x142549b0, frame=...) at qpid/broker/SessionState.cpp:362 #17 0x000000360b5d2d65 in qpid::amqp_0_10::SessionHandler::handleIn ( this=0x1424db30, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #18 0x000000360bb5cee1 in operator() (this=0x1424ae60, frame=...) at qpid/framing/Handler.h:42 #19 qpid::broker::ConnectionHandler::handle (this=0x1424ae60, frame=...) at qpid/broker/ConnectionHandler.cpp:87 #20 0x000000360bb520a8 in qpid::broker::Connection::received ( this=0x1424ac80, frame=...) at qpid/broker/Connection.cpp:159 #21 0x000000360bb22364 in qpid::amqp_0_10::Connection::decode ( this=0x14255550, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #22 0x000000360b605662 in qpid::sys::AsynchIOHandler::readbuff ( this=0x14254710, buff=0x1424cc60) at qpid/sys/AsynchIOHandler.cpp:135 #23 0x000000360b529d4a in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x2aaab402b320, a0=..., a1=0x2aaab000fd10) at /usr/include/boost/function/function_template.hpp:576 #24 0x000000360b527af0 in qpid::sys::posix::AsynchIO::readable ( this=0x14209a70, h=...) at qpid/sys/posix/AsynchIO.cpp:446 #25 0x000000360b60b17a in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2aaab402b320, a0=...) at /usr/include/boost/function/function_template.hpp:576 #26 0x000000360b60a87f in qpid::sys::DispatchHandle::processEvent ( this=0x14209a78, type=READABLE) at qpid/sys/DispatchHandle.cpp:280 #27 0x000000360b534ed4 in process (this=0x1417bca0) at qpid/sys/Poller.h:131 #28 qpid::sys::Poller::run (this=0x1417bca0) at qpid/sys/epoll/EpollPoller.cpp:524 #29 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable ( p=0x2aaab40158c0) at qpid/sys/posix/Thread.cpp:35 #30 0x000000360900677d in start_thread () from /lib64/libpthread.so.0 #31 0x00000036088d49ad in clone () from /lib64/libc.so.6 (gdb) quit