Bug 1023418

Summary: [amqp1.0] receiver on exchange causes broker crash
Product: Red Hat Enterprise MRG Reporter: Petr Matousek <pematous>
Component: qpid-cppAssignee: Gordon Sim <gsim>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Matousek <pematous>
Severity: high Docs Contact:
Priority: high    
Version: DevelopmentCC: esammons, gsim, iboverma, jross
Target Milestone: 3.0   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qpid-cpp-0.22-23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-21 12:54:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
qpidd.log none

Description Petr Matousek 2013-10-25 11:29:23 UTC
Description of problem:
 	
Creating consumer on an exchange over amqp1.0 causes broker crash.

Version-Release number of selected component (if applicable):
qpid-cpp-*-0.22-22

How reproducible:
100%

Steps to Reproduce:
1. drain  --connection-options "{  protocol : 'amqp1.0' }" amq.direct
2. broker crash

Actual results:
broker crash while receiving from exchange using amqp1.0

Expected results:
no crash

Additional info:
more info coming

Comment 1 Petr Matousek 2013-10-25 11:32:16 UTC
Created attachment 816111 [details]
qpidd.log

qpidd.log (debug+)

Comment 2 Petr Matousek 2013-10-25 11:46:14 UTC
coredump:

Core was generated by `qpidd --log-enable=debug+'.
Program terminated with signal 6, Aborted.
#0  0x0000003e718328e5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install nspr-4.9.5-2.el6_4.x86_64 nss-3.14.3-4.el6_4.x86_64 nss-util-3.14.3-3.el6_4.x86_64 qpid-proton-c-0.5-6.el6.x86_64
(gdb) i th
  4 Thread 0x7fa0ecad8700 (LWP 28760)  0x0000003e718e8f43 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  3 Thread 0x7fa0ed4d9700 (LWP 28759)  0x0000003e718e8f43 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  2 Thread 0x7fa0ee05c700 (LWP 28758)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
* 1 Thread 0x7fa0f08e17a0 (LWP 28757)  0x0000003e718328e5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
(gdb) t a a bt

Thread 4 (Thread 0x7fa0ecad8700 (LWP 28760)):
#0  0x0000003e718e8f43 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fa0f0c4dcad in qpid::sys::Poller::wait (this=0x1bfa2e0, timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:566
#2  0x00007fa0f0c4e381 in qpid::sys::Poller::run (this=0x1bfa2e0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:518
#3  0x00007fa0f0c4389a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Thread.cpp:35
#4  0x0000003e72007851 in start_thread (arg=0x7fa0ecad8700) at pthread_create.c:301
#5  0x0000003e718e894d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fa0ed4d9700 (LWP 28759)):
#0  0x0000003e718e8f43 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007fa0f0c4dcad in qpid::sys::Poller::wait (this=0x1bfa2e0, timeout=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:566
#2  0x00007fa0f0c4e381 in qpid::sys::Poller::run (this=0x1bfa2e0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:518
#3  0x00007fa0f0c4389a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Thread.cpp:35
#4  0x0000003e72007851 in start_thread (arg=0x7fa0ed4d9700) at pthread_create.c:301
#5  0x0000003e718e894d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fa0ee05c700 (LWP 28758)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
#1  0x00007fa0f0ca29ed in wait (this=0x1bfe5d0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Condition.h:69
#2  wait (this=0x1bfe5d0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0x1bfe5d0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Timer.cpp:186
#4  0x00007fa0f0c4389a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Thread.cpp:35
#5  0x0000003e72007851 in start_thread (arg=0x7fa0ee05c700) at pthread_create.c:301
#6  0x0000003e718e894d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fa0f08e17a0 (LWP 28757)):
#0  0x0000003e718328e5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003e718340c5 in abort () at abort.c:92
#2  0x0000003e7182ba0e in __assert_fail_base (fmt=<value optimized out>, assertion=0x7fa0f1244276 "consumers > 0", 
    file=0x7fa0f1244570 "/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/broker/Queue.cpp", line=<value optimized out>, function=<value optimized out>)
    at assert.c:96
#3  0x0000003e7182bad0 in __assert_fail (assertion=0x7fa0f1244276 "consumers > 0", 
    file=0x7fa0f1244570 "/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/broker/Queue.cpp", line=1759, 
    function=0x7fa0f1244e60 "void qpid::broker::Queue::QueueUsers::removeConsumer()") at assert.c:105
#4  0x00007fa0f112dd33 in qpid::broker::Queue::QueueUsers::removeConsumer (this=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:1759
#5  0x00007fa0f113b7d4 in qpid::broker::Queue::cancel (this=0x1c31a10, c=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:585
#6  0x00007fa0f042f167 in qpid::broker::amqp::OutgoingFromQueue::detached (this=0x1c34770)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Outgoing.cpp:177
#7  0x00007fa0f0440dab in qpid::broker::amqp::Session::close (this=0x1c13630) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Session.cpp:581
#8  0x00007fa0f0403dee in qpid::broker::amqp::Connection::closed (this=0x7fa0e0002458)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Connection.cpp:193
#9  0x00007fa0f0404445 in qpid::broker::amqp::Connection::close (this=0x7fa0e0002458)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Connection.cpp:200
#10 0x00007fa0f04054d0 in qpid::broker::amqp::Connection::decode (this=0x7fa0e0002458, buffer=<value optimized out>, size=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Connection.cpp:110
#11 0x00007fa0f04387b1 in qpid::broker::amqp::Sasl::decode (this=0x7fa0e0002400, buffer=<value optimized out>, size=24)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp/Sasl.cpp:49
#12 0x00007fa0f0c98d40 in qpid::sys::AsynchIOHandler::readbuff (this=0x7fa0e0001470, buff=0x7fa0e00019c0)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/AsynchIOHandler.cpp:130
#13 0x00007fa0f0c2bc34 in operator() (this=0x7fa0e0001a20, h=...) at /usr/include/boost/function/function_template.hpp:1013
#14 qpid::sys::posix::AsynchIO::readable (this=0x7fa0e0001a20, h=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/AsynchIO.cpp:453
#15 0x00007fa0f0c9d3f3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#16 0x00007fa0f0c9c541 in qpid::sys::DispatchHandle::processEvent (this=0x7fa0e0001a28, type=qpid::sys::Poller::READABLE)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/DispatchHandle.cpp:280
#17 0x00007fa0f0c4e372 in process (this=0x1bfa2e0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Poller.h:131
#18 qpid::sys::Poller::run (this=0x1bfa2e0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:522
#19 0x00007fa0f1102802 in qpid::broker::Broker::run (this=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Broker.cpp:433
#20 0x00000000004073b7 in qpid::broker::QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/posix/QpiddBroker.cpp:206
#21 0x000000000040cb04 in qpid::broker::run_broker (argc=2, argv=0x7fff5e44ee08, hidden=<value optimized out>)
    at /usr/src/debug/qpid-0.22/cpp/src/qpidd.cpp:108
#22 0x0000003e7181ecdd in __libc_start_main (main=0x406c60 <main(int, char**)>, argc=2, ubp_av=0x7fff5e44ee08, init=<value optimized out>, 
    fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff5e44edf8) at libc-start.c:226
#23 0x0000000000406b99 in _start ()

Comment 3 Gordon Sim 2013-10-25 13:38:01 UTC
Fixed upstream: https://svn.apache.org/viewvc?view=revision&revision=r1535731

Comment 4 Petr Matousek 2013-10-29 09:41:36 UTC
This issue has been fixed. Verified on rhel6.4 (i386, x86_64).

packages used for testing:
qpid-cpp-*-0.22-23.el6

-> VERIFIED