Created attachment 410521 [details] glibc error printout and core file backtrace While running perftest on RHEL-6, I see the following failure reasonably often: *** glibc detected *** /home/kpvdr/qpid/cpp/src/tests/.libs/lt-perftest: invalid fastbin entry (free): 0x00007f924c0013a0 *** Machine: mrg10 OS: RHEL6.0-20100428.n.0_nfs-Server This box has locally checked out and built qpid and store: qpid: svn r.939663 store: svn r.3943 Both modules are built without optimization: make CXXFLAGS="-O0 -ggdb" -j8 Broker start (from src dir): ./qpidd --load-module /home/kpvdr/store/lib/.libs/msgstore.so --data-dir=/tmp --auth no --log-enable info+ Client start (from test dir): ./perftest --iterations 1 --summary --unique-data yes --log-enable info+ --mode topic --qt 1 --durable yes --npubs 2 --nsubs 3 --count 200000 --size 1024 --tx 0 --async-commit no NOTE: This test can also cause a broker failure in the store module (also related to freeing memory) - being treated as a separate bug.
See attachment for glibc error in full and core file backtrace.
Created attachment 410561 [details] Second occurrence - glibc error and core backtrace
Similar error seen on broker (without store): Program terminated with signal 6, Aborted. #0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt #0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00000031d6835135 in abort () at abort.c:92 #2 0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:186 #3 0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280 #4 0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42 #5 intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57 #6 ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101 #7 qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46 #8 0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115 #9 pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237 #10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96 #11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214 #12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013 #13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536 #14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:289 #16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123 #17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483 #18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301 #20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) threads apply all bt Undefined command: "threads". Try "help". (gdb) thread apply all bt Thread 10 (Thread 28610): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f288ff49710) at pthread_create.c:301 #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 9 (Thread 28594): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890f6c8b2 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:338 #4 0x000000000040d83e in QpiddBroker::execute (this=<value optimized out>, options=0x1dc9630) at ../../src/posix/QpiddBroker.cpp:176 #5 0x000000000040a2a2 in main (argc=5, argv=0x7fffd48070b8) at ../../src/qpidd.cpp:80 Thread 8 (Thread 28617): #0 0x00007f2890f65e65 in release (this=0x7f2887ffe398, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143 #1 boost::detail::shared_count::~shared_count (this=0x7f2887ffe398, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217 #2 0x00007f2890fcb93a in ~shared_ptr (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169 #3 qpid::broker::Queue::getNextMessage (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>) at ../../src/qpid/broker/Queue.cpp:295 #4 0x00007f2890fcba50 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:391 #5 0x00007f2890fee36a in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x1fb1fd0) at ../../src/qpid/broker/SemanticState.cpp:685 #6 0x00007f2890b8b06d in qpid::sys::AggregateOutput::doOutput (this=0x20353f0) at ../../src/qpid/sys/AggregateOutput.cpp:59 #7 0x00007f2890f7dc83 in qpid::broker::Connection::doOutput (this=0x2035380) at ../../src/qpid/broker/Connection.cpp:282 #8 0x00007f2890f5df6d in qpid::amqp_0_10::Connection::canEncode (this=0x1f29600) at ../../src/qpid/amqp_0_10/Connection.cpp:67 #9 0x00007f2890b8d09b in qpid::sys::AsynchIOHandler::idle (this=0x1f6cdc0) at ../../src/qpid/sys/AsynchIOHandler.cpp:210 #10 0x00007f2890ac9723 in operator() (this=0x1ea12d0, h=...) at /usr/include/boost/function/function_template.hpp:1013 #11 qpid::sys::posix::AsynchIO::writeable (this=0x1ea12d0, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536 #12 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #13 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:289 #14 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123 #15 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483 #16 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #17 0x00000031d6c07951 in start_thread (arg=0x7f2887fff710) at pthread_create.c:301 ---Type <return> to continue, or q <return> to quit--- #18 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 7 (Thread 28615): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f288d745710) at pthread_create.c:301 #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 6 (Thread 28614): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f288e146710) at pthread_create.c:301 #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 5 (Thread 28609): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:212 #1 0x00007f2890b97293 in wait (this=0x1dcdf70) at ../../include/qpid/sys/posix/Condition.h:69 #2 wait (this=0x1dcdf70) at ../../include/qpid/sys/Monitor.h:45 #3 qpid::sys::Timer::run (this=0x1dcdf70) at ../../src/qpid/sys/Timer.cpp:155 #4 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #5 0x00000031d6c07951 in start_thread (arg=0x7f289097b710) at pthread_create.c:301 #6 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 4 (Thread 28616): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f288cd44710) at pthread_create.c:301 #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 3 (Thread 28613): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f287e146710) at pthread_create.c:301 ---Type <return> to continue, or q <return> to quit--- #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 2 (Thread 28612): #0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479 #3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000031d6c07951 in start_thread (arg=0x7f288eb47710) at pthread_create.c:301 #5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 1 (Thread 28611): #0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00000031d6835135 in abort () at abort.c:92 #2 0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:186 #3 0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280 #4 0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42 #5 intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57 #6 ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101 #7 qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46 #8 0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115 #9 pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237 #10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96 #11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214 #12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013 #13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536 #14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:289 #16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123 #17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483 #18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35 #19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301 #20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb)
This is essentially a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=589706, tracked here for MRG.
Fixed in glibc-2.12-1.1.el6.1 *** This bug has been marked as a duplicate of bug 589706 ***