Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 587694 - perftest fails on RHEL-6 with glibc error "invalid fastbin entry (free)"
perftest fails on RHEL-6 with glibc error "invalid fastbin entry (free)"
Status: CLOSED DUPLICATE of bug 589706
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
All Linux
urgent Severity urgent
: 1.3
: ---
Assigned To: Gordon Sim
MRG Quality Engineering
:
Depends On: 589706
Blocks:
  Show dependency treegraph
 
Reported: 2010-04-30 11:29 EDT by Kim van der Riet
Modified: 2010-05-19 14:33 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-05-19 14:33:53 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
glibc error printout and core file backtrace (60.32 KB, text/plain)
2010-04-30 11:29 EDT, Kim van der Riet
no flags Details
Second occurrence - glibc error and core backtrace (67.39 KB, text/plain)
2010-04-30 14:17 EDT, Kim van der Riet
no flags Details

  None (edit)
Description Kim van der Riet 2010-04-30 11:29:46 EDT
Created attachment 410521 [details]
glibc error printout and core file backtrace

While running perftest on RHEL-6, I see the following failure reasonably often:

*** glibc detected *** /home/kpvdr/qpid/cpp/src/tests/.libs/lt-perftest: invalid fastbin entry (free): 0x00007f924c0013a0 ***

Machine: mrg10
OS: RHEL6.0-20100428.n.0_nfs-Server
This box has locally checked out and built qpid and store:
qpid:  svn r.939663
store: svn r.3943
Both modules are built without optimization: make CXXFLAGS="-O0 -ggdb" -j8

Broker start (from src dir):
./qpidd --load-module /home/kpvdr/store/lib/.libs/msgstore.so --data-dir=/tmp --auth no --log-enable info+

Client start (from test dir):
./perftest --iterations 1 --summary --unique-data yes --log-enable info+ --mode topic --qt 1 --durable yes --npubs 2 --nsubs 3 --count 200000 --size 1024 --tx 0 --async-commit no

NOTE: This test can also cause a broker failure in the store module (also related to freeing memory) - being treated as a separate bug.
Comment 1 Kim van der Riet 2010-04-30 11:31:38 EDT
See attachment for glibc error in full and core file backtrace.
Comment 2 Kim van der Riet 2010-04-30 14:17:10 EDT
Created attachment 410561 [details]
Second occurrence - glibc error and core backtrace
Comment 3 Gordon Sim 2010-05-05 14:35:38 EDT
Similar error seen on broker (without store):

Program terminated with signal 6, Aborted.
#0  0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00000031d6835135 in abort () at abort.c:92
#2  0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3  0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280
#4  0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42
#5  intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57
#6  ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
#7  qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46
#8  0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
    at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115
#9  pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
    at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237
#10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96
#11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214
#12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013
#13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:289
#16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301
#20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) threads apply all bt
Undefined command: "threads".  Try "help".
(gdb) thread apply all bt

Thread 10 (Thread 28610):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f288ff49710) at pthread_create.c:301
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 28594):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890f6c8b2 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:338
#4  0x000000000040d83e in QpiddBroker::execute (this=<value optimized out>, options=0x1dc9630) at ../../src/posix/QpiddBroker.cpp:176
#5  0x000000000040a2a2 in main (argc=5, argv=0x7fffd48070b8) at ../../src/qpidd.cpp:80

Thread 8 (Thread 28617):
#0  0x00007f2890f65e65 in release (this=0x7f2887ffe398, __in_chrg=<value optimized out>)
    at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143
#1  boost::detail::shared_count::~shared_count (this=0x7f2887ffe398, __in_chrg=<value optimized out>)
    at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#2  0x00007f2890fcb93a in ~shared_ptr (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>)
    at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
#3  qpid::broker::Queue::getNextMessage (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>) at ../../src/qpid/broker/Queue.cpp:295
#4  0x00007f2890fcba50 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:391
#5  0x00007f2890fee36a in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x1fb1fd0) at ../../src/qpid/broker/SemanticState.cpp:685
#6  0x00007f2890b8b06d in qpid::sys::AggregateOutput::doOutput (this=0x20353f0) at ../../src/qpid/sys/AggregateOutput.cpp:59
#7  0x00007f2890f7dc83 in qpid::broker::Connection::doOutput (this=0x2035380) at ../../src/qpid/broker/Connection.cpp:282
#8  0x00007f2890f5df6d in qpid::amqp_0_10::Connection::canEncode (this=0x1f29600) at ../../src/qpid/amqp_0_10/Connection.cpp:67
#9  0x00007f2890b8d09b in qpid::sys::AsynchIOHandler::idle (this=0x1f6cdc0) at ../../src/qpid/sys/AsynchIOHandler.cpp:210
#10 0x00007f2890ac9723 in operator() (this=0x1ea12d0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#11 qpid::sys::posix::AsynchIO::writeable (this=0x1ea12d0, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#12 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#13 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:289
#14 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#15 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#16 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#17 0x00000031d6c07951 in start_thread (arg=0x7f2887fff710) at pthread_create.c:301
---Type <return> to continue, or q <return> to quit---
#18 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 28615):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f288d745710) at pthread_create.c:301
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 28614):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f288e146710) at pthread_create.c:301
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 28609):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:212
#1  0x00007f2890b97293 in wait (this=0x1dcdf70) at ../../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0x1dcdf70) at ../../include/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0x1dcdf70) at ../../src/qpid/sys/Timer.cpp:155
#4  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#5  0x00000031d6c07951 in start_thread (arg=0x7f289097b710) at pthread_create.c:301
#6  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 28616):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f288cd44710) at pthread_create.c:301
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 28613):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f287e146710) at pthread_create.c:301
---Type <return> to continue, or q <return> to quit---
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 28612):
#0  0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4  0x00000031d6c07951 in start_thread (arg=0x7f288eb47710) at pthread_create.c:301
#5  0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 28611):
#0  0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00000031d6835135 in abort () at abort.c:92
#2  0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3  0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280
#4  0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42
#5  intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57
#6  ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
#7  qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46
#8  0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
    at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115
#9  pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
    at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237
#10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96
#11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214
#12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013
#13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
    at ../../src/qpid/sys/DispatchHandle.cpp:289
#16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301
#20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)
Comment 4 Gordon Sim 2010-05-12 08:26:17 EDT
This is essentially a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=589706, tracked here for MRG.
Comment 5 Gordon Sim 2010-05-19 14:33:53 EDT
Fixed in glibc-2.12-1.1.el6.1

*** This bug has been marked as a duplicate of bug 589706 ***

Note You need to log in before you can comment on or make changes to this bug.