Bug 587694
| Summary: | perftest fails on RHEL-6 with glibc error "invalid fastbin entry (free)" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Kim van der Riet <kim.vdriet> | ||||||
| Component: | qpid-cpp | Assignee: | Gordon Sim <gsim> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | MRG Quality Engineering <mrgqe-bugs> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | Development | ||||||||
| Target Milestone: | 1.3 | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2010-05-19 18:33:53 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 589706 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
See attachment for glibc error in full and core file backtrace. Created attachment 410561 [details]
Second occurrence - glibc error and core backtrace
Similar error seen on broker (without store):
Program terminated with signal 6, Aborted.
#0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00000031d6835135 in abort () at abort.c:92
#2 0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3 0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280
#4 0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42
#5 intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57
#6 ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
#7 qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46
#8 0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115
#9 pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237
#10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96
#11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214
#12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013
#13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
at /usr/include/boost/function/function_template.hpp:1013
#15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
at ../../src/qpid/sys/DispatchHandle.cpp:289
#16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301
#20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) threads apply all bt
Undefined command: "threads". Try "help".
(gdb) thread apply all bt
Thread 10 (Thread 28610):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f288ff49710) at pthread_create.c:301
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 9 (Thread 28594):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890f6c8b2 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:338
#4 0x000000000040d83e in QpiddBroker::execute (this=<value optimized out>, options=0x1dc9630) at ../../src/posix/QpiddBroker.cpp:176
#5 0x000000000040a2a2 in main (argc=5, argv=0x7fffd48070b8) at ../../src/qpidd.cpp:80
Thread 8 (Thread 28617):
#0 0x00007f2890f65e65 in release (this=0x7f2887ffe398, __in_chrg=<value optimized out>)
at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143
#1 boost::detail::shared_count::~shared_count (this=0x7f2887ffe398, __in_chrg=<value optimized out>)
at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#2 0x00007f2890fcb93a in ~shared_ptr (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>)
at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
#3 qpid::broker::Queue::getNextMessage (this=0x1ee69d0, m=<value optimized out>, c=<value optimized out>) at ../../src/qpid/broker/Queue.cpp:295
#4 0x00007f2890fcba50 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:391
#5 0x00007f2890fee36a in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x1fb1fd0) at ../../src/qpid/broker/SemanticState.cpp:685
#6 0x00007f2890b8b06d in qpid::sys::AggregateOutput::doOutput (this=0x20353f0) at ../../src/qpid/sys/AggregateOutput.cpp:59
#7 0x00007f2890f7dc83 in qpid::broker::Connection::doOutput (this=0x2035380) at ../../src/qpid/broker/Connection.cpp:282
#8 0x00007f2890f5df6d in qpid::amqp_0_10::Connection::canEncode (this=0x1f29600) at ../../src/qpid/amqp_0_10/Connection.cpp:67
#9 0x00007f2890b8d09b in qpid::sys::AsynchIOHandler::idle (this=0x1f6cdc0) at ../../src/qpid/sys/AsynchIOHandler.cpp:210
#10 0x00007f2890ac9723 in operator() (this=0x1ea12d0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#11 qpid::sys::posix::AsynchIO::writeable (this=0x1ea12d0, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#12 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
at /usr/include/boost/function/function_template.hpp:1013
#13 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
at ../../src/qpid/sys/DispatchHandle.cpp:289
#14 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#15 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#16 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#17 0x00000031d6c07951 in start_thread (arg=0x7f2887fff710) at pthread_create.c:301
---Type <return> to continue, or q <return> to quit---
#18 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 7 (Thread 28615):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f288d745710) at pthread_create.c:301
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 6 (Thread 28614):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f288e146710) at pthread_create.c:301
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 5 (Thread 28609):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:212
#1 0x00007f2890b97293 in wait (this=0x1dcdf70) at ../../include/qpid/sys/posix/Condition.h:69
#2 wait (this=0x1dcdf70) at ../../include/qpid/sys/Monitor.h:45
#3 qpid::sys::Timer::run (this=0x1dcdf70) at ../../src/qpid/sys/Timer.cpp:155
#4 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#5 0x00000031d6c07951 in start_thread (arg=0x7f289097b710) at pthread_create.c:301
#6 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 4 (Thread 28616):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f288cd44710) at pthread_create.c:301
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 3 (Thread 28613):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f287e146710) at pthread_create.c:301
---Type <return> to continue, or q <return> to quit---
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 2 (Thread 28612):
#0 0x00000031d68e5333 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f2890ad36b3 in qpid::sys::Poller::wait (this=0x1dcbfd0, timeout=...) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
#2 0x00007f2890ad40e9 in qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:479
#3 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#4 0x00000031d6c07951 in start_thread (arg=0x7f288eb47710) at pthread_create.c:301
#5 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Thread 1 (Thread 28611):
#0 0x00000031d6833955 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00000031d6835135 in abort () at abort.c:92
#2 0x00000031d68716bb in __libc_message (do_abort=2, fmt=0x31d69489e8 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3 0x00000031d6877076 in malloc_printerr (action=3, str=0x31d6945cc9 "invalid fastbin entry (free)", ptr=<value optimized out>) at malloc.c:6280
#4 0x00007f2890b570e6 in release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:42
#5 intrusive_ptr_release (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/RefCounted.h:57
#6 ~intrusive_ptr (this=0x7f28805202f8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
#7 qpid::framing::AMQFrame::~AMQFrame (this=0x7f28805202f8, __in_chrg=<value optimized out>) at ../../src/qpid/framing/AMQFrame.cpp:46
#8 0x00007f2890f5ec26 in destroy (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/ext/new_allocator.h:115
#9 pop_front (this=0x1f2aed0, buffer=<value optimized out>, size=65536)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.3/../../../../include/c++/4.4.3/bits/stl_deque.h:1237
#10 qpid::amqp_0_10::Connection::encode (this=0x1f2aed0, buffer=<value optimized out>, size=65536) at ../../src/qpid/amqp_0_10/Connection.cpp:96
#11 0x00007f2890b8d0c7 in qpid::sys::AsynchIOHandler::idle (this=0x7f286c002250) at ../../src/qpid/sys/AsynchIOHandler.cpp:214
#12 0x00007f2890ac9723 in operator() (this=0x7f286c000910, h=...) at /usr/include/boost/function/function_template.hpp:1013
#13 qpid::sys::posix::AsynchIO::writeable (this=0x7f286c000910, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:536
#14 0x00007f2890b920f8 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
at /usr/include/boost/function/function_template.hpp:1013
#15 0x00007f2890b913ea in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=qpid::sys::Poller::WRITABLE)
at ../../src/qpid/sys/DispatchHandle.cpp:289
#16 0x00007f2890ad40da in process (this=0x1dcbfd0) at ../../src/qpid/sys/Poller.h:123
#17 qpid::sys::Poller::run (this=0x1dcbfd0) at ../../src/qpid/sys/epoll/EpollPoller.cpp:483
#18 0x00007f2890acc1ba in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at ../../src/qpid/sys/posix/Thread.cpp:35
#19 0x00000031d6c07951 in start_thread (arg=0x7f288f548710) at pthread_create.c:301
#20 0x00000031d68e4d3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)
This is essentially a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=589706, tracked here for MRG. Fixed in glibc-2.12-1.1.el6.1 *** This bug has been marked as a duplicate of bug 589706 *** |
Created attachment 410521 [details] glibc error printout and core file backtrace While running perftest on RHEL-6, I see the following failure reasonably often: *** glibc detected *** /home/kpvdr/qpid/cpp/src/tests/.libs/lt-perftest: invalid fastbin entry (free): 0x00007f924c0013a0 *** Machine: mrg10 OS: RHEL6.0-20100428.n.0_nfs-Server This box has locally checked out and built qpid and store: qpid: svn r.939663 store: svn r.3943 Both modules are built without optimization: make CXXFLAGS="-O0 -ggdb" -j8 Broker start (from src dir): ./qpidd --load-module /home/kpvdr/store/lib/.libs/msgstore.so --data-dir=/tmp --auth no --log-enable info+ Client start (from test dir): ./perftest --iterations 1 --summary --unique-data yes --log-enable info+ --mode topic --qt 1 --durable yes --npubs 2 --nsubs 3 --count 200000 --size 1024 --tx 0 --async-commit no NOTE: This test can also cause a broker failure in the store module (also related to freeing memory) - being treated as a separate bug.