Bug 492334 - qpidd+store startup crash in mrg::msgstore::MessageStoreImpl::init()
qpidd+store startup crash in mrg::msgstore::MessageStoreImpl::init()
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.1
All Linux
medium Severity medium
: 1.3
: ---
Assigned To: Kim van der Riet
Frantisek Reznicek
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-26 10:38 EDT by Frantisek Reznicek
Modified: 2015-11-15 19:07 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Rarely, starting the broker with a store while resource starved could cause a start failure. This no longer occurs with this update.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-10-14 12:12:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Frantisek Reznicek 2009-03-26 10:38:38 EDT
Description of problem:
The MRG bug 456272 test (pid_broker_jfail_bz456272) when run in a longer loop triggered qpidd startup crash in mrg::msgstore::MessageStoreImpl::init().

There were multiple machines used for this test and just the less powerful 
athlon3.rhts.bos.redhat.com failed:
[10:19:00]     os: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
[10:19:00]  uname: Linux athlon3.rhts.bos.redhat.com 2.6.9-78.EL #1 Wed Jul 9 15:27:01 EDT 2008 i686 athlon i386 GNU/Linux
[10:19:00] whoami: root (USER:root)
[10:19:00]   date: 2009-03-26 10:19:00 1238077140
[10:19:00]    pwd: /root/qpid_broker_jfail_bz456272
[10:19:00]     df: /dev/mapper/VolGroup00-LogVol00  5.3G  4.7G  322M  94% /
[10:19:00]      w: root     pts/0    dhcp-lab-200.eng Mon05    0.00s  0.74s  0.00s make rb
[10:19:00] get_cpu_info():CPU information:
processor       : 0
vendor_id       : AuthenticAMD
model name      : AMD Athlon(tm) Processor
cpu MHz         : 700.365
bogomips        : 1401.98

The reproducer and the failing data stored here:
mrg3.lab.bos.redhat.com:/root/qpid_broker_jfail_bz456272_fail090326_rhel4.7_i386.tar.bz2
including core files and journals.

Version-Release number of selected component (if applicable):
 qmf-0.5.752581-1.el4
 qmf-devel-0.5.752581-1.el4
 qpidc-0.5.752581-1.el4
 qpidc-debuginfo-0.5.752581-1.el4
 qpidc-devel-0.5.752581-1.el4
 qpidc-perftest-0.5.752581-1.el4
 qpidc-ssl-0.5.752581-1.el4
 qpidd-0.5.752581-1.el4
 qpidd-acl-0.5.752581-1.el4
 qpidd-devel-0.5.752581-1.el4
 qpidd-ssl-0.5.752581-1.el4
 qpidd-xml-0.5.752581-1.el4
 qpid-java-client-0.5.751061-1.el4
 qpid-java-common-0.5.751061-1.el4
 rhm-0.5.3153-1.el4
 rhm-docs-0.5.756148-1.el4


How reproducible:
>50% on athlon3.rhts.bos.redhat.com when qpid_broker_jfail_bz456272 

Steps to Reproduce:
1. change NR_OF_LOOPS=100 in runtest.sh and run pid_broker_jfail_bz456272 test (make rebuild)
2. wait for the failure (pick not much powerfull machine)
  
Actual results:
qpidd crashes during startup.

Expected results:
qpidd should not crash.

Additional info (backtrace):

[root@athlon3 qpid_broker_jfail_bz456272]# gdb `which qpidd` core.3340
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
Program terminated with signal 6, Aborted.
Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidbroker.so.0
Reading symbols from /usr/lib/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidcommon.so.0
Reading symbols from /usr/lib/libboost_program_options.so.1...done.
Loaded symbols for /usr/lib/libboost_program_options.so.1
Reading symbols from /usr/lib/libboost_filesystem.so.1...done.
Loaded symbols for /usr/lib/libboost_filesystem.so.1
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/tls/librt.so.1...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/ssl.so
Reading symbols from /usr/lib/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libsslcommon.so.0
Reading symbols from /usr/lib/libnss3.so...done.
Loaded symbols for /usr/lib/libnss3.so
Reading symbols from /usr/lib/libssl3.so...done.
Loaded symbols for /usr/lib/libssl3.so
Reading symbols from /usr/lib/libnspr4.so...done.
Loaded symbols for /usr/lib/libnspr4.so
Reading symbols from /usr/lib/libnssutil3.so...done.
Loaded symbols for /usr/lib/libnssutil3.so
Reading symbols from /usr/lib/libplc4.so...done.
Loaded symbols for /usr/lib/libplc4.so
Reading symbols from /usr/lib/libplds4.so...done.
Loaded symbols for /usr/lib/libplds4.so
Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/acl.so
Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/xml.so
Reading symbols from /usr/lib/libxerces-c.so.28...done.
Loaded symbols for /usr/lib/libxerces-c.so.28
Reading symbols from /usr/lib/libxqilla.so.3...done.
Loaded symbols for /usr/lib/libxqilla.so.3
Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so
Reading symbols from /usr/lib/qpid/daemon/msgstore.so...done.
Loaded symbols for /usr/lib/qpid/daemon/msgstore.so
Reading symbols from /usr/lib/tls/i686/libdb_cxx-4.2.so...done.
Loaded symbols for /usr/lib/tls/i686/libdb_cxx-4.2.so
Reading symbols from /usr/lib/libaio.so.1...done.
Loaded symbols for /usr/lib/libaio.so.1
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) thread apply all bt

Thread 4 (process 4561):
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x005cdf7c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#2  0x005424de in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
#3  0x003027d6 in qpid::broker::Timer::run (this=0xb7ec60b0) at ./qpid/sys/posix/Condition.h:69
#4  0x00aad521 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
#5  0x005cb3cc in start_thread () from /lib/tls/libpthread.so.0
#6  0x0053596e in clone () from /lib/tls/libc.so.6

Thread 3 (process 4562):
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x005cdf7c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#2  0x005424de in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
#3  0x003027d6 in qpid::broker::Timer::run (this=0x8241748) at ./qpid/sys/posix/Condition.h:69
#4  0x00aad521 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
#5  0x005cb3cc in start_thread () from /lib/tls/libpthread.so.0
#6  0x0053596e in clone () from /lib/tls/libc.so.6

Thread 2 (process 4563):
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x005cdcf6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#2  0x00542497 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
#3  0x00302612 in qpid::broker::Timer::run (this=0x82417c8) at ./qpid/sys/posix/Condition.h:63
#4  0x00aad521 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
#5  0x005cb3cc in start_thread () from /lib/tls/libpthread.so.0
#6  0x0053596e in clone () from /lib/tls/libc.so.6

Thread 1 (process 3340):
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00491825 in raise () from /lib/tls/libc.so.6
#2  0x00493289 in abort () from /lib/tls/libc.so.6
#3  0x0071b1bb in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x00718ed1 in __cxa_call_unexpected () from /usr/lib/libstdc++.so.6
#5  0x00718f06 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x0071904f in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x00c9f9b0 in DbEnv::runtime_error () from /usr/lib/tls/i686/libdb_cxx-4.2.so
#8  0x00ca5433 in DbTxn::abort () from /usr/lib/tls/i686/libdb_cxx-4.2.so
#9  0x00bf8f85 in mrg::msgstore::TxnCtxt::abort () from /usr/lib/qpid/daemon/msgstore.so
#10 0x00bd9558 in mrg::msgstore::MessageStoreImpl::init () from /usr/lib/qpid/daemon/msgstore.so
---Type <return> to continue, or q <return> to quit---
#11 0x00be39dd in mrg::msgstore::MessageStoreImpl::init () from /usr/lib/qpid/daemon/msgstore.so
#12 0x00bab690 in qpid::broker::StorePlugin::earlyInitialize () from /usr/lib/qpid/daemon/msgstore.so
#13 0x001f1d57 in Broker (this=0x8241528, conf=@0x823227c)
    at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
#14 0x08053c54 in QpiddBroker::execute (this=0xbfeb73c7, options=0x8232170) at /usr/include/boost/intrusive_ptr.hpp:62
#15 0x0804fd80 in main (argc=17, argv=0xbfeb7624) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
#16 0x0047edf3 in __libc_start_main () from /lib/tls/libc.so.6
#17 0x0804c4e1 in _start ()
(gdb) backtrace
#0  0x004507a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00491825 in raise () from /lib/tls/libc.so.6
#2  0x00493289 in abort () from /lib/tls/libc.so.6
#3  0x0071b1bb in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x00718ed1 in __cxa_call_unexpected () from /usr/lib/libstdc++.so.6
#5  0x00718f06 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x0071904f in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x00c9f9b0 in DbEnv::runtime_error () from /usr/lib/tls/i686/libdb_cxx-4.2.so
#8  0x00ca5433 in DbTxn::abort () from /usr/lib/tls/i686/libdb_cxx-4.2.so
#9  0x00bf8f85 in mrg::msgstore::TxnCtxt::abort () from /usr/lib/qpid/daemon/msgstore.so
#10 0x00bd9558 in mrg::msgstore::MessageStoreImpl::init () from /usr/lib/qpid/daemon/msgstore.so
#11 0x00be39dd in mrg::msgstore::MessageStoreImpl::init () from /usr/lib/qpid/daemon/msgstore.so
#12 0x00bab690 in qpid::broker::StorePlugin::earlyInitialize () from /usr/lib/qpid/daemon/msgstore.so
#13 0x001f1d57 in Broker (this=0x8241528, conf=@0x823227c)
    at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
#14 0x08053c54 in QpiddBroker::execute (this=0xbfeb73c7, options=0x8232170) at /usr/include/boost/intrusive_ptr.hpp:62
#15 0x0804fd80 in main (argc=17, argv=0xbfeb7624) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
#16 0x0047edf3 in __libc_start_main () from /lib/tls/libc.so.6
#17 0x0804c4e1 in _start ()
(gdb) [root@athlon3 qpid_broker_jfail_bz456272]#

Let's look on #10 and #11 both pointing to mrg::msgstore::MessageStoreImpl::init (). Is that expected?
Comment 1 Kim van der Riet 2009-03-26 11:38:57 EDT
(In reply to comment #0)
 
> Let's look on #10 and #11 both pointing to
> mrg::msgstore::MessageStoreImpl::init (). Is that expected?  

Yes, there are two overloaded init() functions; one calls the other.
Comment 2 Kim van der Riet 2009-03-27 14:04:00 EDT
An initial investigation shows this host is an older machine with limited resources - 750MHz Athlon and 256MB RAM. I suspect that resource starvation lies behind this error, perhaps memory or disk space.

I note that of the three saved journal files, two are incomplete - ie the journal itself is not yet initialized (and no directories for it exist) and the BDB database files are incomplete - ie without exchanges.db, bindings.db etc.

I attempted to recreate the above scenario by restoring these incomplete databases, but no luck.
Comment 3 Kim van der Riet 2009-04-14 10:20:29 EDT
I am going to push this to v.1.2.

I think it needs investigation, but because of the unique circumstances surrounding the machine, I'm also lowering the severity/urgency to medium. This bug should be a reminder to make sure this issue is not forgotten.
Comment 5 Kim van der Riet 2009-10-26 10:47:04 EDT
Now part of v.1.3.

Setting NEEDINFO, have there been any other occurrences of this bug in RHTS?
Comment 6 Frantisek Reznicek 2009-11-05 03:19:24 EST
The test was repeated and some core files generated, but unfortunately all blank (disk space issue).
I'll repeat and keep you updated.
Comment 7 Frantisek Reznicek 2009-11-05 04:55:34 EST
Success I was able to reproduce on virtual machine running RHEL 4.8 x86_64:
Core file: core.13216 generated by /usr/sbin/qpidd -------------------------
core.13216: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13217):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13218):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x55c4b0)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c4e4) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 2 (process 13219):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13216):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13382 generated by /usr/sbin/qpidd -------------------------
core.13382: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13386):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000002a97016148 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 3 (process 13387):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13388):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13382):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13415 generated by /usr/sbin/qpidd -------------------------
core.13415: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13419):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13420):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13421):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13415):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13438 generated by /usr/sbin/qpidd -------------------------
core.13438: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13445):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13446):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13447):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13438):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13653 generated by /usr/sbin/qpidd -------------------------
core.13653: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13657):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000002a97016148 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 3 (process 13658):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13659):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13653):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13768 generated by /usr/sbin/qpidd -------------------------
core.13768: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13773):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13774):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13775):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13768):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()


Let me know if you need more data.
Comment 8 Frantisek Reznicek 2009-11-05 04:57:21 EST
Above posted corefiles are on virtual machine (kvm, F8) using following packages:
[root@rhel48x qpid_broker_jfail_bz456272]# rpm -qa | egrep '(qpid|rhm)' | sort
python-qpid-0.5.760500-6.el4
qpidc-0.5.752581-32.el4
qpidc-debuginfo-0.5.752581-32.el4
qpidc-devel-0.5.752581-32.el4
qpidc-perftest-0.5.752581-32.el4
qpidc-ssl-0.5.752581-32.el4
qpidd-0.5.752581-32.el4
qpidd-acl-0.5.752581-32.el4
qpidd-devel-0.5.752581-32.el4
qpid-dotnet-0.4.738274-2.el4
qpidd-ssl-0.5.752581-32.el4
qpidd-xml-0.5.752581-32.el4
qpid-java-client-0.5.751061-9.el4
qpid-java-common-0.5.751061-9.el4
rhm-0.5.3206-23.el4
rhm-debuginfo-0.5.3206-23.el4
rhm-docs-0.5.756148-1.el4
rh-qpid-tests-0.5.752581-32.el4
Comment 9 Frantisek Reznicek 2009-11-20 09:59:24 EST
Another occurence on qpidc*-34 is here:
https://rhts.redhat.com/testlogs/2009/11/102239/290324/2364024/TESTOUT.log
see bottom of the test log (RHEL 4.8 i386)
Comment 10 Kim van der Riet 2010-03-09 08:49:54 EST
This seems to be an exception thrown within an exception. The stack traces above show that abort() is being called inside init(), this occurs only within the exception handling part of init(). Thus, it seems that abort() itself is also throwing from within db4, and is not being handled or logged.

It seems this error occurs only on RHEL4, and is likely related to resource starvation, but the exact cause is not known. Code inspection rules out abort on a transaction that has not begun, however.

I have added some error logging statements to help identify which of the three catch()s in init() is responsible for the abort. I need to see some broker logs associated with this error.

Setting needinfo.
Comment 11 Kim van der Riet 2010-03-09 09:13:40 EST
svn r.3863 for Comment #10 above.
Comment 12 Frantisek Reznicek 2010-05-07 09:46:49 EDT
I'm sorry I cannot move this forward because of bug 587671.
Test which is proved to trigger the issue is not working because of (most probably) missing property in python API (difference last stable vs candidate version of python-qpid)

Waiting for bug 587671 or at least workaround.

keeping NEEDINFO
Comment 13 Frantisek Reznicek 2010-05-12 07:55:10 EDT
The test over last 30 hours found no occurrence of the problem parallel run on RHEL 4.8 / 5.5 i386 / x86_64 and qpid-cpp-*0.7.935473-1.el[45] package sets.
Comment 14 Frantisek Reznicek 2010-06-08 11:02:07 EDT
There was extensive testing done to trigger it again without success.
Last 8 runs on supported OS / archs prove that the issue is not able to reach. 

Verified on package[s]: qpid-cpp-server-store-0.7.946106-2.el5

Discussion with Kim resulted in changing state...

-> VERIFIED
Comment 15 Kim van der Riet 2010-10-05 11:13:39 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
This bug was not strictly fixed, it is now unreproducible. It is possible that the conditions that allowed it to occur have been precluded by more recent changes to the code.

Cause: Unknown, resource starvation on RHEL-4 when starting broker with store is suspected
Consequence: Broker startup fails with core
Fix: None at this time
Result: This bug is no longer reproducible.
Comment 17 Douglas Silas 2010-10-05 15:27:38 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,6 +1 @@
-This bug was not strictly fixed, it is now unreproducible. It is possible that the conditions that allowed it to occur have been precluded by more recent changes to the code.
+Rarely, starting the broker with a store could have caused resource starvation. This no longer occurs with this update.-
-Cause: Unknown, resource starvation on RHEL-4 when starting broker with store is suspected
-Consequence: Broker startup fails with core
-Fix: None at this time
-Result: This bug is no longer reproducible.
Comment 18 Kim van der Riet 2010-10-06 06:57:31 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Rarely, starting the broker with a store could have caused resource starvation. This no longer occurs with this update.+Rarely, starting the broker with a store while resource starved could cause a start failure. This no longer occurs with this update.
Comment 20 errata-xmlrpc 2010-10-14 12:12:29 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html

Note You need to log in before you can comment on or make changes to this bug.