(In reply to comment #0)
 
> Let's look on #10 and #11 both pointing to
> mrg::msgstore::MessageStoreImpl::init (). Is that expected?  

Yes, there are two overloaded init() functions; one calls the other.

An initial investigation shows this host is an older machine with limited resources - 750MHz Athlon and 256MB RAM. I suspect that resource starvation lies behind this error, perhaps memory or disk space.

I note that of the three saved journal files, two are incomplete - ie the journal itself is not yet initialized (and no directories for it exist) and the BDB database files are incomplete - ie without exchanges.db, bindings.db etc.

I attempted to recreate the above scenario by restoring these incomplete databases, but no luck.

I am going to push this to v.1.2.

I think it needs investigation, but because of the unique circumstances surrounding the machine, I'm also lowering the severity/urgency to medium. This bug should be a reminder to make sure this issue is not forgotten.

Now part of v.1.3.

Setting NEEDINFO, have there been any other occurrences of this bug in RHTS?

The test was repeated and some core files generated, but unfortunately all blank (disk space issue).
I'll repeat and keep you updated.

Success I was able to reproduce on virtual machine running RHEL 4.8 x86_64:
Core file: core.13216 generated by /usr/sbin/qpidd -------------------------
core.13216: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13217):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13218):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x55c4b0)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c4e4) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 2 (process 13219):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13216):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13382 generated by /usr/sbin/qpidd -------------------------
core.13382: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13386):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000002a97016148 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 3 (process 13387):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13388):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13382):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13415 generated by /usr/sbin/qpidd -------------------------
core.13415: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13419):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13420):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13421):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13415):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13438 generated by /usr/sbin/qpidd -------------------------
core.13438: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13445):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13446):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13447):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13438):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13653 generated by /usr/sbin/qpidd -------------------------
core.13653: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13657):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000002a97016148 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 3 (process 13658):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13659):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13653):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()

Core file: core.13768 generated by /usr/sbin/qpidd -------------------------
core.13768: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
  Core was generated by `/usr/sbin/qpidd --log-enable info+ --auth no --tpl-wcache-page-size 128 --tpl-j'.
  Program terminated with signal 6, Aborted.
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  Thread 4 (process 13773):
  #0  0x0000003836808d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c099ae in qpid::broker::Timer::run (this=0x2a97016140)
      at ./qpid/sys/posix/Condition.h:69
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2a97016174) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 3 (process 13774):
  #0  0x000000383680b09d in __lll_mutex_unlock_wake ()
     from /lib64/tls/libpthread.so.0
  #1  0x000000000055c4b8 in ?? ()
  #2  0x0000000000000000 in ?? ()
  Thread 2 (process 13775):
  #0  0x0000003836808b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
     from /lib64/tls/libpthread.so.0
  #1  0x0000003781c096ef in qpid::broker::Timer::run (this=0x55c578)
      at ./qpid/sys/posix/Condition.h:63
  #2  0x00000037817bdfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x55c5ac) at qpid/sys/posix/Thread.cpp:35
  #3  0x0000003836806137 in start_thread () from /lib64/tls/libpthread.so.0
  #4  0x00000038355c9f03 in clone () from /lib64/tls/libc.so.6
  Thread 1 (process 13768):
  #0  0x000000383552e26d in raise () from /lib64/tls/libc.so.6
  #1  0x000000383552fa6e in abort () from /lib64/tls/libc.so.6
  #2  0x00000038375b1148 in __gnu_cxx::__verbose_terminate_handler ()
     from /usr/lib64/libstdc++.so.6
  #3  0x00000038375af176 in __cxa_call_unexpected ()
     from /usr/lib64/libstdc++.so.6
  #4  0x00000038375af1a3 in std::terminate () from /usr/lib64/libstdc++.so.6
  #5  0x00000038375af2a3 in __cxa_throw () from /usr/lib64/libstdc++.so.6
  #6  0x0000002a96d4be2a in DbEnv::runtime_error ()
     from /usr/lib64/tls/libdb_cxx-4.2.so
  #7  0x0000002a96d50ec8 in DbTxn::abort () from /usr/lib64/tls/libdb_cxx-4.2.so
  #8  0x0000002a96b90e65 in mrg::msgstore::TxnCtxt::abort ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #9  0x0000002a96b71948 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #10 0x0000002a96b72c46 in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #11 0x0000002a96b7c52b in mrg::msgstore::MessageStoreImpl::init ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #12 0x0000002a96b48ee1 in qpid::broker::StorePlugin::earlyInitialize ()
     from /usr/lib64/qpid/daemon/msgstore.so
  #13 0x0000003781b09dab in Broker (this=0x55c0b0, conf=@0x542d08)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_iterator.h:614
  #14 0x000000000040b791 in QpiddBroker::execute (this=Variable "this" is not available.
      at /usr/include/boost/intrusive_ptr.hpp:62
  #15 0x00000000004085c8 in main (argc=17, argv=0x7fbffff4d8)
      at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #16 0x000000383551c40b in __libc_start_main () from /lib64/tls/libc.so.6
  #17 0x000000000040549a in _start ()
  #18 0x0000007fbffff4c8 in ?? ()
  #19 0x000000000000001c in ?? ()
  #20 0x0000000000000011 in ?? ()
  #21 0x0000007fbffff7ba in ?? ()
  #22 0x0000007fbffff7ca in ?? ()
  #23 0x0000007fbffff7d7 in ?? ()
  #24 0x0000007fbffff7dd in ?? ()
  #25 0x0000007fbffff7e4 in ?? ()
  #26 0x0000007fbffff7e7 in ?? ()
  #27 0x0000007fbffff7fe in ?? ()
  #28 0x0000007fbffff802 in ?? ()
  #29 0x0000007fbffff817 in ?? ()
  #30 0x0000007fbffff81a in ?? ()
  #31 0x0000007fbffff827 in ?? ()
  #32 0x0000007fbffff82a in ?? ()
  #33 0x0000007fbffff83b in ?? ()
  #34 0x0000007fbffff83e in ?? ()
  #35 0x0000007fbffff849 in ?? ()
  #36 0x0000007fbffff89f in ?? ()
  #37 0x0000007fbffff8a6 in ?? ()
  #38 0x0000000000000000 in ?? ()


Let me know if you need more data.

Above posted corefiles are on virtual machine (kvm, F8) using following packages:
[root@rhel48x qpid_broker_jfail_bz456272]# rpm -qa | egrep '(qpid|rhm)' | sort
python-qpid-0.5.760500-6.el4
qpidc-0.5.752581-32.el4
qpidc-debuginfo-0.5.752581-32.el4
qpidc-devel-0.5.752581-32.el4
qpidc-perftest-0.5.752581-32.el4
qpidc-ssl-0.5.752581-32.el4
qpidd-0.5.752581-32.el4
qpidd-acl-0.5.752581-32.el4
qpidd-devel-0.5.752581-32.el4
qpid-dotnet-0.4.738274-2.el4
qpidd-ssl-0.5.752581-32.el4
qpidd-xml-0.5.752581-32.el4
qpid-java-client-0.5.751061-9.el4
qpid-java-common-0.5.751061-9.el4
rhm-0.5.3206-23.el4
rhm-debuginfo-0.5.3206-23.el4
rhm-docs-0.5.756148-1.el4
rh-qpid-tests-0.5.752581-32.el4

Another occurence on qpidc*-34 is here:
https://rhts.redhat.com/testlogs/2009/11/102239/290324/2364024/TESTOUT.log
see bottom of the test log (RHEL 4.8 i386)

This seems to be an exception thrown within an exception. The stack traces above show that abort() is being called inside init(), this occurs only within the exception handling part of init(). Thus, it seems that abort() itself is also throwing from within db4, and is not being handled or logged.

It seems this error occurs only on RHEL4, and is likely related to resource starvation, but the exact cause is not known. Code inspection rules out abort on a transaction that has not begun, however.

I have added some error logging statements to help identify which of the three catch()s in init() is responsible for the abort. I need to see some broker logs associated with this error.

Setting needinfo.

svn r.3863 for Comment #10 above.

I'm sorry I cannot move this forward because of bug 587671.
Test which is proved to trigger the issue is not working because of (most probably) missing property in python API (difference last stable vs candidate version of python-qpid)

Waiting for bug 587671 or at least workaround.

keeping NEEDINFO

The test over last 30 hours found no occurrence of the problem parallel run on RHEL 4.8 / 5.5 i386 / x86_64 and qpid-cpp-*0.7.935473-1.el[45] package sets.

There was extensive testing done to trigger it again without success.
Last 8 runs on supported OS / archs prove that the issue is not able to reach. 

Verified on package[s]: qpid-cpp-server-store-0.7.946106-2.el5

Discussion with Kim resulted in changing state...

-> VERIFIED

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
This bug was not strictly fixed, it is now unreproducible. It is possible that the conditions that allowed it to occur have been precluded by more recent changes to the code.

Cause: Unknown, resource starvation on RHEL-4 when starting broker with store is suspected
Consequence: Broker startup fails with core
Fix: None at this time
Result: This bug is no longer reproducible.

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,6 +1 @@
-This bug was not strictly fixed, it is now unreproducible. It is possible that the conditions that allowed it to occur have been precluded by more recent changes to the code.
+Rarely, starting the broker with a store could have caused resource starvation. This no longer occurs with this update.-
-Cause: Unknown, resource starvation on RHEL-4 when starting broker with store is suspected
-Consequence: Broker startup fails with core
-Fix: None at this time
-Result: This bug is no longer reproducible.

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Rarely, starting the broker with a store could have caused resource starvation. This no longer occurs with this update.+Rarely, starting the broker with a store while resource starved could cause a start failure. This no longer occurs with this update.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html