Bug 501668 - perftest --unique-data yes --qt 1 --durable yes --npubs 2 --nsubs 1 -tx 0 --async-commit no hangs when run against standalone broker
perftest --unique-data yes --qt 1 --durable yes --npubs 2 --nsubs 1 -tx 0 --a...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.1.1
All Linux
medium Severity medium
: 1.3
: ---
Assigned To: Gordon Sim
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-20 05:04 EDT by Frantisek Reznicek
Modified: 2015-11-15 19:07 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-04-30 06:08:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Detailed perftest and qpidd program stack traces (94.65 KB, application/x-tbz)
2009-05-20 05:04 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2009-05-20 05:04:02 EDT
Created attachment 344761 [details]
Detailed perftest and qpidd program stack traces

Description of problem:

There are hangs observed when a perftest is run against standalove broker when
perftest and qpidd are launched following ways (3 cases):

A] intel-d3c4702-01.rhts.bos.redhat.com RHEL4-U7 x86_64

  qpidd:
    /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG_Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20090520_012812/b.9660 --port 41563 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
  perftest:
    perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 41563 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 400000 --size 128 --tx 0 --async-commit no

B] gs-intel-s3ea2-01.rhts.bos.redhat.com RHEL5-Server-U3 x86_64

  qpidd:
    /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG_Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20090519_221146/b.4850 --port 43545 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
  perftest:
    perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 43545 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no

C] intel-s3ea2-03.rhts.bos.redhat.com RHEL5-Server-U3 i386

  qpidd:
    /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG_Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20090519_224306/b.2405 --port 42224 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
  perftest:
    perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 42224 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no


Please find attached pstack and backtrace of qpidd and perftest in the hang situation.


Version-Release number of selected component (if applicable):
[root@intel-s3ea2-03 ~]# rpm -qa | egrep '(qpid|rhm|openais)' | sort -u
openais-0.80.3-22.el5_3.4
python-qpid-0.5.752581-1.el5
qpidc-0.5.752581-5.el5
qpidc-debuginfo-0.5.752581-5.el5
qpidc-devel-0.5.752581-5.el5
qpidc-perftest-0.5.752581-5.el5
qpidc-rdma-0.5.752581-5.el5
qpidc-ssl-0.5.752581-5.el5
qpidd-0.5.752581-5.el5
qpidd-acl-0.5.752581-5.el5
qpidd-cluster-0.5.752581-5.el5
qpidd-devel-0.5.752581-5.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-5.el5
qpidd-ssl-0.5.752581-5.el5
qpidd-xml-0.5.752581-5.el5
qpid-java-client-0.5.751061-2.el5
qpid-java-common-0.5.751061-2.el5
rhm-0.5.3206-1.el5
rhm-docs-0.5.756148-1.el5


How reproducible:
~90%

Steps to Reproduce:
1. run RHTS test qpid_test_qpidd-perftest_performance or just launch qpidd and perftest with parameters above in loop and watch for hang
  
Actual results:
perftest did not finish basause of hang either in perftest/c++ client library or in broker. Test timeouts 8 hours threshold because of that.

Expected results:
perftest should finish within reasonable time

Additional info:

run 0: 20090520_040813 =====================================================
ps listing for process 24859
  root     24859  0.0  0.7 569156 191252 ?       Sl   May19   0:13 /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG_Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20090519_221146/b.4850 --port 43545 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
pstree -p 24859
  qpidd(24859)-+-{qpidd}(24864)
               |-{qpidd}(24865)
               |-{qpidd}(24866)
               |-{qpidd}(24867)
               |-{qpidd}(24868)
               |-{qpidd}(24869)
               |-{qpidd}(24870)
               |-{qpidd}(24871)
               |-{qpidd}(24872)
               |-{qpidd}(24873)
               |-{qpidd}(24874)
               |-{qpidd}(24875)
               |-{qpidd}(24876)
               |-{qpidd}(24877)
               |-{qpidd}(24878)
               |-{qpidd}(24879)
               |-{qpidd}(24880)
               |-{qpidd}(24881)
               `-{qpidd}(24882)
pstack 24859
  Thread 20 (Thread 0x41b31940 (LWP 24864)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  #1  0x00000030ba787cbf in qpid::broker::Timer::run ()
  #2  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 19 (Thread 0x42532940 (LWP 24865)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  #1  0x00000030ba787cbf in qpid::broker::Timer::run ()
  #2  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 18 (Thread 0x42f33940 (LWP 24866)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  #1  0x00000030ba787cbf in qpid::broker::Timer::run ()
  #2  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 17 (Thread 0x43934940 (LWP 24867)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 16 (Thread 0x44335940 (LWP 24868)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 15 (Thread 0x44d36940 (LWP 24869)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 14 (Thread 0x45737940 (LWP 24870)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 13 (Thread 0x46138940 (LWP 24871)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 12 (Thread 0x46b39940 (LWP 24872)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 11 (Thread 0x4753a940 (LWP 24873)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 10 (Thread 0x47f3b940 (LWP 24874)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 9 (Thread 0x4893c940 (LWP 24875)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 8 (Thread 0x4933d940 (LWP 24876)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 7 (Thread 0x49d3e940 (LWP 24877)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 6 (Thread 0x4a73f940 (LWP 24878)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 5 (Thread 0x4b140940 (LWP 24879)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 4 (Thread 0x4bb41940 (LWP 24880)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 3 (Thread 0x4c542940 (LWP 24881)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 2 (Thread 0x4cf43940 (LWP 24882)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 1 (Thread 0x2b1803a5e7a0 (LWP 24859)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030ba6cc286 in qpid::broker::Broker::run ()
  #4  0x0000000000406948 in ?? ()
  #5  0x0000000000405438 in __cxa_pure_virtual ()
  #6  0x00000030b661d974 in __libc_start_main () from /lib64/libc.so.6
  #7  0x0000000000404eb9 in __cxa_pure_virtual ()
  #8  0x00007fffa7062338 in ?? ()
  #9  0x0000000000000000 in ?? ()
gdb backtrace 24859
  GNU gdb Fedora (6.8-27.el5)
  Copyright (C) 2008 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu".
  (gdb) Attaching to process 24859
  Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done.
  done.
  ...
  Reading symbols from /lib64/libpthread.so.0...done.
  [Thread debugging using libthread_db enabled]
  [New Thread 0x2b1803a5e7a0 (LWP 24859)]
  [New Thread 0x4cf43940 (LWP 24882)]
  [New Thread 0x4c542940 (LWP 24881)]
  [New Thread 0x4bb41940 (LWP 24880)]
  [New Thread 0x4b140940 (LWP 24879)]
  [New Thread 0x4a73f940 (LWP 24878)]
  [New Thread 0x49d3e940 (LWP 24877)]
  [New Thread 0x4933d940 (LWP 24876)]
  [New Thread 0x4893c940 (LWP 24875)]
  [New Thread 0x47f3b940 (LWP 24874)]
  [New Thread 0x4753a940 (LWP 24873)]
  [New Thread 0x46b39940 (LWP 24872)]
  [New Thread 0x46138940 (LWP 24871)]
  [New Thread 0x45737940 (LWP 24870)]
  [New Thread 0x44d36940 (LWP 24869)]
  [New Thread 0x44335940 (LWP 24868)]
  [New Thread 0x43934940 (LWP 24867)]
  [New Thread 0x42f33940 (LWP 24866)]
  [New Thread 0x42532940 (LWP 24865)]
  [New Thread 0x41b31940 (LWP 24864)]
  Loaded symbols for /lib64/libpthread.so.0
  Reading symbols from /lib64/libresolv.so.2...done.
  ...
from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done.
  done.
  Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so
  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  (gdb) 
  Thread 20 (Thread 0x41b31940 (LWP 24864)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00000030ba787cbf in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
  #2  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 19 (Thread 0x42532940 (LWP 24865)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00000030ba787cbf in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
  #2  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 18 (Thread 0x42f33940 (LWP 24866)):
  #0  0x00000030b6e0ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00000030ba787cbf in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
  #2  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #3  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #4  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 17 (Thread 0x43934940 (LWP 24867)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 16 (Thread 0x44335940 (LWP 24868)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 15 (Thread 0x44d36940 (LWP 24869)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 14 (Thread 0x45737940 (LWP 24870)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 13 (Thread 0x46138940 (LWP 24871)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 12 (Thread 0x46b39940 (LWP 24872)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 11 (Thread 0x4753a940 (LWP 24873)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 10 (Thread 0x47f3b940 (LWP 24874)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 9 (Thread 0x4893c940 (LWP 24875)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 8 (Thread 0x4933d940 (LWP 24876)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 7 (Thread 0x49d3e940 (LWP 24877)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 6 (Thread 0x4a73f940 (LWP 24878)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 5 (Thread 0x4b140940 (LWP 24879)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 4 (Thread 0x4bb41940 (LWP 24880)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 3 (Thread 0x4c542940 (LWP 24881)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 2 (Thread 0x4cf43940 (LWP 24882)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #4  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #5  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 1 (Thread 0x2b1803a5e7a0 (LWP 24859)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030ba6cc286 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:319
  #4  0x0000000000406948 in QpiddBroker::execute (this=<value optimized out>, options=0x7fc6d50)
      at posix/QpiddBroker.cpp:165
  #5  0x0000000000405438 in main (argc=23, argv=0x7fffa7062348) at qpidd.cpp:77
  (gdb) quit
  The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
  Detaching from program: /usr/sbin/qpidd, process 24859
ps listing for process 24925
  root     24925  0.0  0.0 351232  7288 ?        Sl   May19   0:06 perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 43545 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no
pstree -p 24925
  perftest(24925)-+-{perftest}(24947)
                  |-{perftest}(24948)
                  `-{perftest}(24949)
pstack 24925
  Thread 4 (Thread 0x4409e940 (LWP 24947)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b9a65edb in qpid::client::TCPConnector::run ()
  #4  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #5  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 3 (Thread 0x44a9f940 (LWP 24948)):
  #0  0x00000030b6e0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
  #1  0x00000030b9a89cda in qpid::client::LocalQueue::get ()
  #2  0x00000030b9a8a4b9 in qpid::client::LocalQueue::get ()
  #3  0x00000030b9a8a779 in qpid::client::LocalQueue::pop ()
  #4  0x0000000000417183 in ?? ()
  #5  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #6  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #7  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 2 (Thread 0x454a0940 (LWP 24949)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait ()
  #2  0x00000030b7775787 in qpid::sys::Poller::run ()
  #3  0x00000030b9a65edb in qpid::client::TCPConnector::run ()
  #4  0x00000030b776c76a in ?? () from /usr/lib64/libqpidcommon.so.0
  #5  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  Thread 1 (Thread 0x2b48e0d8b7d0 (LWP 24925)):
  #0  0x00000030b6e0a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
  #1  0x00000030b9a89cda in qpid::client::LocalQueue::get ()
  #2  0x00000030b9a8a4b9 in qpid::client::LocalQueue::get ()
  #3  0x00000030b9a8a779 in qpid::client::LocalQueue::pop ()
  #4  0x0000000000419703 in ?? ()
  #5  0x0000000000419ee3 in ?? ()
  #6  0x000000000040cabc in qpid::framing::TransferContent::getData ()
  #7  0x00000030b661d974 in __libc_start_main () from /lib64/libc.so.6
  #8  0x000000000040b869 in qpid::framing::TransferContent::getData ()
  #9  0x00007fffc9d36068 in ?? ()
  #10 0x0000000000000000 in ?? ()
gdb backtrace 24925
  GNU gdb Fedora (6.8-27.el5)
  Copyright (C) 2008 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu".
  (gdb) Attaching to process 24925
  Reading symbols from /usr/bin/perftest...Reading symbols from /usr/lib/debug/usr/bin/perftest.debug...done.
  ...
  Loaded symbols for /usr/lib64/sasl2/libplain.so.2
  0x00000030b6e0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  (gdb) 
  Thread 4 (Thread 0x4409e940 (LWP 24947)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b9a65edb in qpid::client::TCPConnector::run (this=<value optimized out>)
      at qpid/client/Connector.cpp:393
  #4  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #5  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 3 (Thread 0x44a9f940 (LWP 24948)):
  #0  0x00000030b6e0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00000030b9a89cda in qpid::client::LocalQueue::get (this=<value optimized out>, 
      result=<value optimized out>, timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
  #2  0x00000030b9a8a4b9 in qpid::client::LocalQueue::get (this=<value optimized out>, 
      timeout=<value optimized out>) at qpid/client/LocalQueue.cpp:40
  #3  0x00000030b9a8a779 in qpid::client::LocalQueue::pop (this=<value optimized out>, 
      timeout=<value optimized out>) at qpid/client/LocalQueue.cpp:36
  #4  0x0000000000417183 in SubscribeThread::run (this=0x2aaaac025df0) at perftest.cpp:611
  #5  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #6  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #7  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 2 (Thread 0x454a0940 (LWP 24949)):
  #0  0x00000030b66d3498 in epoll_wait () from /lib64/libc.so.6
  #1  0x00000030b77749ad in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:432
  #2  0x00000030b7775787 in qpid::sys::Poller::run (this=<value optimized out>)
      at qpid/sys/epoll/EpollPoller.cpp:398
  #3  0x00000030b9a65edb in qpid::client::TCPConnector::run (this=<value optimized out>)
      at qpid/client/Connector.cpp:393
  #4  0x00000030b776c76a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #5  0x00000030b6e06367 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000030b66d30ad in clone () from /lib64/libc.so.6
  
  Thread 1 (Thread 0x2b48e0d8b7d0 (LWP 24925)):
  #0  0x00000030b6e0a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00000030b9a89cda in qpid::client::LocalQueue::get (this=<value optimized out>, 
      result=<value optimized out>, timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
  #2  0x00000030b9a8a4b9 in qpid::client::LocalQueue::get (this=<value optimized out>, 
      timeout=<value optimized out>) at qpid/client/LocalQueue.cpp:40
  #3  0x00000030b9a8a779 in qpid::client::LocalQueue::pop (this=<value optimized out>, 
      timeout=<value optimized out>) at qpid/client/LocalQueue.cpp:36
  #4  0x0000000000419703 in Controller::process (this=<value optimized out>, n=2, lq=
            {queue = {px = 0x7fffc9d35900, pn = {pi_ = 0x419703}}, subscription = {<qpid::client::Handle<qpid::client::SubscriptionImpl>> = {impl = 0x3300000217}, <No data fields>}}, queue=
          {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fffc9d35a50 "XÖ$¬ª*"}}, msgFn=
          {<boost::function1<void,const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&,std::allocator<void> >> = {<boost::function_base> = {manager = 0x7fffc9d358e0, functor = {obj_ptr = 0x7fffc9d35900, const_obj_ptr = 0x7fffc9d35900, func_ptr = 0x7fffc9d35900, data = ""}}, static args = <optimized out>, static arity = <optimized out>, invoker = 0x419703 <Controller::process(unsigned long, qpid::client::LocalQueue, std::string, boost::function<void ()(std::string const&), std::allocator<void> >)+195>}, <No data fields>}) at perftest.cpp:368
  #5  0x0000000000419ee3 in Controller::run (this=0x7fffc9d35da0) at perftest.cpp:408
  #6  0x000000000040cabc in main (argc=<value optimized out>, argv=<value optimized out>) at perftest.cpp:699
  (gdb) quit
  The program is running.  Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
  Detaching from program: /usr/bin/perftest, process 24925
Comment 1 Frantisek Reznicek 2009-08-14 12:08:33 EDT
The issue is still present on 1.1.6 packages on both RHEL 4.7 (4.8) / 5.3.
The issue causes multiple perftest related automated test to hang which causes test report to FAIL.

See RHEL 5.3 x86_64 data:

[root@dell-per905-01 ~]# ps auxw| grep perftest
root      6089  0.0  0.0  58884   548 ?        S    05:03   0:00 tee /mnt/tests/distribution/MRG/Messaging/qpid_test_perftest_topic/qpid_test_perftest_topic.log
root     10571  0.4  0.4 376248 36608 ?        Ssl  05:10   1:48 /usr/sbin/qpidd --auth no --daemon --port 0 --log-enable info+ --log-to-file qpidd.log --data-dir /mnt/tests/distribution/MRG/Messaging/qpid_test_perftest_topic/rhts_qpidd/broker.568 --no-module-dir
root     10594  0.0  0.1 615612 10052 ?        Sl   05:10   0:04 /usr/bin/perftest --port 48048 --mode topic --iterations 23 --count 25000 --size 256 --nsubs 5 --summary
root     10996  0.0  0.0  61144   748 pts/0    S+   11:59   0:00 grep perftest
[root@dell-per905-01 ~]# uptime
 11:59:30 up  7:17,  1 user,  load average: 0.03, 0.05, 0.00
[root@dell-per905-01 ~]# strace -p 10594
Process 10594 attached - interrupt to quit
futex(0x2aaaac037bac, FUTEX_WAIT_PRIVATE, 11, NULL <unfinished ...>
Process 10594 detached
[root@dell-per905-01 ~]# strace -p 10571
Process 10571 attached - interrupt to quit
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], ~[KILL STOP RTMIN RT_1], 8) = 0
epoll_wait(6,  <unfinished ...>
Process 10571 detached
[root@dell-per905-01 ~]# pstack 10594
Thread 14 (Thread 0x416fd940 (LWP 10596)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x42803940 (LWP 10597)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x000000000041a60d in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x43204940 (LWP 10598)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x43c05940 (LWP 10599)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000418f79 in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x44606940 (LWP 10600)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x45007940 (LWP 10601)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000418f79 in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x45a08940 (LWP 10602)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x46409940 (LWP 10603)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000418f79 in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x46e0a940 (LWP 10604)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x4780b940 (LWP 10605)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000418f79 in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x40a38940 (LWP 10606)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4820c940 (LWP 10607)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000418f79 in ?? ()
#5  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x48c0d940 (LWP 10608)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4682ab in qpid::client::TCPConnector::run ()
#4  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2acc8746c7d0 (LWP 10594)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x000000339d489d8a in qpid::client::LocalQueue::get ()
#2  0x000000339d48a569 in qpid::client::LocalQueue::get ()
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop ()
#4  0x0000000000416f86 in ?? ()
#5  0x000000000041787a in ?? ()
#6  0x000000000040d695 in qpid::framing::TransferContent::getData ()
#7  0x000000339981d974 in __libc_start_main () from /lib64/libc.so.6
#8  0x000000000040b889 in qpid::framing::TransferContent::getData ()
#9  0x00007fff236548d8 in ?? ()
#10 0x0000000000000000 in ?? ()
[root@dell-per905-01 ~]# pstack 10571
Thread 20 (Thread 0x420ba940 (LWP 10572)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x000000339d592139 in qpid::broker::Timer::run ()
#2  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x42abb940 (LWP 10573)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x000000339d592139 in qpid::broker::Timer::run ()
#2  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x434bc940 (LWP 10574)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x000000339d592139 in qpid::broker::Timer::run ()
#2  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x43ebd940 (LWP 10575)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x448be940 (LWP 10576)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x452bf940 (LWP 10577)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x45cc0940 (LWP 10578)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x466c1940 (LWP 10579)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x470c2940 (LWP 10580)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x47ac3940 (LWP 10581)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x484c4940 (LWP 10582)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x48ec5940 (LWP 10583)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x498c6940 (LWP 10584)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x4a2c7940 (LWP 10585)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x4acc8940 (LWP 10586)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x4b6c9940 (LWP 10587)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x4c0ca940 (LWP 10588)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4cacb940 (LWP 10589)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4d4cc940 (LWP 10590)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339c573cea in ?? () from /usr/lib64/libqpidcommon.so.0
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b7ea998a7a0 (LWP 10571)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait ()
#2  0x000000339c57dc87 in qpid::sys::Poller::run ()
#3  0x000000339d4c9b9e in qpid::broker::Broker::run ()
#4  0x000000000040917b in ?? ()
#5  0x000000339d50631e in qpid::broker::Daemon::fork ()
#6  0x0000000000406a95 in ?? ()
#7  0x00000000004054a8 in __cxa_pure_virtual ()
#8  0x000000339981d974 in __libc_start_main () from /lib64/libc.so.6
#9  0x0000000000404f29 in __cxa_pure_virtual ()
#10 0x00007fff01136368 in ?? ()
#11 0x0000000000000000 in ?? ()

[root@dell-per905-01 ~]# gdb
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
(gdb) attach 10594
Attaching to process 10594
Reading symbols from /usr/bin/perftest...Reading symbols from /usr/lib/debug/usr/bin/perftest.debug...done.
done.
Reading symbols from /usr/lib64/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2acc8746c7d0 (LWP 10594)]
[New Thread 0x48c0d940 (LWP 10608)]
[New Thread 0x4820c940 (LWP 10607)]
[New Thread 0x40a38940 (LWP 10606)]
[New Thread 0x4780b940 (LWP 10605)]
[New Thread 0x46e0a940 (LWP 10604)]
[New Thread 0x46409940 (LWP 10603)]
[New Thread 0x45a08940 (LWP 10602)]
[New Thread 0x45007940 (LWP 10601)]
[New Thread 0x44606940 (LWP 10600)]
[New Thread 0x43c05940 (LWP 10599)]
[New Thread 0x43204940 (LWP 10598)]
[New Thread 0x42803940 (LWP 10597)]
[New Thread 0x416fd940 (LWP 10596)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.0
Reading symbols from /usr/lib64/libnss3.so...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/rdmaconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/rdmaconnector.so
Reading symbols from /usr/lib64/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib64/librdmawrap.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/librdmawrap.so.0
Reading symbols from /usr/lib64/librdmacm.so.1...done.
Loaded symbols for /usr/lib64/librdmacm.so.1
Reading symbols from /usr/lib64/libibverbs.so.1...done.
Loaded symbols for /usr/lib64/libibverbs.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) info threads
  14 Thread 0x416fd940 (LWP 10596)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  13 Thread 0x42803940 (LWP 10597)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12 Thread 0x43204940 (LWP 10598)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  11 Thread 0x43c05940 (LWP 10599)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 0x44606940 (LWP 10600)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  9 Thread 0x45007940 (LWP 10601)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 0x45a08940 (LWP 10602)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  7 Thread 0x46409940 (LWP 10603)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x46e0a940 (LWP 10604)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  5 Thread 0x4780b940 (LWP 10605)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x40a38940 (LWP 10606)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  3 Thread 0x4820c940 (LWP 10607)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 0x48c0d940 (LWP 10608)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 0x2acc8746c7d0 (LWP 10594)  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) thread apply all bt

Thread 14 (Thread 0x416fd940 (LWP 10596)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x42803940 (LWP 10597)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x000000000041a60d in PublishThread::run (this=0x2aaaac0008c0) at perftest.cpp:501
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x43204940 (LWP 10598)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x43c05940 (LWP 10599)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000418f79 in SubscribeThread::run (this=0x12a2b2a0) at perftest.cpp:602
---Type <return> to continue, or q <return> to quit---
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x44606940 (LWP 10600)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x45007940 (LWP 10601)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000418f79 in SubscribeThread::run (this=0x2aaaac0244f0) at perftest.cpp:602
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x45a08940 (LWP 10602)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x46409940 (LWP 10603)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
---Type <return> to continue, or q <return> to quit---
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000418f79 in SubscribeThread::run (this=0x2aaaac026640) at perftest.cpp:602
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x46e0a940 (LWP 10604)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x4780b940 (LWP 10605)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000418f79 in SubscribeThread::run (this=0x2aaaac039890) at perftest.cpp:602
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x40a38940 (LWP 10606)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x4820c940 (LWP 10607)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000418f79 in SubscribeThread::run (this=0x12c35bf0) at perftest.cpp:602
#5  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#6  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x48c0d940 (LWP 10608)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4682ab in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#4  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2acc8746c7d0 (LWP 10594)):
#0  0x000000339a00a899 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d489d8a in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/posix/Condition.h:63
#2  0x000000339d48a569 in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#3  0x000000339d48a829 in qpid::client::LocalQueue::pop (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:36
#4  0x0000000000416f86 in Controller::process (this=<value optimized out>, n=5, lq=
          {queue = {px = 0x7fff236541b0, pn = {pi_ = 0x416f86}}, subscription = {<qpid::client::Handle<qpid::client::SubscriptionImpl>> = {impl = 0x3300000217}, <No data fields>}}, queue=
        {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fff236542d0 "x\t�\022"}}, msgFn=
        {<boost::function1<void,const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&,std::allocator<void> >> = {<boost::function_base> = {manager = 0x7fff23654190, functor = {obj_ptr = 0x7fff236541b0, const_obj_ptr = 0x7fff236541b0, func_ptr = 0x7fff236541b0, data = "�"}}, static args = <optimized out>, static arity = <optimized out>, invoker = 0x416f86 <Controller::process(unsigned long, qpid::client::LocalQueue, std::string, boost::function<void ()(std::string const&), std::allocator<void> >)+214>}, <No data fields>}) at perftest.cpp:368
#5  0x000000000041787a in Controller::run (this=0x7fff23654610) at perftest.cpp:409
#6  0x000000000040d695 in main (argc=-1340140960, argv=<value optimized out>) at perftest.cpp:699
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/bin/perftest, process 10594
[root@dell-per905-01 ~]# gdb
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
(gdb) attach 10571
Attaching to process 10571
Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done.
done.
Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b7ea998a7a0 (LWP 10571)]
[New Thread 0x4d4cc940 (LWP 10590)]
[New Thread 0x4cacb940 (LWP 10589)]
[New Thread 0x4c0ca940 (LWP 10588)]
[New Thread 0x4b6c9940 (LWP 10587)]
[New Thread 0x4acc8940 (LWP 10586)]
[New Thread 0x4a2c7940 (LWP 10585)]
[New Thread 0x498c6940 (LWP 10584)]
[New Thread 0x48ec5940 (LWP 10583)]
[New Thread 0x484c4940 (LWP 10582)]
[New Thread 0x47ac3940 (LWP 10581)]
[New Thread 0x470c2940 (LWP 10580)]
[New Thread 0x466c1940 (LWP 10579)]
[New Thread 0x45cc0940 (LWP 10578)]
[New Thread 0x452bf940 (LWP 10577)]
[New Thread 0x448be940 (LWP 10576)]
[New Thread 0x43ebd940 (LWP 10575)]
[New Thread 0x434bc940 (LWP 10574)]
[New Thread 0x42abb940 (LWP 10573)]
[New Thread 0x420ba940 (LWP 10572)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
(gdb) info threads
  20 Thread 0x420ba940 (LWP 10572)  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 0x42abb940 (LWP 10573)  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 0x434bc940 (LWP 10574)  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 Thread 0x43ebd940 (LWP 10575)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  16 Thread 0x448be940 (LWP 10576)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  15 Thread 0x452bf940 (LWP 10577)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  14 Thread 0x45cc0940 (LWP 10578)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  13 Thread 0x466c1940 (LWP 10579)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  12 Thread 0x470c2940 (LWP 10580)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  11 Thread 0x47ac3940 (LWP 10581)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  10 Thread 0x484c4940 (LWP 10582)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  9 Thread 0x48ec5940 (LWP 10583)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  8 Thread 0x498c6940 (LWP 10584)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  7 Thread 0x4a2c7940 (LWP 10585)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  6 Thread 0x4acc8940 (LWP 10586)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  5 Thread 0x4b6c9940 (LWP 10587)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  4 Thread 0x4c0ca940 (LWP 10588)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  3 Thread 0x4cacb940 (LWP 10589)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
  2 Thread 0x4d4cc940 (LWP 10590)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 0x2b7ea998a7a0 (LWP 10571)  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 20 (Thread 0x420ba940 (LWP 10572)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d592139 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
#2  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x42abb940 (LWP 10573)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d592139 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
#2  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x434bc940 (LWP 10574)):
#0  0x000000339a00ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000339d592139 in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
#2  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#3  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x43ebd940 (LWP 10575)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x448be940 (LWP 10576)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x452bf940 (LWP 10577)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x45cc0940 (LWP 10578)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x466c1940 (LWP 10579)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x470c2940 (LWP 10580)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x47ac3940 (LWP 10581)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x484c4940 (LWP 10582)):
---Type <return> to continue, or q <return> to quit---
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x48ec5940 (LWP 10583)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x498c6940 (LWP 10584)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x4a2c7940 (LWP 10585)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x4acc8940 (LWP 10586)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 5 (Thread 0x4b6c9940 (LWP 10587)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x4c0ca940 (LWP 10588)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x4cacb940 (LWP 10589)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x4d4cc940 (LWP 10590)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339c573cea in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x000000339a006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033998d30ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b7ea998a7a0 (LWP 10571)):
#0  0x00000033998d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x000000339c57d0dd in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x000000339c57dc87 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x000000339d4c9b9e in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:319
---Type <return> to continue, or q <return> to quit---
#4  0x000000000040917b in QpiddDaemon::child (this=0x7fff01135d80) at posix/QpiddBroker.cpp:128
#5  0x000000339d50631e in qpid::broker::Daemon::fork (this=<value optimized out>) at qpid/broker/Daemon.cpp:85
#6  0x0000000000406a95 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>)
    at posix/QpiddBroker.cpp:159
#7  0x00000000004054a8 in main (argc=13, argv=0x7fff01136378) at qpidd.cpp:77
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/sbin/qpidd, process 10571

[02:52:06] Machine and user info:
[02:52:06]     os: Red Hat Enterprise Linux Server release 5.3 (Tikanga)
[02:52:06]    who:
[02:52:06]  uname: Linux dell-per905-01.rhts.bos.redhat.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
[02:52:06] uptime:  02:52:06 up 16 min,  0 users,  load average: 0.29, 0.47, 0.44
[02:52:06] whoami: root (USER:, LOGNAME:, id:root, HOME:/)
[02:52:06]   date: 2009-08-13 02:52:06 1250146326
[02:52:06]    pwd: /mnt/tests/distribution/MRG/Messaging/qpid_common
[02:52:06]     df: /dev/mapper/VolGroup00-LogVol00   57G  2.2G   51G   5% /
[02:52:06] get_cpu_info():CPU information:
processor       : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
vendor_id       : AuthenticAMD
model name      : Quad-Core AMD Opteron(tm) Processor 8354
cpu MHz         : 1100.000
cpu cores       : 4
bogomips        : 4390.41 4388.54 4388.59 4388.36 4388.22 4387.99 4387.09 4388.49 4388.43 4388.75 4388.39 4388.65 4388.60 4388.75 4388.04 4389.15
[02:52:06] Memory info:
             total       used       free     shared    buffers     cached
Mem:       8238776     996364    7242412          0      60688     754956
-/+ buffers/cache:     180720    8058056
Swap:     10289144          0   10289144


python-qpid-0.5.752581-3.el5
qpidc-0.5.752581-25.el5
qpidc-debuginfo-0.5.752581-25.el5
qpidc-devel-0.5.752581-25.el5
qpidc-perftest-0.5.752581-25.el5
qpidc-rdma-0.5.752581-25.el5
qpidc-ssl-0.5.752581-25.el5
qpidd-0.5.752581-25.el5
qpidd-acl-0.5.752581-25.el5
qpidd-cluster-0.5.752581-25.el5
qpidd-devel-0.5.752581-25.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-25.el5
qpidd-ssl-0.5.752581-25.el5
qpidd-xml-0.5.752581-25.el5
qpid-java-client-0.5.751061-8.el5
qpid-java-common-0.5.751061-8.el5
Comment 2 Frantisek Reznicek 2009-08-14 12:09:57 EDT
Another data analysis from RHEL 4.7 (similar one from 4.8 got as well):




[root@hp-bl460c-02 ~]# ps auxw | grep perftest
root     21114  0.0  0.0 49932  476 ?        S    09:15   0:00 tee /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/qpid_test_qpidd-perftest_performance.log
root     26687  0.1  9.6 317600 198784 ?     Sl   09:22   0:11 /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20090814_091505/b.6678 --port 43356 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
root     26749  0.0  0.0  2540  332 ?        S    09:22   0:00 /usr/bin/time -f %e -o time.transcript.log perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 43356 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no
root     26750  0.0  0.4 134872 9904 ?       Sl   09:22   0:04 perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 43356 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no
root     30325  0.0  0.0 51104  676 pts/0    S+   11:44   0:00 grep perftest


[root@hp-bl460c-02 ~]# pstree -p 26750
perftest(26750)
[root@hp-bl460c-02 ~]# pstree -p 26687
qpidd(26687)

[root@hp-bl460c-02 ~]# strace -p 26750
Process 26750 attached - interrupt to quit
futex(0x2a97101b4c, FUTEX_WAIT, 1, NULL <unfinished ...>
Process 26750 detached
[root@hp-bl460c-02 ~]# strace -p 26687
Process 26687 attached - interrupt to quit
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], ~[KILL STOP RTMIN RT_1], 8) = 0
epoll_wait(6,  <unfinished ...>
Process 26687 detached


[root@hp-bl460c-02 ~]# pstack 26750
Thread 4 (Thread 1126189408 (LWP 26780)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b038d733 in qpid::client::TCPConnector::run ()
#4  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#5  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#6  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 3 (Thread 1136679264 (LWP 26781)):
#0  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00000037b03a7432 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop () from /usr/lib64/libqpidclient.so.0
#2  0x00000037b03b660a in qpid::client::LocalQueue::get ()
#3  0x00000037b03b8147 in qpid::client::LocalQueue::get ()
#4  0x00000037b03ba259 in qpid::client::LocalQueue::pop ()
#5  0x0000000000423bcd in qpid::optValue<bool> ()
#6  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#7  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#8  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 2 (Thread 1147169120 (LWP 26782)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b038d733 in qpid::client::TCPConnector::run ()
#4  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#5  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#6  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 1 (Thread 182894103040 (LWP 26750)):
#0  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00000037b03a7432 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop () from /usr/lib64/libqpidclient.so.0
#2  0x00000037b03b660a in qpid::client::LocalQueue::get ()
#3  0x00000037b03b8147 in qpid::client::LocalQueue::get ()
#4  0x00000037b03ba259 in qpid::client::LocalQueue::pop ()
#5  0x0000000000414cf1 in qpid::client::Message::~Message$delete ()
#6  0x000000000041647a in qpid::client::Message::~Message$delete ()
#7  0x000000000040a3d6 in ?? ()
#8  0x00000037af21c40b in __libc_start_main () from /lib64/tls/libc.so.6
#9  0x00000000004098ea in ?? ()
#10 0x0000007fbffff138 in ?? ()
#11 0x000000000000001c in ?? ()
#12 0x000000000000001c in ?? ()
#13 0x0000007fbffff570 in ?? ()
#14 0x0000007fbffff579 in ?? ()
#15 0x0000007fbffff586 in ?? ()
#16 0x0000007fbffff588 in ?? ()
#17 0x0000007fbffff592 in ?? ()
#18 0x0000007fbffff5a0 in ?? ()
#19 0x0000007fbffff5a4 in ?? ()
#20 0x0000007fbffff5b1 in ?? ()
#21 0x0000007fbffff5b7 in ?? ()
#22 0x0000007fbffff5ba in ?? ()
#23 0x0000007fbffff5c0 in ?? ()
#24 0x0000007fbffff5c7 in ?? ()
#25 0x0000007fbffff5ce in ?? ()
#26 0x0000007fbffff5d3 in ?? ()
#27 0x0000007fbffff5d5 in ?? ()
#28 0x0000007fbffff5df in ?? ()
#29 0x0000007fbffff5e3 in ?? ()
#30 0x0000007fbffff5eb in ?? ()
#31 0x0000007fbffff5ed in ?? ()
#32 0x0000007fbffff5f5 in ?? ()
#33 0x0000007fbffff5f7 in ?? ()
#34 0x0000007fbffff5ff in ?? ()
#35 0x0000007fbffff606 in ?? ()
#36 0x0000007fbffff60d in ?? ()
#37 0x0000007fbffff612 in ?? ()
#38 0x0000007fbffff617 in ?? ()
#39 0x0000007fbffff619 in ?? ()
#40 0x0000007fbffff628 in ?? ()
#41 0x0000000000000000 in ?? ()
[root@hp-bl460c-02 ~]# pstack 26687
Thread 8 (Thread 1084229984 (LWP 26692)):
#0  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x00000037b050338e in qpid::broker::Timer::run ()
#2  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#3  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#4  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 7 (Thread 1094719840 (LWP 26693)):
#0  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x00000037b050338e in qpid::broker::Timer::run ()
#2  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#3  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#4  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 6 (Thread 1105209696 (LWP 26702)):
#0  0x00000037af70af8b in __lll_mutex_lock_wait ()
#1  0x0000000000567698 in ?? ()
#2  0x0000002a962a5158 in vtable for mrg::msgstore::InactivityFireEvent ()
#3  0x00000037af707d54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4  0x0000000000000000 in ?? ()
Thread 5 (Thread 1115699552 (LWP 26703)):
#0  0x00000037af70af8b in __lll_mutex_lock_wait ()
#1  0x00000037b047e9f0 in qpid::broker::IncompleteMessageList::process ()
#2  0x0000002a962a5350 in vtable for mrg::msgstore::JournalImpl ()
#3  0x00000037af708edb in pthread_cond_signal@@GLIBC_2.3.2 ()
#4  0x00000037b047ea2c in qpid::broker::IncompleteMessageList::enqueueComplete
#5  0x00000037b0480071 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::broker::IncompleteMessageList, boost::intrusive_ptr<qpid::broker::Message> const&>, boost::_bi::list2<boost::_bi::value<qpid::broker::IncompleteMessageList*>, boost::arg<1> > >, void, boost::intrusive_ptr<qpid::broker::Message> const&>::invoke ()
#6  0x00000037b047f3a7 in boost::function1<void, boost::intrusive_ptr<qpid::broker::Message> const&, std::allocator<void> >::operator() ()
#7  0x00000037b049e4c9 in qpid::broker::Message::allEnqueuesComplete ()
#8  0x00000037b043a829 in qpid::broker::PersistableMessage::enqueueComplete ()
#9  0x0000002a960e3e55 in mrg::msgstore::JournalImpl::wr_aio_cb ()
#10 0x0000002a9616c0e3 in mrg::journal::wmgr::get_events ()
#11 0x0000002a96167c3f in mrg::journal::wmgr::write_flush ()
#12 0x0000002a9616a24a in mrg::journal::wmgr::dequeue ()
#13 0x0000002a9613e1f5 in mrg::journal::jcntl::dequeue_data_record ()
#14 0x0000002a960eae2a in mrg::msgstore::JournalImpl::dequeue_data_record ()
#15 0x0000002a9610149d in mrg::msgstore::MessageStoreImpl::async_dequeue ()
#16 0x0000002a9610a662 in mrg::msgstore::MessageStoreImpl::dequeue ()
#17 0x00000037b04a64e0 in qpid::broker::MessageStoreModule::dequeue ()
#18 0x00000037b0428428 in qpid::broker::Queue::dequeue ()
#19 0x00000037b04d25f6 in qpid::broker::SemanticState::ConsumerImpl::deliver ()
#20 0x00000037b042ea87 in qpid::broker::Queue::dispatch ()
#21 0x00000037b04c9281 in qpid::broker::SemanticState::ConsumerImpl::doOutput
#22 0x00000037b0115783 in qpid::sys::AggregateOutput::doOutput ()
#23 0x00000037b04d2770 in qpid::broker::SemanticState::doOutput ()
#24 0x00000037b0115783 in qpid::sys::AggregateOutput::doOutput ()
#25 0x00000037b0444769 in qpid::broker::Connection::doOutput ()
#26 0x00000037b03f6752 in qpid::amqp_0_10::Connection::canEncode ()
#27 0x00000037b04c4afa in qpid::broker::SecureConnection::canEncode ()
#28 0x00000037b011748b in qpid::sys::AsynchIOHandler::idle ()
#29 0x00000037b0526e71 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list2<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1> > >, void, qpid::sys::AsynchIO&>::invoke ()
#30 0x00000037b00b5477 in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.0
#31 0x00000037b00b3d63 in qpid::sys::posix::AsynchIO::writeable ()
#32 0x00000037b00b8041 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke ()
#33 0x00000037b011e9e7 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#34 0x00000037b011d4e6 in qpid::sys::DispatchHandle::processEvent ()
#35 0x00000037b00c656c in qpid::sys::Poller::run ()
#36 0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#37 0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#38 0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 4 (Thread 1126189408 (LWP 26704)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 3 (Thread 1136679264 (LWP 26705)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 2 (Thread 1147169120 (LWP 26706)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable ()
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6
Thread 1 (Thread 182894103008 (LWP 26687)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait ()
#2  0x00000037b00c6537 in qpid::sys::Poller::run ()
#3  0x00000037b0400848 in qpid::broker::Broker::run ()
#4  0x000000000040b842 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<char*> ()
#5  0x00000000004085c8 in ?? ()
#6  0x00000037af21c40b in __libc_start_main () from /lib64/tls/libc.so.6
#7  0x000000000040549a in ?? ()
#8  0x0000007fbffff0e8 in ?? ()
#9  0x000000000000001c in ?? ()
#10 0x0000000000000017 in ?? ()
#11 0x0000007fbffff4f7 in ?? ()
#12 0x0000007fbffff507 in ?? ()
#13 0x0000007fbffff512 in ?? ()
#14 0x0000007fbffff57f in ?? ()
#15 0x0000007fbffff586 in ?? ()
#16 0x0000007fbffff58c in ?? ()
#17 0x0000007fbffff593 in ?? ()
#18 0x0000007fbffff596 in ?? ()
#19 0x0000007fbffff5a3 in ?? ()
#20 0x0000007fbffff5a9 in ?? ()
#21 0x0000007fbffff5b6 in ?? ()
#22 0x0000007fbffff5b9 in ?? ()
#23 0x0000007fbffff5ca in ?? ()
#24 0x0000007fbffff5cd in ?? ()
#25 0x0000007fbffff5e0 in ?? ()
#26 0x0000007fbffff5e3 in ?? ()
#27 0x0000007fbffff5f4 in ?? ()
#28 0x0000007fbffff5f7 in ?? ()
#29 0x0000007fbffff60c in ?? ()
#30 0x0000007fbffff60f in ?? ()
#31 0x0000007fbffff626 in ?? ()
#32 0x0000007fbffff629 in ?? ()
#33 0x0000007fbffff637 in ?? ()
#34 0x0000000000000000 in ?? ()







[root@hp-bl460c-02 ~]# gdb
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
(gdb) attach 26750
Attaching to process 26750
Reading symbols from /usr/bin/perftest...Reading symbols from /usr/lib/debug/usr/bin/perftest.debug...done.
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
done.
Reading symbols from /usr/lib64/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.1...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.1
Reading symbols from /usr/lib64/libboost_filesystem.so.1...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.1
Reading symbols from /lib64/tls/librt.so.1...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/tls/libm.so.6...done.
Loaded symbols for /lib64/tls/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894103040 (LWP 26750)]
[New Thread 1147169120 (LWP 26782)]
[New Thread 1136679264 (LWP 26781)]
[New Thread 1126189408 (LWP 26780)]
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.0
Reading symbols from /usr/lib64/libnss3.so...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/sasl2/libcrammd5.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libcrammd5.so.2
Reading symbols from /usr/lib64/sasl2/libdigestmd5.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libdigestmd5.so.2
Reading symbols from /lib64/libcrypto.so.4...done.
Loaded symbols for /lib64/libcrypto.so.4
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
Reading symbols from /usr/lib64/sasl2/libgssapiv2.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libgssapiv2.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
(gdb) info threads
  4 Thread 1126189408 (LWP 26780)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
  3 Thread 1136679264 (LWP 26781)  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  2 Thread 1147169120 (LWP 26782)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
  1 Thread 182894103040 (LWP 26750)  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
(gdb) thread apply all bt

Thread 4 (Thread 1126189408 (LWP 26780)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0xa940b0, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0xa940b0) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00000037b038d733 in qpid::client::TCPConnector::run (this=0xa94110) at qpid/client/Connector.cpp:396
#4  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0xc) at qpid/sys/posix/Thread.cpp:35
#5  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#6  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 3 (Thread 1136679264 (LWP 26781)):
#0  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
#1  0x00000037b03a7432 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0xcba810,
    result=@0x43c04760, timeout=Variable "timeout" is not available.
) at ./qpid/sys/posix/Condition.h:63
#2  0x00000037b03b660a in qpid::client::LocalQueue::get (this=0x43c04ef0, result=@0x43c04a10, timeout={nanosecs = 110491})
    at /usr/include/boost/shared_ptr.hpp:252
#3  0x00000037b03b8147 in qpid::client::LocalQueue::get (this=0x43c04ef0, timeout={nanosecs = 9223372036854775807})
    at qpid/client/LocalQueue.cpp:40
#4  0x00000037b03ba259 in qpid::client::LocalQueue::pop (this=0x0, timeout={nanosecs = 110491}) at qpid/client/LocalQueue.cpp:36
#5  0x0000000000423bcd in SubscribeThread::run (this=0xa920a0) at perftest.cpp:611
#6  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0xcba83c) at qpid/sys/posix/Thread.cpp:35
#7  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#8  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 2 (Thread 1147169120 (LWP 26782)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0x661a90, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0x661a90) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00000037b038d733 in qpid::client::TCPConnector::run (this=0x661940) at qpid/client/Connector.cpp:396
#4  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0xe) at qpid/sys/posix/Thread.cpp:35
#5  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#6  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 1 (Thread 182894103040 (LWP 26750)):
#0  0x00000037af708b3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
#1  0x00000037b03a7432 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=0x2a97101b20,
    result=@0x7fbfffe3a0, timeout=Variable "timeout" is not available.
) at ./qpid/sys/posix/Condition.h:63
#2  0x00000037b03b660a in qpid::client::LocalQueue::get (this=0x7fbfffe8a0, result=@0x7fbfffe580, timeout={nanosecs = 1})
    at /usr/include/boost/shared_ptr.hpp:252
#3  0x00000037b03b8147 in qpid::client::LocalQueue::get (this=0x7fbfffe8a0, timeout={nanosecs = 9223372036854775807})
    at qpid/client/LocalQueue.cpp:40
#4  0x00000037b03ba259 in qpid::client::LocalQueue::pop (this=0x0, timeout={nanosecs = 1}) at qpid/client/LocalQueue.cpp:36
#5  0x0000000000414cf1 in Controller::process (this=Variable "this" is not available.
) at perftest.cpp:368
---Type <return> to continue, or q <return> to quit---
#6  0x000000000041647a in Controller::run (this=0x7fbfffec20) at ../../src/boost/detail/sp_counted_base_gcc_x86.hpp:159
#7  0x000000000040a3d6 in main (argc=Variable "argc" is not available.
) at ./../gen/qpid/client/Session_0_10.h:552
#8  0x00000037af21c40b in __libc_start_main () from /lib64/tls/libc.so.6
#9  0x00000000004098ea in _start ()
#10 0x0000007fbffff138 in ?? ()
#11 0x000000000000001c in ?? ()
#12 0x000000000000001c in ?? ()
#13 0x0000007fbffff570 in ?? ()
#14 0x0000007fbffff579 in ?? ()
#15 0x0000007fbffff586 in ?? ()
#16 0x0000007fbffff588 in ?? ()
#17 0x0000007fbffff592 in ?? ()
#18 0x0000007fbffff5a0 in ?? ()
#19 0x0000007fbffff5a4 in ?? ()
#20 0x0000007fbffff5b1 in ?? ()
#21 0x0000007fbffff5b7 in ?? ()
#22 0x0000007fbffff5ba in ?? ()
#23 0x0000007fbffff5c0 in ?? ()
#24 0x0000007fbffff5c7 in ?? ()
#25 0x0000007fbffff5ce in ?? ()
#26 0x0000007fbffff5d3 in ?? ()
#27 0x0000007fbffff5d5 in ?? ()
#28 0x0000007fbffff5df in ?? ()
#29 0x0000007fbffff5e3 in ?? ()
#30 0x0000007fbffff5eb in ?? ()
#31 0x0000007fbffff5ed in ?? ()
#32 0x0000007fbffff5f5 in ?? ()
#33 0x0000007fbffff5f7 in ?? ()
#34 0x0000007fbffff5ff in ?? ()
#35 0x0000007fbffff606 in ?? ()
#36 0x0000007fbffff60d in ?? ()
#37 0x0000007fbffff612 in ?? ()
#38 0x0000007fbffff617 in ?? ()
#39 0x0000007fbffff619 in ?? ()
#40 0x0000007fbffff628 in ?? ()
#41 0x0000000000000000 in ?? ()


[root@hp-bl460c-02 ~]# gdb
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
(gdb) attach 26687
Attaching to process 26687
Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done.
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
done.
Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.1...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.1
Reading symbols from /usr/lib64/libboost_filesystem.so.1...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.1
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/tls/librt.so.1...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/tls/libm.so.6...done.
Loaded symbols for /lib64/tls/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894103008 (LWP 26687)]
[New Thread 1147169120 (LWP 26706)]
[New Thread 1136679264 (LWP 26705)]
[New Thread 1126189408 (LWP 26704)]
[New Thread 1115699552 (LWP 26703)]
[New Thread 1105209696 (LWP 26702)]
[New Thread 1094719840 (LWP 26693)]
[New Thread 1084229984 (LWP 26692)]
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so
Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/ssl.so
Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.0
Reading symbols from /usr/lib64/libnss3.so...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...done.
Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so
Reading symbols from /usr/lib64/tls/libdb_cxx-4.2.so...done.
Loaded symbols for /usr/lib64/tls/libdb_cxx-4.2.so
Reading symbols from /usr/lib64/libaio.so.1...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/acl.so
Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/xml.so
Reading symbols from /usr/lib64/libxerces-c.so.28...done.
Loaded symbols for /usr/lib64/libxerces-c.so.28
Reading symbols from /usr/lib64/libxqilla.so.3...done.
Loaded symbols for /usr/lib64/libxqilla.so.3
0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
(gdb) info threads
  8 Thread 1084229984 (LWP 26692)  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  7 Thread 1094719840 (LWP 26693)  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  6 Thread 1105209696 (LWP 26702)  0x00000037af70af8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
  5 Thread 1115699552 (LWP 26703)  0x00000037af70af8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
  4 Thread 1126189408 (LWP 26704)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
  3 Thread 1136679264 (LWP 26705)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
  2 Thread 1147169120 (LWP 26706)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
  1 Thread 182894103008 (LWP 26687)  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
(gdb) thread apply all bt

Thread 8 (Thread 1084229984 (LWP 26692)):
#0  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
#1  0x00000037b050338e in qpid::broker::Timer::run (this=0x55e500) at ./qpid/sys/posix/Condition.h:69
#2  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x55e534) at qpid/sys/posix/Thread.cpp:35
#3  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#4  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 7 (Thread 1094719840 (LWP 26693)):
#0  0x00000037af708d2f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
#1  0x00000037b050338e in qpid::broker::Timer::run (this=0x55e5c8) at ./qpid/sys/posix/Condition.h:69
#2  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x55e5fc) at qpid/sys/posix/Thread.cpp:35
#3  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#4  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 6 (Thread 1105209696 (LWP 26702)):
#0  0x00000037af70af8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1  0x0000000000567698 in ?? ()
#2  0x0000002a962a5158 in vtable for mrg::msgstore::InactivityFireEvent () from /usr/lib64/qpid/daemon/msgstore.so
#3  0x00000037af707d54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4  0x0000000000000000 in ?? ()

Thread 5 (Thread 1115699552 (LWP 26703)):
#0  0x00000037af70af8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1  0x00000037b047e9f0 in qpid::broker::IncompleteMessageList::process ()
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/streambuf:499
#2  0x0000002a962a5350 in vtable for mrg::msgstore::JournalImpl () from /usr/lib64/qpid/daemon/msgstore.so
#3  0x00000037af708edb in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
#4  0x00000037b047ea2c in qpid::broker::IncompleteMessageList::enqueueComplete (this=Variable "this" is not available.
) at ./qpid/sys/posix/Condition.h:78
#5  0x00000037b0480071 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::broker::IncompleteMessageList, boost::intrusive_ptr<qpid::broker::Message> const&>, boost::_bi::list2<boost::_bi::value<qpid::broker::IncompleteMessageList*>, boost::arg<1> > >, void, boost::intrusive_ptr<qpid::broker::Message> const&>::invoke (
    function_obj_ptr=Variable "function_obj_ptr" is not available.
) at /usr/include/boost/bind/mem_fn_template.hpp:149
#6  0x00000037b047f3a7 in boost::function1<void, boost::intrusive_ptr<qpid::broker::Message> const&, std::allocator<void> >::operator() (this=0xfffffffffffffffc, a0=@0x0) at /usr/include/boost/function/function_template.hpp:581
#7  0x00000037b049e4c9 in qpid::broker::Message::allEnqueuesComplete (this=0xbc6070) at ./boost/detail/atomic_count_gcc.hpp:47
#8  0x00000037b043a829 in qpid::broker::PersistableMessage::enqueueComplete (this=0xbc6070)
    at qpid/broker/PersistableMessage.cpp:82
#9  0x0000002a960e3e55 in mrg::msgstore::JournalImpl::wr_aio_cb () from /usr/lib64/qpid/daemon/msgstore.so
#10 0x0000002a9616c0e3 in mrg::journal::wmgr::get_events () from /usr/lib64/qpid/daemon/msgstore.so
#11 0x0000002a96167c3f in mrg::journal::wmgr::write_flush () from /usr/lib64/qpid/daemon/msgstore.so
#12 0x0000002a9616a24a in mrg::journal::wmgr::dequeue () from /usr/lib64/qpid/daemon/msgstore.so
#13 0x0000002a9613e1f5 in mrg::journal::jcntl::dequeue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
---Type <return> to continue, or q <return> to quit---
#14 0x0000002a960eae2a in mrg::msgstore::JournalImpl::dequeue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#15 0x0000002a9610149d in mrg::msgstore::MessageStoreImpl::async_dequeue () from /usr/lib64/qpid/daemon/msgstore.so
#16 0x0000002a9610a662 in mrg::msgstore::MessageStoreImpl::dequeue () from /usr/lib64/qpid/daemon/msgstore.so
#17 0x00000037b04a64e0 in qpid::broker::MessageStoreModule::dequeue (this=Variable "this" is not available.
) at qpid/broker/MessageStoreModule.cpp:126
#18 0x00000037b0428428 in qpid::broker::Queue::dequeue (this=0x5af540, ctxt=0x0, msg=Variable "msg" is not available.
) at qpid/broker/Queue.cpp:727
#19 0x00000037b04d25f6 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x6852d0, msg=@0x42801f60)
    at /usr/include/boost/shared_ptr.hpp:252
#20 0x00000037b042ea87 in qpid::broker::Queue::dispatch (this=Variable "this" is not available.
) at /usr/include/boost/shared_ptr.hpp:252
#21 0x00000037b04c9281 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x6852d0)
    at /usr/include/boost/shared_ptr.hpp:252
#22 0x00000037b0115783 in qpid::sys::AggregateOutput::doOutput (this=0x644080)
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462
#23 0x00000037b04d2770 in qpid::broker::SemanticState::doOutput (this=Variable "this" is not available.
) at qpid/broker/SemanticState.h:213
#24 0x00000037b0115783 in qpid::sys::AggregateOutput::doOutput (this=0x643150)
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462
#25 0x00000037b0444769 in qpid::broker::Connection::doOutput (this=0x6430e0) at qpid/broker/Connection.cpp:283
#26 0x00000037b03f6752 in qpid::amqp_0_10::Connection::canEncode (this=0x602510)
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
#27 0x00000037b04c4afa in qpid::broker::SecureConnection::canEncode (this=Variable "this" is not available.
)
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
#28 0x00000037b011748b in qpid::sys::AsynchIOHandler::idle (this=0x601260) at qpid/sys/AsynchIOHandler.cpp:202
#29 0x00000037b0526e71 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list2<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1> > >, void, qpid::sys::AsynchIO&>::invoke (function_obj_ptr=Variable "function_obj_ptr" is not available.
) at /usr/include/boost/bind/mem_fn_template.hpp:149
#30 0x00000037b00b5477 in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (
    this=0xfffffffffffffffc, a0=@0x0) at /usr/include/boost/function/function_template.hpp:581
#31 0x00000037b00b3d63 in qpid::sys::posix::AsynchIO::writeable (this=0x6021f0, h=@0x6021f8) at qpid/sys/posix/AsynchIO.cpp:561
#32 0x00000037b00b8041 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=Variable "function_obj_ptr" is not available.
) at /usr/include/boost/bind/mem_fn_template.hpp:149
#33 0x00000037b011e9e7 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator()
    (this=0xfffffffffffffffc, a0=@0x0) at /usr/include/boost/function/function_template.hpp:581
#34 0x00000037b011d4e6 in qpid::sys::DispatchHandle::processEvent (this=0x6021f8, type=qpid::sys::Poller::INVALID)
    at qpid/sys/DispatchHandle.cpp:447
#35 0x00000037b00c656c in qpid::sys::Poller::run (this=0x556620) at ./qpid/sys/Poller.h:122
#36 0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x5ac328) at qpid/sys/posix/Thread.cpp:35
#37 0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#38 0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 4 (Thread 1126189408 (LWP 26704)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0x556620, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0x556620) at qpid/sys/epoll/EpollPoller.cpp:405
---Type <return> to continue, or q <return> to quit---
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 3 (Thread 1136679264 (LWP 26705)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0x556620, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0x556620) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 2 (Thread 1147169120 (LWP 26706)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0x556620, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0x556620) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00000037b00ba5da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at qpid/sys/posix/Thread.cpp:35
#4  0x00000037af706137 in start_thread () from /lib64/tls/libpthread.so.0
#5  0x00000037af2c9883 in clone () from /lib64/tls/libc.so.6

Thread 1 (Thread 182894103008 (LWP 26687)):
#0  0x00000037af2c9c5c in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000037b00c4f8c in qpid::sys::Poller::wait (this=0x556620, timeout=Variable "timeout" is not available.
) at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00000037b00c6537 in qpid::sys::Poller::run (this=0x556620) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00000037b0400848 in qpid::broker::Broker::run (this=Variable "this" is not available.
) at qpid/broker/Broker.cpp:319
#4  0x000000000040b842 in QpiddBroker::execute (this=Variable "this" is not available.
) at /usr/include/boost/intrusive_ptr.hpp:119
#5  0x00000000004085c8 in main (argc=23, argv=0x7fbffff0f8)
    at /usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
#6  0x00000037af21c40b in __libc_start_main () from /lib64/tls/libc.so.6
#7  0x000000000040549a in _start ()
#8  0x0000007fbffff0e8 in ?? ()
#9  0x000000000000001c in ?? ()
#10 0x0000000000000017 in ?? ()
#11 0x0000007fbffff4f7 in ?? ()
#12 0x0000007fbffff507 in ?? ()
#13 0x0000007fbffff512 in ?? ()
#14 0x0000007fbffff57f in ?? ()
#15 0x0000007fbffff586 in ?? ()
#16 0x0000007fbffff58c in ?? ()
#17 0x0000007fbffff593 in ?? ()
#18 0x0000007fbffff596 in ?? ()
#19 0x0000007fbffff5a3 in ?? ()
#20 0x0000007fbffff5a9 in ?? ()
---Type <return> to continue, or q <return> to quit---
#21 0x0000007fbffff5b6 in ?? ()
#22 0x0000007fbffff5b9 in ?? ()
#23 0x0000007fbffff5ca in ?? ()
#24 0x0000007fbffff5cd in ?? ()
#25 0x0000007fbffff5e0 in ?? ()
#26 0x0000007fbffff5e3 in ?? ()
#27 0x0000007fbffff5f4 in ?? ()
#28 0x0000007fbffff5f7 in ?? ()
#29 0x0000007fbffff60c in ?? ()
#30 0x0000007fbffff60f in ?? ()
#31 0x0000007fbffff626 in ?? ()
#32 0x0000007fbffff629 in ?? ()
#33 0x0000007fbffff637 in ?? ()
#34 0x0000000000000000 in ?? ()
(gdb) The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/sbin/qpidd, process 26687


[09:15:05] Machine and user info:
[09:15:05]     os: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
[09:15:05]    who:
[09:15:05]  uname: Linux hp-bl460c-02.rhts.bos.redhat.com 2.6.9-78.ELsmp #1 SMP Wed Jul 9 15:46:26 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
[09:15:05] uptime:  09:15:05 up 55 min,  0 users,  load average: 0.32, 1.65, 3.30
[09:15:05] whoami: root (USER:, LOGNAME:, id:root, HOME:/)
[09:15:05]   date: 2009-08-14 09:15:05 1250255705
[09:15:05]    pwd: /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance
[09:15:05]     df: /dev/mapper/VolGroup00-LogVol00   30G  3.0G   25G  11% /
[09:15:05] get_cpu_info():CPU information:
processor       : 0 1 2 3
vendor_id       : GenuineIntel
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
cpu MHz         : 3000.113
cpu cores       : 2
bogomips        : 6004.33 6000.25 6000.19 6000.25
[09:15:05] Memory info:
             total       used       free     shared    buffers     cached
Mem:       2057008     448536    1608472          0      22776     314300
-/+ buffers/cache:     111460    1945548
Swap:      4095992          0    4095992


python-qpid-0.5.760500-5.el4
qpidc-0.5.752581-25.el4
qpidc-debuginfo-0.5.752581-25.el4
qpidc-devel-0.5.752581-25.el4
qpidc-perftest-0.5.752581-25.el4
qpidc-ssl-0.5.752581-25.el4
qpidd-0.5.752581-25.el4
qpidd-acl-0.5.752581-25.el4
qpidd-devel-0.5.752581-25.el4
qpid-dotnet-0.4.738274-2.el4
qpidd-ssl-0.5.752581-25.el4
qpidd-xml-0.5.752581-25.el4
qpid-java-client-0.5.751061-8.el4
qpid-java-common-0.5.751061-8.el4
Comment 4 Frantisek Reznicek 2009-10-29 06:51:50 EDT
I'm wondering whether combinations when --npubs does not match --nsubs are valid. Another point where it hangs is --qt 8.

The issue retested on the -26 packages with the same result:
[root@hp-dl2x170g6-02 MRG]# ps auxw | grep perftest
root     10901  0.0  0.0   3708   508 ?        S    Oct28   0:00 tee /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/qpid_test_qpidd-perftest_performance.log
root     18129  0.0  0.0   3912   708 pts/0    S+   06:39   0:00 grep perftest
root     21861  0.0  1.7 196432 109112 ?       Sl   Oct28   0:10 /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20091028_225501/b.1749 --port 42119 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
root     22145  0.0  0.0   1652   336 ?        S    Oct28   0:00 /usr/bin/time -f %e -o time.transcript.log perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 42119 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no
root     22146  0.0  0.1  94820  8364 ?        Sl   Oct28   0:05 perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 42119 --mode shared --qt 1 --durable yes --npubs 2 --nsubs 1 --count 200000 --size 1024 --tx 0 --async-commit no

[root@hp-dl2x170g6-02 MRG]# pstree -apA 22145
time,22145 -f %e -o time.transcript.log perftest --iterations 1 --summary --unique-data yes --log-enableinfo
  `-perftest,22146 --iterations 1 --summary --unique-data yes --log-enable info+ -p 42119 --mode shared ...
      |-{perftest},22194
      |-{perftest},22195
      `-{perftest},22196
[root@hp-dl2x170g6-02 MRG]# pstack 22145
#0  0x00f5b410 in __kernel_vsyscall ()
#1  0x00b89e6e in wait4 () from /lib/libc.so.6
#2  0x00b89e47 in wait3 () from /lib/libc.so.6
#3  0x08049766 in ?? ()
#4  0x08048c75 in ?? ()
#5  0x00b0ee8c in __libc_start_main () from /lib/libc.so.6
#6  0x080488d1 in ?? ()
[root@hp-dl2x170g6-02 MRG]# pstack 22146
Thread 4 (Thread 0xb55fcb90 (LWP 22194)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x00161505 in qpid::client::TCPConnector::run ()
#6  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#7  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#8  0x00bca42e in clone () from /lib/libc.so.6
Thread 3 (Thread 0xb4bfbb90 (LWP 22195)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00c4e595 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00bd6b3d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x001889d1 in qpid::sys::Waitable::wait () from /usr/lib/libqpidclient.so.0
#4  0x00187386 in qpid::client::LocalQueue::get ()
#5  0x00187dae in qpid::client::LocalQueue::get ()
#6  0x001880d7 in qpid::client::LocalQueue::pop ()
#7  0x08061863 in ?? ()
#8  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#9  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#10 0x00bca42e in clone () from /lib/libc.so.6
Thread 2 (Thread 0xb41fab90 (LWP 22196)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x00161505 in qpid::client::TCPConnector::run ()
#6  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#7  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#8  0x00bca42e in clone () from /lib/libc.so.6
Thread 1 (Thread 0xb7feb720 (LWP 22146)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00c4e595 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00bd6b3d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x001889d1 in qpid::sys::Waitable::wait () from /usr/lib/libqpidclient.so.0
#4  0x00187386 in qpid::client::LocalQueue::get ()
#5  0x00187dae in qpid::client::LocalQueue::get ()
#6  0x001880d7 in qpid::client::LocalQueue::pop ()
#7  0x0805bce5 in ?? ()
#8  0x0805f57e in ?? ()
#9  0x08052cbd in qpid::framing::TransferContent::getData ()
#10 0x00b0ee8c in __libc_start_main () from /lib/libc.so.6
#11 0x08051c01 in qpid::framing::TransferContent::getData ()
[root@hp-dl2x170g6-02 MRG]# gdb
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
(gdb) attach 22146
Attaching to process 22146
Reading symbols from /usr/bin/perftest...Reading symbols from /usr/lib/debug/usr/bin/perftest.debug...done.
done.
Reading symbols from /usr/lib/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidclient.so.0
Reading symbols from /usr/lib/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidcommon.so.0
Reading symbols from /usr/lib/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib/libboost_program_options.so.2
Reading symbols from /usr/lib/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib/libboost_filesystem.so.2
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7feb720 (LWP 22146)]
[New Thread 0xb41fab90 (LWP 22196)]
[New Thread 0xb4bfbb90 (LWP 22195)]
[New Thread 0xb55fcb90 (LWP 22194)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/client/sslconnector.so
Reading symbols from /usr/lib/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libsslcommon.so.0
Reading symbols from /usr/lib/libnss3.so...done.
Loaded symbols for /usr/lib/libnss3.so
Reading symbols from /usr/lib/libssl3.so...done.
Loaded symbols for /usr/lib/libssl3.so
Reading symbols from /usr/lib/libnspr4.so...done.
Loaded symbols for /usr/lib/libnspr4.so
Reading symbols from /usr/lib/libnssutil3.so...done.
Loaded symbols for /usr/lib/libnssutil3.so
Reading symbols from /usr/lib/libplc4.so...done.
Loaded symbols for /usr/lib/libplc4.so
Reading symbols from /usr/lib/libplds4.so...done.
Loaded symbols for /usr/lib/libplds4.so
Reading symbols from /usr/lib/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/rdmaconnector.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/client/rdmaconnector.so
Reading symbols from /usr/lib/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib/librdmawrap.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/librdmawrap.so.0
Reading symbols from /usr/lib/librdmacm.so.1...done.
Loaded symbols for /usr/lib/librdmacm.so.1
Reading symbols from /usr/lib/libibverbs.so.1...done.
Loaded symbols for /usr/lib/libibverbs.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib/sasl2/libsasldb.so.2
Reading symbols from /usr/lib/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib/sasl2/libplain.so.2
Reading symbols from /usr/lib/sasl2/libgssapiv2.so.2...done.
Loaded symbols for /usr/lib/sasl2/libgssapiv2.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /lib/libsepol.so.1...done.
Loaded symbols for /lib/libsepol.so.1
Reading symbols from /usr/lib/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib/sasl2/libanonymous.so.2
Reading symbols from /usr/lib/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib/sasl2/liblogin.so.2
0x00d3b410 in __kernel_vsyscall ()
(gdb) info threads
  4 Thread 0xb55fcb90 (LWP 22194)  0x00d3b410 in __kernel_vsyscall ()
  3 Thread 0xb4bfbb90 (LWP 22195)  0x00d3b410 in __kernel_vsyscall ()
  2 Thread 0xb41fab90 (LWP 22196)  0x00d3b410 in __kernel_vsyscall ()
* 1 Thread 0xb7feb720 (LWP 22146)  0x00d3b410 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 4 (Thread 0xb55fcb90 (LWP 22194)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait (this=<value optimized out>, timeout=Could not find the frame base for "qpid::sys::Poller::wait(qpid::sys::Duration)".
)
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00586fa6 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x005d7084 in qpid::sys::Dispatcher::run (this=Could not find the frame base for "qpid::sys::Dispatcher::run()".
) at qpid/sys/Dispatcher.cpp:37
#5  0x00161505 in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#6  0x0057bf01 in runRunnable (p=Could not find the frame base for "runRunnable".
) at qpid/sys/posix/Thread.cpp:35
#7  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#8  0x00bca42e in clone () from /lib/libc.so.6

Thread 3 (Thread 0xb4bfbb90 (LWP 22195)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00c4e595 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00bd6b3d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x001889d1 in qpid::sys::Waitable::wait (this=<value optimized out>) at qpid/sys/posix/Condition.h:63
#4  0x00187386 in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/BlockingQueue.h:55
#5  0x00187dae in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
#6  0x001880d7 in qpid::client::LocalQueue::pop (this=Could not find the frame base for "qpid::client::LocalQueue::pop(qpid::sys::Duration)".
) at qpid/client/LocalQueue.cpp:36
#7  0x08061863 in SubscribeThread::run (this=0xb74028d8) at perftest.cpp:611
#8  0x0057bf01 in runRunnable (p=Could not find the frame base for "runRunnable".
) at qpid/sys/posix/Thread.cpp:35
#9  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#10 0x00bca42e in clone () from /lib/libc.so.6

Thread 2 (Thread 0xb41fab90 (LWP 22196)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait (this=<value optimized out>, timeout=Could not find the frame base for "qpid::sys::Poller::wait(qpid::sys::Duration)".
)
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00586fa6 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x005d7084 in qpid::sys::Dispatcher::run (this=Could not find the frame base for "qpid::sys::Dispatcher::run()".
) at qpid/sys/Dispatcher.cpp:37
#5  0x00161505 in qpid::client::TCPConnector::run (this=<value optimized out>) at qpid/client/Connector.cpp:396
#6  0x0057bf01 in runRunnable (p=Could not find the frame base for "runRunnable".
) at qpid/sys/posix/Thread.cpp:35
#7  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#8  0x00bca42e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb7feb720 (LWP 22146)):
#0  0x00d3b410 in __kernel_vsyscall ()
#1  0x00c4e595 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00bd6b3d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x001889d1 in qpid::sys::Waitable::wait (this=<value optimized out>) at qpid/sys/posix/Condition.h:63
#4  0x00187386 in qpid::client::LocalQueue::get (this=<value optimized out>, result=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/BlockingQueue.h:55
#5  0x00187dae in qpid::client::LocalQueue::get (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/client/LocalQueue.cpp:40
---Type <return> to continue, or q <return> to quit---
#6  0x001880d7 in qpid::client::LocalQueue::pop (this=Could not find the frame base for "qpid::client::LocalQueue::pop(qpid::sys::Duration)".
) at qpid/client/LocalQueue.cpp:36
#7  0x0805bce5 in Controller::process (this=0xbfd8b7b0, n=2, lq=
          {queue = {px = 0xbfd8b57c, pn = {pi_ = 0xb7449448}}, subscription = {<qpid::client::Handle<qpid::client::SubscriptionImpl>> = {impl = 0x4}, <No data fields>}}, queue=
        {static npos = 4294967295, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xbfd8b5f0 "\\\224D·t¶Ø¿(¶Ø¿¸·Ø¿Ø·Ø¿Ð·Ø¿Ô·Ø¿dÔ\006\bh]C·dÔ\006\bû\225\032"}}, msgFn=
        {<boost::function1<void,const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&,std::allocator<void> >> = {<boost::function_base> = {manager = 0xbfd8b570, functor = {obj_ptr = 0xbfd8b5f0, const_obj_ptr = 0xbfd8b5f0, func_ptr = 0xbfd8b5f0, data = "ð"}}, static args = <optimized out>, static arity = <optimized out>, invoker = 0xbfd8b57c}, <No data fields>}) at perftest.cpp:368
#8  0x0805f57e in Controller::run (this=0xbfd8b7b0) at perftest.cpp:408
#9  0x08052cbd in main (argc=Cannot access memory at address 0x80
) at perftest.cpp:699
(gdb)
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /usr/bin/perftest, process 22146
[root@hp-dl2x170g6-02 MRG]# ps auxw | grep qpidd
root     10901  0.0  0.0   3708   508 ?        S    Oct28   0:00 tee /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/qpid_test_qpidd-perftest_performance.log
root     20025  0.0  0.0   3912   684 pts/0    S+   06:43   0:00 grep qpidd
root     21861  0.0  1.7 196432 109112 ?       Sl   Oct28   0:10 /usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20091028_225501/b.1749 --port 42119 --auth no --log-enable info+ --num-jfiles 32 --jfile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 32 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 64 --mgmt-enable 0
[root@hp-dl2x170g6-02 MRG]# netstat -nlp | grep qpidd
tcp        0      0 0.0.0.0:42119               0.0.0.0:*                   LISTEN      21861/qpidd
[root@hp-dl2x170g6-02 MRG]# pstree -apA  21861
qpidd,21861 --data-dir/mnt/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd
  |-{qpidd},21866
  |-{qpidd},21867
  |-{qpidd},21876
  |-{qpidd},21877
  |-{qpidd},21878
  |-{qpidd},21879
  `-{qpidd},21880
[root@hp-dl2x170g6-02 MRG]# pstree  21861
qpidd───7*[{qpidd}]
[root@hp-dl2x170g6-02 MRG]# pstack  21861
Thread 8 (Thread 0xb7f18b90 (LWP 21866)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00c4e8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#2  0x00bd6b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x002a747e in qpid::broker::Timer::run () from /usr/lib/libqpidbroker.so.0
#4  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#5  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#6  0x00bca42e in clone () from /lib/libc.so.6
Thread 7 (Thread 0xb7517b90 (LWP 21867)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00c4e8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#2  0x00bd6b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x002a747e in qpid::broker::Timer::run () from /usr/lib/libqpidbroker.so.0
#4  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#5  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#6  0x00bca42e in clone () from /lib/libc.so.6
Thread 6 (Thread 0xb6a18b90 (LWP 21876)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00c4e8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#2  0x00bd6b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x002a747e in qpid::broker::Timer::run () from /usr/lib/libqpidbroker.so.0
#4  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#5  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#6  0x00bca42e in clone () from /lib/libc.so.6
Thread 5 (Thread 0xb6017b90 (LWP 21877)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#6  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#7  0x00bca42e in clone () from /lib/libc.so.6
Thread 4 (Thread 0xb5616b90 (LWP 21878)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#6  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#7  0x00bca42e in clone () from /lib/libc.so.6
Thread 3 (Thread 0xb4c15b90 (LWP 21879)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#6  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#7  0x00bca42e in clone () from /lib/libc.so.6
Thread 2 (Thread 0xb4214b90 (LWP 21880)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x0057bf01 in ?? () from /usr/lib/libqpidcommon.so.0
#6  0x00c4a49b in start_thread () from /lib/libpthread.so.0
#7  0x00bca42e in clone () from /lib/libc.so.6
Thread 1 (Thread 0xb7f19720 (LWP 21861)):
#0  0x007bf410 in __kernel_vsyscall ()
#1  0x00bcaaa6 in epoll_wait () from /lib/libc.so.6
#2  0x00585f79 in qpid::sys::Poller::wait () from /usr/lib/libqpidcommon.so.0
#3  0x00586fa6 in qpid::sys::Poller::run () from /usr/lib/libqpidcommon.so.0
#4  0x005d7084 in qpid::sys::Dispatcher::run ()
#5  0x001ce051 in qpid::broker::Broker::run () from /usr/lib/libqpidbroker.so.0
#6  0x0804dd44 in ?? ()
#7  0x0804c6c7 in __cxa_pure_virtual ()
#8  0x00b0ee8c in __libc_start_main () from /lib/libc.so.6
#9  0x0804c051 in __cxa_pure_virtual ()
Comment 5 Gordon Sim 2010-04-28 13:59:24 EDT
I am trying (unsuccessfully thus far) to reproduce this problem. If I use the size and counts in the description I hit capacity exceptions on the journal even with the sizings described. If I lower the count a little for each size to stay in the limits then the test will happily loop for quite some time and I have not observed any hangs yet.

Any clues? Is this still reproducible for you?
Comment 6 Frantisek Reznicek 2010-04-30 06:01:15 EDT
The issue is still reproducible, but always shows in connection with
- queue policy exception (resource-limit-exceeded: Policy exceeded on)
or
- Enqueue capacity threshold exceeded  exception

After discussion we concluded that in this particular case it is correct behavior, please?

If so, feel free to move bug to CLOSE/NOTABUG, because if confirmed bug has no valid content then.
Comment 7 Gordon Sim 2010-04-30 06:08:57 EDT
That is correct; if the journal or policy limit is exceeded then the subscribers will not receive all expected messages and the test will not complete.

Note You need to log in before you can comment on or make changes to this bug.