598948 – qpid c++ multithreaded client (perftest) occasionly hangs in qpid::sys::Thread::join() -> pthread_join() when authenticating GSSAPI

Bug 598948 - qpid c++ multithreaded client (perftest) occasionly hangs in qpid::sys::Thread::join() -> pthread_join() when authenticating GSSAPI

Summary: qpid c++ multithreaded client (perftest) occasionly hangs in qpid::sys::Threa...

Keywords:
Status:	CLOSED DUPLICATE of bug 617267
Alias:	None
Product:	Red Hat Enterprise MRG
Classification:	Red Hat
Component:	qpid-cpp
Sub Component:
Version:	Development
Hardware:	All
OS:	Linux
Priority:	high
Severity:	medium
Target Milestone:	1.3
Target Release:	---
Assignee:	Gordon Sim
QA Contact:	Frantisek Reznicek
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2010-06-02 12:03 UTC by Frantisek Reznicek
Modified:	2015-11-16 01:12 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-12-11 20:41:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
log file for failure case (23.77 KB, text/plain) 2010-07-20 16:14 UTC, Gordon Sim	no flags	Details
View All

Description Frantisek Reznicek 2010-06-02 12:03:53 UTC

Description of problem:

Long run of qpid clients Kerberos5 GSSAPI authentication tests showed that perftest client occasionly hangs at end while joining threads:

(rhel4 case)
  Thread 1 (Thread -1208305888 (LWP 10407)):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00d52497 in pthread_join () from /lib/tls/libpthread.so.0
  #2  0x004159fa in qpid::sys::Thread::join (this=0x8f57c48)
      at /usr/include/boost/shared_ptr.hpp:252
  #3  0x080502c2 in main (argc=11, argv=0xbfefef74)
      at ../../src/boost/ptr_container/detail/void_ptr_iterator.hpp:70
  #4  0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6
  #5  0x0804f281 in _start ()

(rhel5 case)

  Thread 1 (Thread 0xb7f0d720 (LWP 9636)):
  #0  0x00a74410 in __kernel_vsyscall ()
  #1  0x003a4a77 in pthread_join () from /lib/libpthread.so.0
  #2  0x063a4108 in qpid::sys::Thread::join (this=0xb7300540)
      at qpid/sys/posix/Thread.cpp:70
  #3  0x08053790 in main (argc=Cannot access memory at address 0x0) 
      at qpid-perftest.cpp:731


The hang is much more rapidly seen on x86_64 than i386.

The hang seems to appear only in the client which create own threads using libthread lib, because another simple client qc_client does not hang.

qc_client: http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/c++/qc_client.cpp

This hang should be new behavior. (Might be related to bug 580577 fix).


Version-Release number of selected component (if applicable):
python-qpid-0.7.946106-1.el5
python-saslwrapper-0.1.934605-2.el5
qpid-cpp-client-0.7.946106-2.el5
qpid-cpp-client-devel-0.7.946106-2.el5
qpid-cpp-client-devel-docs-0.7.946106-2.el5
qpid-cpp-client-ssl-0.7.946106-2.el5
qpid-cpp-mrg-debuginfo-0.7.946106-2.el5
qpid-cpp-server-0.7.946106-2.el5
qpid-cpp-server-cluster-0.7.946106-2.el5
qpid-cpp-server-devel-0.7.946106-2.el5
qpid-cpp-server-ssl-0.7.946106-2.el5
qpid-cpp-server-store-0.7.946106-2.el5
qpid-cpp-server-xml-0.7.946106-2.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-4.el5
ruby-qpid-0.7.946106-1.el5
ruby-saslwrapper-0.1.934605-2.el5
saslwrapper-0.1.934605-2.el5
saslwrapper-devel-0.1.934605-2.el5

python-qpid-0.7.946106-1.el4
qpid-cpp-client-0.7.946106-2.el4
qpid-cpp-client-devel-0.7.946106-2.el4
qpid-cpp-client-devel-docs-0.7.946106-2.el4
qpid-cpp-client-ssl-0.7.946106-2.el4
qpid-cpp-mrg-debuginfo-0.7.946106-2.el4
qpid-cpp-server-0.7.946106-2.el4
qpid-cpp-server-devel-0.7.946106-2.el4
qpid-cpp-server-ssl-0.7.946106-2.el4
qpid-cpp-server-store-0.7.946106-2.el4
qpid-cpp-server-xml-0.7.946106-2.el4
qpid-dotnet-0.4.738274-2.el4
qpid-java-client-0.7.946106-3.el4
qpid-java-common-0.7.946106-3.el4
qpid-tests-0.7.946106-1.el4
qpid-tools-0.7.946106-4.el4


How reproducible:
~10%

Steps to Reproduce:
1. run qpid_ptest_authentication_krb5
   or mimic http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_ptest_authentication_krb5/runtest.sh?rev=1.61
   or loop 
   perftest --mechanism GSSAPI --count 1 -p ${port} --user ${user} --broker ${host}
   for valid user with/without valid kerberos ticket / invalid user
2. The cases when kerberos ticket is valid and perftest is running with valid user tend to be problematic.
3. hanging perftest should print the summary and then hang
  
Actual results:
Perftest client hangs when authenticating with GSSAPI method.

Expected results:
Perftest should not be hanging.

Additional info:

perftest --mechanism GSSAPI --count 1 -p 51778 --user tpjviZVFQCiQj3gr1u9Rp --broker 10.34.33.55
Processing 1 messages from qpid-perftest_sub_ready . done.
Sending start 1 times to qpid-perftest_pub_start
Processing 1 messages from qpid-perftest_pub_done . done.
Processing 1 messages from qpid-perftest_sub_done . done.

Total 2 transfers of 1024 bytes in 0.047487 seconds.

Publish transfers/sec:
415.973
Average: 415.973

Subscribe transfers/sec:
0.0174084
Average: 0.0174084

Total transfers/sec:      42.1168
Total Mbytes/sec: 0.0411297
controller>process 27837 not found - skipping
controller>/root/MRG/Messaging/qpid_ptest_authentication_krb5/include/functions.inc: line 56: 27834 Terminated              ( $*; echo $? > ${ecode_fn} )
2010-06-02 07:47:30 warning Connection closed
2010-06-02 07:47:30 warning Connection closed
SubscribeThread exception: Connection closed
PublishThread exception: Connection closed


GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) Attaching to process 27836
Reading symbols from /usr/bin/qpid-perftest...Reading symbols from /usr/lib/debug/usr/bin/qpid-perftest.debug...warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
warning: section .dynbss not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
done.
done.
Reading symbols from /usr/lib64/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.2
Reading symbols from /usr/lib64/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.2
Reading symbols from /usr/lib64/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_program_options.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_filesystem.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x43493940 (LWP 27842)]
[New Thread 0x42a92940 (LWP 27841)]
[New Thread 0x42091940 (LWP 27840)]
[New Thread 0x40c31940 (LWP 27839)]
[New Thread 0x41690940 (LWP 27837)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/sasl2/libgssapiv2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libgssapiv2.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
warning: section .gnu.liblist not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
warning: section .dynbss not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
warning: section .gnu.conflict not found in /usr/lib/debug/usr/bin/qpid-perftest.debug
0x00000036a1c07b35 in pthread_join () from /lib64/libpthread.so.0
(gdb) rax            0xfffffffffffffffc -4
rbx            0x40c31940       1086527808
rcx            0xffffffffffffffff       -1
rdx            0x6cbf   27839
rsi            0x0      0
rdi            0x40c319d0       1086527952
rbp            0x2b3c8c408870   0x2b3c8c408870
rsp            0x7fff8b3ead50   0x7fff8b3ead50
r8             0x0      0
r9             0x2aaaac006b58   46912518515544
r10            0x0      0
r11            0x246    582
r12            0x0      0
r13            0x1      1
r14            0x0      0
r15            0x1      1
rip            0x36a1c07b35     0x36a1c07b35 <pthread_join+293>
eflags         0x246    [ PF ZF IF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
fctrl          0x37f    895
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]
(gdb) Using memory regions provided by the target.
There are no memory regions defined.
(gdb) From                To                  Syms Read   Shared Object Library
0x00000038c383fef0  0x00000038c38b8a18  Yes (*)     /usr/lib64/libqpidclient.so.2
0x00000038c330ee00  0x00000038c340f4f8  Yes (*)     /usr/lib64/libqpidcommon.so.2
0x0000003674010aa0  0x000000367402dae8  Yes (*)     /usr/lib64/libboost_program_options.so.2
0x0000003674404810  0x000000367440cff8  Yes (*)     /usr/lib64/libboost_filesystem.so.2
0x00000036a1800e10  0x00000036a1801a08  Yes (*)     /lib64/libdl.so.2
0x00000036a2402220  0x00000036a2405cc8  Yes (*)     /lib64/librt.so.1
0x00000036b66046e0  0x00000036b6613be8  Yes (*)     /usr/lib64/libsasl2.so.2
0x00000036b5e01500  0x00000036b5e02918  Yes (*)     /lib64/libuuid.so.1
0x00000036b3a4f430  0x00000036b3ac3058  Yes (*)     /usr/lib64/libstdc++.so.6
0x00000036a1403e60  0x00000036a1443e38  Yes (*)     /lib64/libm.so.6
0x00000036af201e50  0x00000036af20b018  Yes (*)     /lib64/libgcc_s.so.1
0x00000036a101d780  0x00000036a1109ff8  Yes (*)     /lib64/libc.so.6
0x00000036a0c00a70  0x00000036a0c1671e  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00000036a1c051f0  0x00000036a1c10258  Yes (*)     /lib64/libpthread.so.0
0x00000036a54032a0  0x00000036a540e2d8  Yes (*)     /lib64/libresolv.so.2
0x00000036b1e009f0  0x00000036b1e06918  Yes (*)     /lib64/libcrypt.so.1
0x00002b3c8c4139f0  0x00002b3c8c4202f8  Yes (*)     /usr/lib64/qpid/client/sslconnector.so
0x00002b3c8c648640  0x00002b3c8c65f0a8  Yes (*)     /usr/lib64/libsslcommon.so.2
0x00000030b8c183b0  0x00000030b8cf6f08  Yes (*)     /usr/lib64/libnss3.so
0x00000030b98085e0  0x00000030b982b638  Yes (*)     /usr/lib64/libssl3.so
0x00000030b800cf30  0x00000030b802b738  Yes (*)     /usr/lib64/libnspr4.so
0x00000030b9008340  0x00000030b9012c38  Yes (*)     /usr/lib64/libnssutil3.so
0x00000030b8401370  0x00000030b8402978  Yes (*)     /usr/lib64/libplc4.so
0x00000030b8800e10  0x00000030b8801c08  Yes (*)     /usr/lib64/libplds4.so
0x00000036a2001fd0  0x00000036a200cac8  Yes (*)     /usr/lib64/libz.so.1
0x00002aaaaaaad700  0x00002aaaaaab1768  Yes (*)     /usr/lib64/sasl2/libgssapiv2.so.2
0x0000003515e09200  0x0000003515e26e48  Yes (*)     /usr/lib64/libgssapi_krb5.so.2
0x0000003516219160  0x000000351627df88  Yes (*)     /usr/lib64/libkrb5.so.3
0x00000035156056b0  0x00000035156191f8  Yes (*)     /usr/lib64/libk5crypto.so.3
0x00000036a6800ad0  0x00000036a68011b8  Yes (*)     /lib64/libcom_err.so.2
0x0000003515a02280  0x0000003515a06068  Yes (*)     /usr/lib64/libkrb5support.so.0
0x00000036a7400b60  0x00000036a7401018  Yes (*)     /lib64/libkeyutils.so.1
0x00000036a2c04560  0x00000036a2c102c8  Yes (*)     /lib64/libselinux.so.1
0x00000036a2803d40  0x00000036a2830158  Yes (*)     /lib64/libsepol.so.1
0x00002aaaaaccfe60  0x00002aaaaad6d388  Yes (*)     /usr/lib64/sasl2/libsasldb.so.2
0x00002aaaaaf8dfb0  0x00002aaaaaf8fbc8  Yes (*)     /usr/lib64/sasl2/libanonymous.so.2
0x00002aaaab191fb0  0x00002aaaab193d58  Yes (*)     /usr/lib64/sasl2/libplain.so.2
0x00002aaaab395fa0  0x00002aaaab397d08  Yes (*)     /usr/lib64/sasl2/liblogin.so.2
0x00002aaaab621fe0  0x00002aaaab628268  Yes (*)     /lib64/libnss_files.so.2
(*): Shared library is missing debugging information.
(gdb)   6 Thread 0x41690940 (LWP 27837)  0x00000036a10d4108 in epoll_wait ()
   from /lib64/libc.so.6
  5 Thread 0x40c31940 (LWP 27839)  0x00000036a1c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 0x42091940 (LWP 27840)  0x00000036a10d4108 in epoll_wait ()
   from /lib64/libc.so.6
  3 Thread 0x42a92940 (LWP 27841)  0x00000036a1c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 0x43493940 (LWP 27842)  0x00000036a10d4108 in epoll_wait ()
   from /lib64/libc.so.6
* 1 Thread 0x2b3c8c408870 (LWP 27836)  0x00000036a1c07b35 in pthread_join ()
   from /lib64/libpthread.so.0
(gdb)
Thread 6 (Thread 0x41690940 (LWP 27837)):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x00000038c332c33f in qpid::sys::Poller::wait (this=0xefa6bc0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000038c332cd22 in qpid::sys::Poller::run (this=0xefa6bc0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000038c33238da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x40c31940 (LWP 27839)):
#0  0x00000036a1c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000038c38873bb in pop (this=0xf1b3ce0, result=...,
    timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:63
#2  qpid::client::LocalQueueImpl::get (this=0xf1b3ce0, result=...,
    timeout=<value optimized out>) at qpid/client/LocalQueueImpl.cpp:49
#3  0x00000038c3887c29 in qpid::client::LocalQueueImpl::get (this=0xf1b3ce0,
    timeout=...) at qpid/client/LocalQueueImpl.cpp:40
#4  0x00000038c3887dc9 in qpid::client::LocalQueueImpl::pop (this=0x80,
    timeout=...) at qpid/client/LocalQueueImpl.cpp:36
#5  0x00000038c3884ddc in qpid::client::LocalQueue::pop (
    this=<value optimized out>, timeout=...) at qpid/client/LocalQueue.cpp:43
#6  0x000000000041c554 in qpid::tests::PublishThread::run (this=0x2aaaac000db0)
    at qpid-perftest.cpp:515
#7  0x00000038c33238da in qpid::sys::(anonymous namespace)::runRunnable (
    p=0xefaa2fc) at qpid/sys/posix/Thread.cpp:35
#8  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#9  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x42091940 (LWP 27840)):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x00000038c332c33f in qpid::sys::Poller::wait (this=0xefa6bc0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000038c332cd22 in qpid::sys::Poller::run (this=0xefa6bc0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000038c33238da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x42a92940 (LWP 27841)):
#0  0x00000036a1c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00000038c38873bb in pop (this=0x2aaaac0072a0, result=...,
    timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:63
#2  qpid::client::LocalQueueImpl::get (this=0x2aaaac0072a0, result=...,
    timeout=<value optimized out>) at qpid/client/LocalQueueImpl.cpp:49
#3  0x00000038c3887c29 in qpid::client::LocalQueueImpl::get (
    this=0x2aaaac0072a0, timeout=...) at qpid/client/LocalQueueImpl.cpp:40
#4  0x00000038c3887dc9 in qpid::client::LocalQueueImpl::pop (this=0x80,
    timeout=...) at qpid/client/LocalQueueImpl.cpp:36
#5  0x00000038c3884ddc in qpid::client::LocalQueue::pop (
    this=<value optimized out>, timeout=...) at qpid/client/LocalQueue.cpp:43
#6  0x000000000041b0e2 in qpid::tests::SubscribeThread::run (
    this=0x2aaaac0022d0) at qpid-perftest.cpp:626
#7  0x00000038c33238da in qpid::sys::(anonymous namespace)::runRunnable (
    p=0x2aaaac00795c) at qpid/sys/posix/Thread.cpp:35
#8  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#9  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x43493940 (LWP 27842)):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x00000038c332c33f in qpid::sys::Poller::wait (this=0xefa6bc0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000038c332cd22 in qpid::sys::Poller::run (this=0xefa6bc0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000038c33238da in qpid::sys::(anonymous namespace)::runRunnable (p=0x6)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b3c8c408870 (LWP 27836)):
#0  0x00000036a1c07b35 in pthread_join () from /lib64/libpthread.so.0
#1  0x00000038c3323e2d in qpid::sys::Thread::join (this=<value optimized out>)
    at qpid/sys/posix/Thread.cpp:70
#2  0x000000000040cf39 in main (argc=<value optimized out>,
    argv=<value optimized out>) at qpid-perftest.cpp:731
(gdb) Detaching from program: /usr/bin/qpid-perftest, process 27836
(gdb) quit

Comment 1 Frantisek Reznicek 2010-06-02 12:07:28 UTC

Analyzing the occurrence the hang is seen only when client is expected to succeed the authentication (valid ticket, valid user id).

Comment 2 Gordon Sim 2010-06-04 07:35:32 UTC

The problem I see is that one of the connections occasionally fails to authenticate. This in general causes the test to hang as one or more of the other connections do not receive a message they expect.

This only occurs when the --username option is specified to perftest rather than relying on cyrus to determine the userid used for the kinit. It looks like this is in some circumstances resulting in an incorrect response being sent by the client and the server fails to authenticate it.

Comment 3 Jan Sarenik 2010-06-04 10:28:03 UTC

I am experiencing the same behaviour on clustered qpidd when the broker
is shut down in the middle of perftest.

On simple non-clustered broker when I run perftest and shut down the broker
(e.g. with "service qpidd stop"), qpid-perftest writes "Connection closed"
and exits the process. But it stays hanging on clustered broker when I turn
off the deamon which perftest initiates connection to.

qpid-cpp-server-cluster-0.7.946106-2.el5
qpid-cpp-client-devel-0.7.946106-2.el5

Comment 4 Gordon Sim 2010-06-08 08:10:47 UTC

The intermittent authentication failure (only apparent when using explicit username) is now fixed on trunk (r952390) and in release repo (http://mrg1.lab.bos.redhat.com/git/?p=qpid.git;a=commitdiff;h=395447da146aac21bc8ced9a1de3c75bc1008233).

Comment 6 Gordon Sim 2010-07-20 16:14:07 UTC

Created attachment 433206 [details]
log file for failure case

cyrus-sasl-2.1.22-5.el5, cyrus-sasl-gssapi-2.1.22-5.el5 on RHEL 5.4 using r965449 with my 'fix' for this reverted (see bug 613011).

Comment 7 Gordon Sim 2010-07-22 16:04:29 UTC

I believe this issue is actually in either the sasl or krb5 libs as I can reproduce something similar using the bundled sample client/server progs (i.e. independent of qpid): https://bugzilla.redhat.com/show_bug.cgi?id=617267

Comment 8 Justin Ross 2012-12-11 20:41:18 UTC


*** This bug has been marked as a duplicate of bug 617267 ***

Note You need to log in before you can comment on or make changes to this bug.