Bug 723812

Summary: BadProcessStatus exception on testagent cpp client
Product: Red Hat Enterprise MRG Reporter: Petr Matousek <pematous>
Component: qpid-cppAssignee: messaging-bugs <messaging-bugs>
Status: NEW --- QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 1.3CC: iboverma, jross
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 716457 Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 716249    

Description Petr Matousek 2011-07-21 09:01:57 UTC 
+++ This bug was initially created as a clone of Bug #716457 +++

Description of problem:

This issue was discovered during testing of BZ682815, please see Bug 682815 for more details. 

A core dump in the testagent client can be seen when running the reproducer.

cluster_tests.LongTests.test_management ................................. fail
Error during test:
  Traceback (most recent call last):
    File "./qpid-python-test", line 311, in run
      phase()
    File "/root/pematous/bz682815/run_cluster_tests2/cluster_tests.py", line
834, in test_management
      c.stop()
    File "/root/pematous/bz682815/run_cluster_tests2/cluster_tests.py", line
777, in stop
      StoppableThread.stop(self)
    File "/root/pematous/bz682815/run_cluster_tests2/brokertest.py", line 565,
in stop
      if self.error: raise self.error
  RethrownException: Error in ClientLoop.run
  Traceback (most recent call last):
    File "/root/pematous/bz682815/run_cluster_tests2/cluster_tests.py", line
760, in run
      self.process.unexpected(
    File "/root/pematous/bz682815/run_cluster_tests2/brokertest.py", line 194,
in unexpected
      raise BadProcessStatus("%s %s%s" % (self.pname, msg, err))
  BadProcessStatus: testagent-31028 client of cluster0-6 exit code -11
Error during teardown:
  Traceback (most recent call last):
    File "./qpid-python-test", line 311, in run
      phase()
    File "/root/pematous/bz682815/run_cluster_tests2/brokertest.py", line 506,
in tearDown
      if err: raise Exception("Unexpected process status:\n    "+"\n   
".join(err))
  Exception: Unexpected process status:
      Wait failed qpid-tool-31737: [Errno 10] No child processes
      Wait failed qpid-perftest-31738: [Errno 10] No child processes
      Wait failed qpid-queue-stats-31739: [Errno 10] No child processes
      Wait failed testagent-31742: [Errno 10] No child processes
      Wait failed qpid-txtest-31740: [Errno 10] No child processes
      Wait failed qpid-perftest-31774: [Errno 10] No child processes

Version-Release number of selected component (if applicable):
qpid-cpp-mrg-0.10-8

How reproducible:
Unknown - please see Bug 682815 for details

Steps to Reproduce:
Please follow the 'Steps to Reproduce' in Bug 682815 to reproduce this issue
  
Actual results:
Core dump in the testagent client

Expected results:
Test passes repeatedly.

Additional info:

test log+data can be found:
mrg-qe-01.lab.eng.brq.redhat.com:/root/pematous/bz682815_data

--- Additional comment from aconway on 2011-06-27 08:19:53 EDT ---

The log messages indicate that the "testagent" client seg-faulted. This is a client side bug rather than a broker bug. There is no core file in the data directory - are the tests run with ulimit -c unlimited? If possible get a stack trace if this happens again.
Comment 1 Petr Matousek 2011-07-21 09:03:27 UTC 
This defect is also valid for 1.3.x

This was seen on RHEL5.7 i86_64, packages installed:
python-qpid-0.7.946106-15.el5
qpid-cpp-client-0.7.946106-33.el5
qpid-cpp-client-devel-0.7.946106-33.el5
qpid-cpp-client-devel-docs-0.7.946106-33.el5
qpid-cpp-client-ssl-0.7.946106-33.el5
qpid-cpp-mrg-debuginfo-0.7.946106-33.el5
qpid-cpp-server-0.7.946106-33.el5
qpid-cpp-server-cluster-0.7.946106-33.el5
qpid-cpp-server-devel-0.7.946106-33.el5
qpid-cpp-server-ssl-0.7.946106-33.el5
qpid-cpp-server-store-0.7.946106-33.el5
qpid-cpp-server-xml-0.7.946106-33.el5
qpid-java-client-0.10-6.el5
qpid-java-common-0.10-6.el5
qpid-java-example-0.10-6.el5
qpid-tools-0.7.946106-12.el5
Comment 2 Petr Matousek 2011-07-21 09:07:18 UTC 
stack trace:

Core was generated by `qpid-perftest --count=5000 --durable=yes --base-name 1da03b9d-2a5c-4fa3-9a7b-7d'.
Program terminated with signal 6, Aborted.
#0  0x0000003428230265 in raise () from /lib64/libc.so.6
(gdb) info thread
  4 Thread 27439  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
  3 Thread 27440  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
  2 Thread 27443  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 0x2ac5c53f9020 (LWP 27436)  0x0000003428230265 in raise () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 4 (Thread 27439):
#0  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
#1  0x000000342bb2a511 in qpid::sys::Poller::wait (this=0x1266a4c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x000000342bb2afa7 in qpid::sys::Poller::run (this=0x1266a4c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x000000342bb211fa in qpid::sys::(anonymous namespace)::runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003428a0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000034282d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 27440):
#0  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
#1  0x000000342bb2a511 in qpid::sys::Poller::wait (this=0x1266a4c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x000000342bb2afa7 in qpid::sys::Poller::run (this=0x1266a4c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x000000342bb211fa in qpid::sys::(anonymous namespace)::runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003428a0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000034282d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 27443):
#0  0x00000034282d48a8 in epoll_wait () from /lib64/libc.so.6
#1  0x000000342bb2a511 in qpid::sys::Poller::wait (this=0x1266a4c0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x000000342bb2afa7 in qpid::sys::Poller::run (this=0x1266a4c0) at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x000000342bb211fa in qpid::sys::(anonymous namespace)::runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003428a0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000034282d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ac5c53f9020 (LWP 27436)):
#0  0x0000003428230265 in raise () from /lib64/libc.so.6
#1  0x0000003428231d10 in abort () from /lib64/libc.so.6
#2  0x000000342c05c13e in qpid::sys::Mutex::~Mutex (this=0x12666c70, __in_chrg=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:112
#3  0x000000342c095c2b in ~Monitor (this=0x12666c00, __in_chrg=<value optimized out>) at ../include/qpid/sys/Monitor.h:33
#4  ~Waitable (this=0x12666c00, __in_chrg=<value optimized out>) at qpid/sys/Waitable.h:42
#5  ~StateMonitor (this=0x12666c00, __in_chrg=<value optimized out>) at qpid/sys/StateMonitor.h:39
#6  qpid::client::SessionImpl::~SessionImpl (this=0x12666c00, __in_chrg=<value optimized out>) at qpid/client/SessionImpl.cpp:89
#7  0x000000342c08f200 in release (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:145
#8  ~shared_count (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/shared_count.hpp:159
#9  ~shared_ptr (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/shared_ptr.hpp:106
#10 qpid::client::SessionBase_0_10::~SessionBase_0_10 (this=<value optimized out>, __in_chrg=<value optimized out>) at qpid/client/SessionBase_0_10.cpp:34
#11 0x000000000040f2bc in ~AsyncSession_0_10 (this=0x12662a50, __in_chrg=<value optimized out>) at ../../include/qpid/client/no_keyword/AsyncSession_0_10.h:48
#12 ~AsyncSession_0_10 (this=0x12662a50, __in_chrg=<value optimized out>) at ../../include/qpid/client/AsyncSession_0_10.h:191
#13 qpid::tests::Client::~Client (this=0x12662a50, __in_chrg=<value optimized out>) at qpid-perftest.cpp:260
#14 0x00000000004165e1 in qpid::tests::SubscribeThread::~SubscribeThread (this=0x12662a50, __in_chrg=<value optimized out>) at qpid-perftest.cpp:562
#15 0x000000000040cff8 in ~reversible_ptr_container (argc=<value optimized out>, argv=<value optimized out>) at /usr/include/boost/checked_delete.hpp:34
#16 ~ptr_sequence_adapter (argc=<value optimized out>, argv=<value optimized out>) at /usr/include/boost/ptr_container/ptr_sequence_adapter.hpp:148
#17 ~ptr_vector (argc=<value optimized out>, argv=<value optimized out>) at /usr/include/boost/ptr_container/ptr_vector.hpp:35
#18 main (argc=<value optimized out>, argv=<value optimized out>) at qpid-perftest.cpp:740
Comment 3 Petr Matousek 2011-07-21 09:22:08 UTC 
core file, data and logs available on mrg-qe-01.lab.eng.brq.redhat.com:/data/bigisis.rhts.eng.bos.redhat.com/bz723812.tar.gz