Bug 814716 - clustered qpidd sometimes aborts during shutdown (when killed using SIGINT)
clustered qpidd sometimes aborts during shutdown (when killed using SIGINT)
Status: CLOSED WONTFIX
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: messaging-bugs
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-20 10:08 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:14 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-06 09:32:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Frantisek Reznicek 2012-04-20 10:08:52 EDT
Description of problem:

Clustered qpidd rarely aborts during shutdown (when killed -2).

Unit tests ran in clustered environment highlighted single occurence of Abort during shutdown, when one of the brokers was killed using SIGINT signal.

This is abort stack trace:

  Core was generated by `/usr/sbin/qpidd --data-dir /mnt/tests/distribution/MRG/Messaging/qpid_ptest_uni'.
  Program terminated with signal 6, Aborted.
  #0  0x0000003858632885 in raise () from /lib64/libc.so.6
  Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.x86_64 boost-program-options-1.41.0-11.el6_1.2.x86_64 boost-system-1.41.
0-11.el6_1.2.x86_64 clusterlib-3.0.12.1-23.el6.x86_64 cyrus-sasl-lib-2.1.23-13.el6.x86_64 db4-cxx-4.7.25-16.el6.x86_64 glibc-2.12-1.47.el6_2.9.x86_64 libaio-0.3.
107-10.el6.x86_64 libgcc-4.4.6-3.el6.x86_64 libibverbs-1.1.5-3.el6.x86_64 librdmacm-1.0.14.1-3.el6.x86_64 libstdc++-4.4.6-3.el6.x86_64 libuuid-2.17.2-12.4.el6.x8
6_64 nspr-4.8.9-3.el6_2.x86_64 nss-3.13.1-7.el6_2.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 nss-util-3.13.1-3.el6_2.x86_64 xerces-c-3.0.1-20.el6.x86_64 xqil
la-2.2.3-8.el6.x86_64 zlib-1.2.3-27.el6.x86_64
  ...
  (*): Shared library is missing debugging information.
  (gdb)   2 Thread 0x7f53e5e1a700 (LWP 12506)  0x0000003858a0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  * 1 Thread 0x7f53e9e5b7a0 (LWP 12502)  0x0000003858632885 in raise () from /lib64/libc.so.6
  Thread 2 (Thread 0x7f53e5e1a700 (LWP 12506)):
  #0  0x0000003858a0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  #1  0x00007f53ea28e662 in wait (this=0x11756d0) at ../include/qpid/sys/posix/Condition.h:69
  #2  wait (this=0x11756d0) at ../include/qpid/sys/Monitor.h:45
  #3  qpid::sys::Timer::run (this=0x11756d0) at qpid/sys/Timer.cpp:159
  #4  0x00007f53ea1c066a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
  #5  0x0000003858a077f1 in start_thread () from /lib64/libpthread.so.0
  #6  0x00000038586e5ccd in clone () from /lib64/libc.so.6
  Thread 1 (Thread 0x7f53e9e5b7a0 (LWP 12502)):
  #0  0x0000003858632885 in raise () from /lib64/libc.so.6
  #1  0x0000003858634065 in abort () from /lib64/libc.so.6
  #2  0x000000385aabea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
  #3  0x000000385aabcc06 in ?? () from /usr/lib64/libstdc++.so.6
  #4  0x000000385aabcc33 in std::terminate() () from /usr/lib64/libstdc++.so.6
  #5  0x000000385aabd55f in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
  #6  0x00007f53ea711643 in qpid::broker::SemanticState::ConsumerImpl::notify (this=0x7f5348065090) at qpid/broker/SemanticState.cpp:771
  #7  0x00007f53ea6f21a8 in qpid::broker::QueueListeners::NotificationSet::notify (this=<value optimized out>) at qpid/broker/QueueListeners.cpp:79
  #8  0x00007f53ea6de517 in qpid::broker::Queue::requeue (this=0x11c1660, msg=...) at qpid/broker/Queue.cpp:229
  #9  0x00007f53ea7170f6 in operator() (this=<value optimized out>, requeue=<value optimized out>) at /usr/include/c++/4.4.6/bits/stl_function.h:587
  #10 for_each<std::reverse_iterator<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*> >, std::con
st_mem_fun_ref_t<void, qpid::broker::DeliveryRecord> > (this=<value optimized out>, requeue=<value optimized out>) at /usr/include/c++/4.4.6/bits/stl_algo.h:4200
  #11 qpid::broker::SemanticState::recover (this=<value optimized out>, requeue=<value optimized out>) at qpid/broker/SemanticState.cpp:554
  #12 0x00007f53ea717235 in qpid::broker::SemanticState::closed (this=0x7f535c021478) at qpid/broker/SemanticState.cpp:91
  #13 0x00007f53ea72f85b in qpid::broker::SessionState::~SessionState (this=0x7f535c0212a0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:100
  #14 0x00007f53ea72fd09 in qpid::broker::SessionState::~SessionState (this=0x7f535c0212a0, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:106
  #15 0x00007f53ea72ad95 in ~auto_ptr (this=0x7f535c01c060, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.6/backward/auto_ptr.h:168
  #16 qpid::broker::SessionHandler::~SessionHandler (this=0x7f535c01c060, __in_chrg=<value optimized out>) at qpid/broker/SessionHandler.cpp:41
  #17 0x00007f53ea72ae19 in qpid::broker::SessionHandler::~SessionHandler (this=0x7f535c01c060, __in_chrg=<value optimized out>) at qpid/broker/SessionHandler.cp
p:41
  #18 0x00007f53ea6803a7 in checked_delete<qpid::broker::SessionHandler const> (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/check
ed_delete.hpp:34
  #19 delete_clone<qpid::broker::SessionHandler> (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/clone_allocator.hpp:5
6
  #20 deallocate_clone<qpid::broker::SessionHandler> (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/clone_allocator.h
pp:74
  #21 deallocate_clone (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:126
  #22 null_policy_deallocate_clone (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp
:276
  #23 remove<boost::ptr_map_iterator<std::_Rb_tree_iterator<std::pair<unsigned short const, void*> >, unsigned short, qpid::broker::SessionHandler* const> > (thi
s=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:250
  #24 remove<boost::ptr_map_iterator<std::_Rb_tree_iterator<std::pair<unsigned short const, void*> >, unsigned short, qpid::broker::SessionHandler* const> > (thi
s=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:257
  #25 remove_all (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:218
  #26 ~reversible_ptr_container (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:46
5
  #27 ~associative_ptr_container (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/detail/associative_ptr_container.hpp:
34
  #28 ~ptr_map_adapter_base (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/ptr_map_adapter.hpp:130
  #29 ~ptr_map_adapter (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/ptr_map_adapter.hpp:388
  #30 ~ptr_map (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at /usr/include/boost/ptr_container/ptr_map.hpp:36
  #31 qpid::broker::Connection::~Connection (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at qpid/broker/Connection.cpp:154
  #32 0x00007f53ea680859 in qpid::broker::Connection::~Connection (this=0x7f535c0e5c60, __in_chrg=<value optimized out>) at qpid/broker/Connection.cpp:154
  #33 0x00007f53e74128d7 in reset (this=0x7f535c0e4b40, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.6/backward/auto_ptr.h:242
  #34 qpid::cluster::Connection::~Connection (this=0x7f535c0e4b40, __in_chrg=<value optimized out>) at qpid/cluster/Connection.cpp:187
  #35 0x00007f53e7413549 in qpid::cluster::Connection::~Connection (this=0x7f535c0e4b40, __in_chrg=<value optimized out>) at qpid/cluster/Connection.cpp:188
  #36 0x00007f53e73f6b60 in release (this=0x1191330, __x=0x7f5348485fa0) at qpid/RefCounted.h:42
  #37 intrusive_ptr_release<qpid::cluster::Connection> (this=0x1191330, __x=0x7f5348485fa0) at qpid/RefCounted.h:59
  #38 ~intrusive_ptr (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
  #39 ~pair (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_pair.h:68
  #40 destroy (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/ext/new_allocator.h:115
  #41 _M_destroy_node (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:383
  #42 _M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:972
  #43 _M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:970
  #44 _M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:970
  #45 _M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:970
  #46 _M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4.6/bits/stl_tree.h:970
  #47 std::_Rb_tree<qpid::cluster::ConnectionId, std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> >, std::_Select1st<
std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > >, std::less<qpid::cluster::ConnectionId>, std::allocator<std::pai
r<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > > >::_M_erase (this=0x1191330, __x=0x7f5348485fa0) at /usr/include/c++/4.4
.6/bits/stl_tree.h:970
  #48 0x00007f53e73f0386 in ~_Rb_tree (this=0x11909c0, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.6/bits/stl_tree.h:614
  #49 ~map (this=0x11909c0, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.6/bits/stl_map.h:87
  #50 qpid::cluster::Cluster::~Cluster (this=0x11909c0, __in_chrg=<value optimized out>) at qpid/cluster/Cluster.cpp:325
  #51 0x00007f53e73f0bd9 in qpid::cluster::Cluster::~Cluster (this=0x11909c0, __in_chrg=<value optimized out>) at qpid/cluster/Cluster.cpp:325
  #52 0x00007f53e73e738a in qpid::cluster::Cluster::brokerShutdown (this=0x11909c0) at qpid/cluster/Cluster.cpp:802
  #53 0x00007f53ea242998 in operator() (f=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
  #54 qpid::(anonymous namespace)::invoke(boost::function<void()>) (f=<value optimized out>) at qpid/Plugin.cpp:38
  #55 0x00007f53ea242cf3 in for_each<__gnu_cxx::__normal_iterator<boost::function<void()>*, std::vector<boost::function<void()>, std::allocator<boost::function<v
oid()> > > >, void (*)(boost::function<void()>)> (this=0x1175688) at /usr/include/c++/4.4.6/bits/stl_algo.h:4200
  #56 qpid::Plugin::Target::finalize (this=0x1175688) at qpid/Plugin.cpp:45
  #57 0x00007f53ea6641a0 in qpid::broker::Broker::~Broker (this=0x1175680, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:421
  #58 0x00007f53ea664ae9 in qpid::broker::Broker::~Broker (this=0x1175680, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:426
  #59 0x000000000040dbd5 in release (this=<value optimized out>, options=<value optimized out>) at qpid/RefCounted.h:42
  #60 intrusive_ptr_release<qpid::broker::Broker> (this=<value optimized out>, options=<value optimized out>) at qpid/RefCounted.h:59
  #61 ~intrusive_ptr (this=<value optimized out>, options=<value optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
  #62 QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:195
  #63 0x000000000040a38a in run_broker (argc=12, argv=0x7fff5cb46368, hidden=<value optimized out>) at qpidd.cpp:83
  #64 0x000000385861ecdd in __libc_start_main () from /lib64/libc.so.6
  #65 0x0000000000409e19 in _start ()


Version-Release number of selected component (if applicable):
> python-qpid-0.14-7.el6_2.noarch
> python-qpid-qmf-0.14-7.el6_2.x86_64
> qpid-cpp-client-0.14-14.el6_2.x86_64
> qpid-cpp-client-devel-0.14-14.el6_2.x86_64
> qpid-cpp-client-devel-docs-0.14-14.el6_2.noarch
> qpid-cpp-client-rdma-0.14-14.el6_2.x86_64
> qpid-cpp-client-ssl-0.14-14.el6_2.x86_64
> qpid-cpp-debuginfo-0.14-14.el6_2.x86_64
> qpid-cpp-server-0.14-14.el6_2.x86_64
> qpid-cpp-server-cluster-0.14-14.el6_2.x86_64
> qpid-cpp-server-devel-0.14-14.el6_2.x86_64
> qpid-cpp-server-rdma-0.14-14.el6_2.x86_64
> qpid-cpp-server-ssl-0.14-14.el6_2.x86_64
> qpid-cpp-server-store-0.14-14.el6_2.x86_64
> qpid-cpp-server-xml-0.14-14.el6_2.x86_64
> qpid-java-client-0.14-3.el6.noarch
> qpid-java-common-0.14-3.el6.noarch
> qpid-java-example-0.14-3.el6.noarch
> qpid-jca-0.14-9.el6.noarch
> qpid-jca-xarecovery-0.14-9.el6.noarch
> qpid-jca-zip-0.14-9.el6.noarch
> qpid-qmf-0.14-7.el6_2.x86_64
> qpid-qmf-debuginfo-0.14-7.el6_2.x86_64
> qpid-qmf-devel-0.14-7.el6_2.x86_64
> qpid-tests-0.14-1.el6_2.noarch
> qpid-tools-0.14-2.el6_2.noarch


How reproducible:
extremely low, depending on timing

Steps to Reproduce:
1. start-up active-active cluster (width 2)
2. Run unit tests from qpid/cpp tree, then qpid-tests, then python-qpid unit tests and then mrg-qe custom unit tests
3. force shutdown of one of the brokers using kill -2
4. qpidd aborts

Actual results:
  Cluster node aborted during shutdown as a reaction on kill -2.

Expected results:
  Cluster node should cleanly shutdown as a reaction on kill -2.

Additional info:
Comment 1 Justin Ross 2012-12-07 09:30:43 EST
Frantisek, a very low priority request: could you retest this with 0.18 if time permits?

Note You need to log in before you can comment on or make changes to this bug.