Bug 509436

Summary: cluster_test causes broker crash on shutdown not detected by test
Product: Red Hat Enterprise MRG Reporter: Kim van der Riet <kim.vdriet>
Component: qpid-cppAssignee: Ted Ross <tross>
Status: CLOSED DUPLICATE QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: Development   
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-07-08 12:31:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kim van der Riet 2009-07-02 18:43:02 UTC
Running cluster_test in a loop with ulimit -c unlimited frequently drops core files, even though the tests all pass. This seems to be a crash on broker shutdown that escapes detection because the test has already ended.

Trace seems to show qpid::management::ManagementAgent raising an event in the Connection destructor, possibly after all the classes around it have already been destroyed.

Core backtrace:

(gdb) bt
#0  memcpy () at ../sysdeps/x86_64/memcpy.S:509
#1  0x00000030ad0a3075 in std::char_traits<char>::copy () at /usr/src/debug/gcc-4.3.2-20081105/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/char_traits.h:274
#2  std::string::_M_copy (__n=<value optimized out>, __s=<value optimized out>, __d=<value optimized out>) at /usr/src/debug/gcc-4.3.2-20081105/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:344
#3  std::string::append (this=0x7fff8ebaa780, __str=@0x7f9339964b88) at /usr/src/debug/gcc-4.3.2-20081105/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:331
#4  0x00007f93396beaa0 in operator+<char, std::char_traits<char>, std::allocator<char> > () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/basic_string.tcc:677
#5  qpid::management::ManagementAgent::raiseEvent (this=0x7f93365c8010, event=<value optimized out>, severity=<value optimized out>) at qpid/management/ManagementAgent.cpp:211
#6  0x00007f933962bade in ~Connection (this=0x1c21b08) at qpid/broker/Connection.cpp:127
#7  0x00007f93389d658f in ~Connection (this=0x1c21a10) at qpid/cluster/Connection.cpp:111
#8  0x00007f93389c219c in qpid::RefCounted::release () at ./qpid/RefCounted.h:42
#9  intrusive_ptr_release (p=<value optimized out>) at ./qpid/RefCounted.h:57
#10 ~intrusive_ptr () at /usr/include/boost/intrusive_ptr.hpp:83
#11 ~pair () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_pair.h:73
#12 __gnu_cxx::new_allocator<std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > >::destroy () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/ext/new_allocator.h:118
#13 std::_Rb_tree<qpid::cluster::ConnectionId, std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> >, std::_Select1st<std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > >, std::less<qpid::cluster::ConnectionId>, std::allocator<std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > > >::_M_destroy_node ()
    at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_tree.h:390
#14 std::_Rb_tree<qpid::cluster::ConnectionId, std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> >, std::_Select1st<std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > >, std::less<qpid::cluster::ConnectionId>, std::allocator<std::pair<qpid::cluster::ConnectionId const, boost::intrusive_ptr<qpid::cluster::Connection> > > >::_M_erase (this=0x1bd2360, __x=0x1c22340)
    at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_tree.h:943
#15 0x00007f93389b9dc9 in ~_Rb_tree () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_tree.h:585
#16 ~map () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_map.h:92
#17 ~Cluster (this=0x1bd1be0) at qpid/cluster/Cluster.cpp:219
#18 0x00007f93389b7e25 in qpid::cluster::Cluster::brokerShutdown (this=0x1bd1be0) at qpid/cluster/Cluster.cpp:573
#19 0x00007f933924df37 in boost::function0<void, std::allocator<void> >::operator() (this=<value optimized out>) at /usr/include/boost/function/function_template.hpp:692
#20 0x00007f933924da3a in for_each<__gnu_cxx::__normal_iterator<boost::function<void ()(), std::allocator<void> >*, std::vector<boost::function<void ()(), std::allocator<void> >, std::allocator<boost::function<void ()(), std::allocator<void> > > > >, void (*)(boost::function<void ()(), std::allocator<void> >)> () at /usr/lib/gcc/x86_64-redhat-linux/4.3.2/../../../../include/c++/4.3.2/bits/stl_algo.h:3791
#21 qpid::Plugin::Target::finalize (this=0x1bcffb8) at qpid/Plugin.cpp:45
#22 0x00007f93396008c4 in ~Broker (this=0x1bcffb0) at qpid/broker/Broker.cpp:338
#23 0x000000309e636960 in __cxa_finalize (d=0x7f9339953880) at cxa_finalize.c:56
#24 0x00007f93395c9a86 in __do_global_dtors_aux () from /home/kpvdr/mrg/qpid/cpp/src/.libs/libqpidbroker.so.0
#25 0x0000000000405a20 in std::basic_streambuf<char, std::char_traits<char> >::~basic_streambuf ()
#26 0x00007fff8ebab030 in ?? ()
#27 0x00007f93396ca281 in _fini () from /home/kpvdr/mrg/qpid/cpp/src/.libs/libqpidbroker.so.0
#28 0x0000000000000019 in ?? ()
#29 0x0000000001bc3230 in ?? ()
#30 0x0000000001bc4b00 in ?? ()
#31 0x0000000001bc6dd0 in ?? ()
#32 0x00007f9339967778 in ?? ()
#33 0x00007f9339530000 in ?? ()
#34 0x00007f9339530508 in ?? ()
#35 0x00007f9339530a08 in ?? ()
#36 0x00007f9338c61000 in ?? ()
#37 0x00007f9338c60000 in ?? ()
#38 0x00007f9338c614c8 in ?? ()
#39 0x00007f9338c61990 in ?? ()
#40 0x0000000001bc7380 in ?? ()
#41 0x00007f9338c604d8 in ?? ()
#42 0x00007f9338c609a8 in ?? ()
#43 0x00007f9338c5f000 in ?? ()
#44 0x00007f9338c5f990 in ?? ()
#45 0x00007f9338c5e128 in ?? ()
#46 0x00007f9338c5e5f0 in ?? ()
#47 0x0000000001bc41c0 in ?? ()
#48 0x0000000001bc4660 in ?? ()
#49 0x00007f9338c5f4c8 in ?? ()
#50 0x000000309d4204e8 in _rtld_local () from /lib64/ld-2.9.so
#51 0x0000000001bc7850 in ?? ()
#52 0x0000000000000000 in ?? ()
Current language:  auto; currently asm

Comment 2 Ted Ross 2009-10-30 18:46:58 UTC
There was a cluster-shutdown bug (BZ490855) that looks like it might be causing this.  BZ490855 was fixed upstream in version 823258.  Is this still occurring or might it be marked duplicate?

-Ted

Comment 3 Kim van der Riet 2010-07-08 12:31:31 UTC
After running this test again using r.961365 in a loop, there is no sign of any core files.

Marking as a dup, as suggested in comment #2.

*** This bug has been marked as a duplicate of bug 490855 ***