Bug 509970 - clustered qpidd crash in qpid::broker::Exchange::propagateFedOp() triggered by failover_soak
clustered qpidd crash in qpid::broker::Exchange::propagateFedOp() triggered b...
Status: CLOSED DUPLICATE of bug 508959
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.1.2
All Linux
high Severity high
: 1.1.6
: ---
Assigned To: messaging-bugs
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-07 04:03 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:11 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-07-07 04:15:58 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Frantisek Reznicek 2009-07-07 04:03:25 EDT
Description of problem:
The c++ broker very rarely crash in qpid::broker::Exchange::propagateFedOp() during long term failover soak test.
The issue has been visible on RHEL 5.3 x86_64.


Version-Release number of selected component (if applicable):
[root@mrg-qe-02 qpid_ptest_cluster_failover_soak]# rpm -qa | egrep '(qpid|rhm|openais)' | sort -u
openais-0.80.3-22.el5_3.8
openais-debuginfo-0.80.3-22.el5_3.8
python-qpid-0.5.752581-3.el5
qpidc-0.5.752581-22.el5
qpidc-debuginfo-0.5.752581-22.el5
qpidc-devel-0.5.752581-22.el5
qpidc-perftest-0.5.752581-22.el5
qpidc-rdma-0.5.752581-22.el5
qpidc-ssl-0.5.752581-22.el5
qpidd-0.5.752581-22.el5
qpidd-acl-0.5.752581-22.el5
qpidd-cluster-0.5.752581-22.el5
qpidd-devel-0.5.752581-22.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-22.el5
qpidd-ssl-0.5.752581-22.el5
qpidd-xml-0.5.752581-22.el5
qpid-java-client-0.5.751061-8.el5
qpid-java-common-0.5.751061-8.el5
rhm-0.5.3206-5.el5
rhm-docs-0.5.756148-1.el5


How reproducible:
very hard to reproduce (~1%)

Steps to Reproduce:
1. run the qpid_ptest_cluster_failover_soak test in the loop
   export TEST_PARAM_RHTS_TEST_MRG_QPID_PTEST_CLUSTER_FAILOVER_SOAK_LOOPS=222
2. wait for core files & examine them
  
Actual results:
Crash one a few hundrets of runs

Expected results:
No crash

Additional info: (backtrace)

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
...
Loaded symbols for /usr/lib64/libplds4.so
Core was generated by `qpidd --cluster-name soakTestCluster_db761df6-d70b-4f79-a49d-521f3a8d9a7a --aut'.
Program terminated with signal 11, Segmentation fault.
[New process 29867]
[New process 29871]
[New process 29870]
#0  0x00000036e047b6c6 in memcpy () from /lib64/libc.so.6
(gdb)
Thread 3 (process 29870):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002b0a30aed379 in qpid::broker::Timer::run (this=<value optimized out>)
    at qpid/sys/posix/Condition.h:69
#2  0x00002b0a30ee1cea in runRunnable (p=<value optimized out>)
    at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 2 (process 29871):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002b0a30aed379 in qpid::broker::Timer::run (this=<value optimized out>)
    at qpid/sys/posix/Condition.h:69
#2  0x00002b0a30ee1cea in runRunnable (p=<value optimized out>)
    at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 1 (process 29867):
#0  0x00000036e047b6c6 in memcpy () from /lib64/libc.so.6
#1  0x00000036eca9c200 in std::string::_Rep::_M_clone ()
   from /usr/lib64/libstdc++.so.6
#2  0x00000036eca9c8ff in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string () from /usr/lib64/libst
dc++.so.6
#3  0x00002b0a30a3b6c6 in qpid::broker::Exchange::propagateFedOp (
    this=<value optimized out>, routingKey=<value optimized out>,
    tags=<value optimized out>, op=<value optimized out>,
    origin=<value optimized out>) at qpid/broker/Exchange.cpp:230
#4  0x00002b0a30a67d70 in qpid::broker::DirectExchange::unbind (
    this=<value optimized out>, queue=<value optimized out>,
    routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:129
#5  0x00002b0a30aa6c67 in qpid::broker::QueueBindings::unbind (
    this=<value optimized out>, exchanges=<value optimized out>,
    queue=<value optimized out>) at qpid/broker/QueueBindings.cpp:39
#6  0x00002b0a30a3de4b in qpid::broker::Queue::unbind (
    this=<value optimized out>, exchanges=<value optimized out>,
    shared_ref=<value optimized out>) at qpid/broker/Queue.cpp:829
#7  0x00002b0a30a45cc6 in qpid::broker::Queue::tryAutoDelete (
    broker=<value optimized out>, queue=<value optimized out>)
    at qpid/broker/Queue.cpp:907
#8  0x00002b0a30ac86b0 in qpid::broker::SemanticState::cancel (
    this=<value optimized out>, c=<value optimized out>)
    at qpid/broker/SemanticState.cpp:336
#9  0x00002b0a30ac9468 in ~SemanticState (this=<value optimized out>)
    at qpid/broker/SemanticState.cpp:74
#10 0x00002b0a30ae2355 in ~SessionState (this=<value optimized out>)
    at qpid/broker/SessionState.cpp:97
#11 0x00002b0a30ae9bb5 in ~SessionHandler (this=<value optimized out>)
    at /usr/include/c++/4.1.2/memory:259
#12 0x00002b0a30a58900 in ~Connection (this=<value optimized out>)
    at /usr/include/boost/checked_delete.hpp:34
#13 0x00002b0a31223a45 in ~Connection (this=0x2aaaac001210)
    at qpid/cluster/Connection.cpp:111
#14 0x00002b0a311f9057 in ~Cluster (this=0x9027aa0) at qpid/RefCounted.h:42
#15 0x00002b0a311f7115 in qpid::cluster::Cluster::brokerShutdown (
    this=0x9027aa0) at qpid/cluster/Cluster.cpp:589
#16 0x00002b0a30efc8af in boost::function0<void, std::allocator<void> >::operator() (this=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:576
#17 0x00002b0a30efc736 in std::for_each<__gnu_cxx::__normal_iterator<boost::function<void ()(), std::allocator<void> >*, std::vector<boost::function<void ()(), std::allocator<void> >, std::allocator<boost::function<void ()(), std::allocator<void> > > > >, void (*)(boost::function<void ()(), std::allocator<void> >)> (
    __first=<value optimized out>, __last=<value optimized out>,
    __f=<value optimized out>) at /usr/include/c++/4.1.2/bits/stl_algo.h:159
#18 0x00002b0a30efbe00 in qpid::Plugin::Target::finalize (
    this=<value optimized out>) at qpid/Plugin.cpp:45
#19 0x00002b0a30a26379 in ~Broker (this=<value optimized out>)
    at qpid/broker/Broker.cpp:337
#20 0x00000036e043363e in __cxa_finalize () from /lib64/libc.so.6
#21 0x00002b0a309e5fb6 in __do_global_dtors_aux ()
   from /usr/lib64/libqpidbroker.so.0
#22 0x0000000000000000 in ?? ()
(gdb) quit
Comment 1 Gordon Sim 2009-07-07 04:15:58 EDT

*** This bug has been marked as a duplicate of bug 508959 ***

Note You need to log in before you can comment on or make changes to this bug.