Bug 492327 - clustered broker crashes in DispatchHandle III (triggered by failover soak)
clustered broker crashes in DispatchHandle III (triggered by failover soak)
Status: CLOSED DUPLICATE of bug 490457
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.1
All Linux
high Severity high
: 1.1.1
: ---
Assigned To: Andrew Stitcher
Frantisek Reznicek
:
Depends On:
Blocks: 487706
  Show dependency treegraph
 
Reported: 2009-03-26 10:04 EDT by Frantisek Reznicek
Modified: 2015-11-15 19:07 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-05 10:30:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
mrg8 persistant failover soak core backtraces (46.63 KB, text/plain)
2009-04-17 04:51 EDT, Frantisek Reznicek
no flags Details
dhcp-lab-200 persistant failover soak core backtraces (73.25 KB, text/plain)
2009-04-17 04:52 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2009-03-26 10:04:53 EDT
Description of problem:
The failover soak soak test running on RHEL 5.3 x86_64 (dell-pesc1425-01)
triggers qpidd trash in DispatchHandle very similar to bug 490457.

The crash was detected after approx 1000 runs of the failover soak test 
(last run exit code was 8), which can be found here
 https://bugzilla.redhat.com/attachment.cgi?id=335676  (part of bug 490855).


Version-Release number of selected component (if applicable):
[root@dell-pesc1425-01 fsoak]# rpm -qa | egrep '(qpid|rhm|openai)' | sort -u
openais-0.80.3-22.el5_3.3
openais-debuginfo-0.80.3-22.el5_3.3
openais-devel-0.80.3-22.el5_3.3
python-qpid-0.5.752581-1.el5
qpidc-0.5.752581-1.el5
qpidc-debuginfo-0.5.752581-1.el5
qpidc-devel-0.5.752581-1.el5
qpidc-perftest-0.5.752581-1.el5
qpidc-rdma-0.5.752581-1.el5
qpidc-ssl-0.5.752581-1.el5
qpidd-0.5.752581-1.el5
qpidd-acl-0.5.752581-1.el5
qpidd-cluster-0.5.752581-1.el5
qpidd-devel-0.5.752581-1.el5
qpidd-rdma-0.5.752581-1.el5
qpidd-ssl-0.5.752581-1.el5
qpidd-xml-0.5.752581-1.el5
qpid-java-client-0.5.751061-1.el5
qpid-java-common-0.5.751061-1.el5
rhm-0.5.3153-1.el5
rhm-docs-0.5.756148-1.el5


How reproducible:
quite hard to reproduce (1 occurence withing 1010 runs on dell-pesc1425-01.rhts.bos.redhat.com)

Steps to Reproduce:
1. install and set up and start openais
2. ulimit -c unlimited
3. extract the fsoak.tar.bz2
4. compile clients using fsoak/run_fs.sh
5. launch using fsoak/run_failover_soak
  
Actual results:
qpidd crashes.

Expected results:
qpidd should not crash.

Additional info: (Backtrace below)

root@dell-pesc1425-01 fsoak]# gdb `which qpidd` core.4166
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/cluster.so
Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/openais/libcpg.so.2
Reading symbols from /usr/lib64/libcman.so.2...done.
Loaded symbols for /usr/lib64/libcman.so.2
Reading symbols from /usr/lib64/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.0
Reading symbols from /usr/lib64/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/rdmaconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/rdmaconnector.so
Reading symbols from /usr/lib64/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib64/librdmawrap.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/librdmawrap.so.0
Reading symbols from /usr/lib64/librdmacm.so.1...done.
Loaded symbols for /usr/lib64/librdmacm.so.1
Reading symbols from /usr/lib64/libibverbs.so.1...done.
Loaded symbols for /usr/lib64/libibverbs.so.1
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.0
Reading symbols from /usr/lib64/libnss3.so...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...done.
Loaded symbols for /usr/lib64/libplds4.so
Core was generated by `qpidd --no-module-dir --load-module /usr/lib64/qpid/daemon/cluster.so --cluster'.
Program terminated with signal 6, Aborted.
[New process 4173]
[New process 4172]
[New process 4171]
[New process 4170]
[New process 4168]
[New process 4167]
[New process 4166]
#0  0x0000003c13030215 in raise () from /lib64/libc.so.6
(gdb) thread apply all backtrace

Thread 7 (process 4166):
#0  0x0000003c130d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c16172e8d in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:432
#2  0x0000003c16173c67 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:398
#3  0x0000003c166ccb86 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:319
#4  0x0000000000406948 in QpiddBroker::execute (this=<value optimized out>, options=0x61c0740) at posix/QpiddBroker.cpp:165
#5  0x0000000000405438 in main (argc=15, argv=0x7fff34280698) at qpidd.cpp:77

Thread 6 (process 4167):
#0  0x0000003c1380ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c16788e6f in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
#2  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#3  0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c130d30ad in clone () from /lib64/libc.so.6

Thread 5 (process 4168):
#0  0x0000003c1380ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c16788e6f in qpid::broker::Timer::run (this=<value optimized out>) at qpid/sys/posix/Condition.h:69
#2  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#3  0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c130d30ad in clone () from /lib64/libc.so.6

Thread 4 (process 4170):
#0  0x0000003c130d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c16172e8d in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:432
#2  0x0000003c16173c67 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:398
#3  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003c130d30ad in clone () from /lib64/libc.so.6

Thread 3 (process 4171):
#0  0x0000003c130d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c16172e8d in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:432
#2  0x0000003c16173c67 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:398
#3  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5  0x0000003c130d30ad in clone () from /lib64/libc.so.6

Thread 2 (process 4172):
#0  0x0000003c130d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c16172e8d in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:432
#2  0x0000003c16173c67 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:398
#3  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003c130d30ad in clone () from /lib64/libc.so.6

Thread 1 (process 4173):
#0  0x0000003c13030215 in raise () from /lib64/libc.so.6
#1  0x0000003c13031cc0 in abort () from /lib64/libc.so.6
#2  0x0000003c140bec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x0000003c140bcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x0000003c140bcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x0000003c140bceca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x0000003c161c0eee in ~ScopedLock (this=<value optimized out>) at qpid/sys/posix/Mutex.h:120
#7  0x0000003c161c07b6 in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=<value optimized out>)
    at qpid/sys/DispatchHandle.cpp:420
#8  0x0000003c16173c93 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/Poller.h:122
#9  0x0000003c1616ac4a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#10 0x0000003c13806367 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003c130d30ad in clone () from /lib64/libc.so.6
Comment 1 Frantisek Reznicek 2009-04-17 04:51:12 EDT
Created attachment 339970 [details]
mrg8 persistant failover soak core backtraces
Comment 2 Frantisek Reznicek 2009-04-17 04:52:40 EDT
Created attachment 339971 [details]
dhcp-lab-200 persistant failover soak core backtraces
Comment 3 Andrew Stitcher 2009-05-05 10:30:00 EDT

*** This bug has been marked as a duplicate of bug 490457 ***

Note You need to log in before you can comment on or make changes to this bug.