Bug 591292 - MRG-M Heartbeat causes core
MRG-M Heartbeat causes core
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.2
All Linux
high Severity high
: 1.3
: ---
Assigned To: Gordon Sim
Frantisek Reznicek
:
: 602268 (view as bug list)
Depends On:
Blocks: 510475
  Show dependency treegraph
 
Reported: 2010-05-11 15:48 EDT by Scott Spurrier
Modified: 2015-11-15 20:12 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, on a 2 node MRG cluster, one of the clients running on it could core after shutting down a single network interface on one of the broker nodes. With this update, clients no longer core.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-10-14 12:07:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Stack Trace (54.56 KB, application/x-download)
2010-05-11 15:51 EDT, Scott Spurrier
no flags Details
test config (3.88 KB, application/zip)
2010-06-02 11:18 EDT, Scott Spurrier
no flags Details
amqpTest.latest (2.74 MB, application/octet-stream)
2010-06-02 11:19 EDT, Scott Spurrier
no flags Details
amqpTest_src_v1.2.2 (27.33 KB, application/x-gzip)
2010-06-02 11:21 EDT, Scott Spurrier
no flags Details
another reproducer (3.94 KB, text/x-c++src)
2010-06-04 13:31 EDT, Gordon Sim
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0773 normal SHIPPED_LIVE Moderate: Red Hat Enterprise MRG Messaging and Grid Version 1.3 2010-10-14 11:56:44 EDT

  None (edit)
Description Scott Spurrier 2010-05-11 15:48:56 EDT
Description of problem:
While testing client failover on a 2 node MRG cluster, the client cored after taking a single network interface down on one of the broker nodes.

How reproducible:
Sporatic client cores.

Steps to Reproduce:
1. Start 25 clients that send a messages to the broker cluster every 5 seconds, and set the heartbeat for the session to 2 seconds.
2. Start 5 clients that receive the messages, also set the heartbeat for this session to 2 seconds.
3. Select one of the broker nodes and shutdown the public network interface (ifdown eth0)
4. Watch for any client cores.  If there are no cores then bring the public interface back up
5. On the other broker node shutdown it's public interface (ifdown eth0)
6. Look for client cores, if none exist repeat the above steps (1 - 5) until a client cores.
Comment 1 Scott Spurrier 2010-05-11 15:51:45 EDT
Created attachment 413243 [details]
Stack Trace
Comment 3 Scott Spurrier 2010-06-02 11:18:47 EDT
Created attachment 419069 [details]
test config
Comment 4 Scott Spurrier 2010-06-02 11:19:49 EDT
Created attachment 419070 [details]
amqpTest.latest
Comment 5 Scott Spurrier 2010-06-02 11:21:28 EDT
Created attachment 419071 [details]
amqpTest_src_v1.2.2
Comment 7 Gordon Sim 2010-06-04 13:31:12 EDT
Created attachment 421304 [details]
another reproducer

This program starts a variable number of connections that intermittently send and receive messages with random times and heartbeats turned on. The first command line arg (if present) specifies the number of connections, the second defines the base name for the queues (e.g. ./test 10 abc wil create 10 connections using queues abc_1 through abc_10). The heartbeat interval (2 secs by default), port and host of initial broker can also be specified.

I used it to reproduce the problem by starting several processes each with different numbers of connections to a two node cluster and then periodically forced failover from one node to the other by sending the first node a STOP signal (with kill), then after a short wait sending a CONT to let it continue. This emulates failover due to a timeout as would be observed with the loss of the network connection.
Comment 8 Gordon Sim 2010-06-04 13:48:54 EDT
reproducer hint: A higher number of connections in the same process seems to speed up reproducing the error (e.g. ./test 100 cores pretty frequently on failover)
Comment 10 Gordon Sim 2010-06-09 07:42:23 EDT
Though much more rare, this is still an issue on latest packages (qpid-cpp-client-0.7.946106-2):

Core was generated by `./test 1 d'.
Program terminated with signal 6, Aborted.
[New process 9569]
[New process 8382]
[New process 10221]
[New process 9552]
[New process 9536]
[New process 9527]
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038fccbec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x00000038fccbcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000038fccbcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x00000038fccbd2ef in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
#6  0x00000035b6260e76 in fire (this=<value optimized out>) at qpid/client/ConnectionImpl.cpp:152
#7  0x00000035b5dfaf60 in qpid::sys::Timer::run (this=<value optimized out>) at qpid/sys/Timer.cpp:119
#8  0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 6 (process 9527):
#0  0x00000038fa407955 in pthread_join () from /lib64/libpthread.so.0
#1  0x00000035b5d23e2d in qpid::sys::Thread::join (this=<value optimized out>) at qpid/sys/posix/Thread.cpp:70
#2  0x000000000040893d in std::for_each<boost::void_ptr_iterator<__gnu_cxx::__normal_iterator<void**, std::vector<void*, std::allocator<void*> > >, Test>, boost::_bi::bind_t<void, boost::_mfi::mf0<void, Test>, boost::_bi::list1<boost::arg<1> > > > (__first={iter_ = {_M_current = 0x0}}, __last=
        {iter_ = {_M_current = 0x2540}}, __f={f_ = {f_ = 0x409790 <Test::join()>}, l_ = {a1_ = {<No data fields>}}})
    at /usr/include/boost/bind/mem_fn_template.hpp:55
#3  0x000000000040854a in main (argc=299481680, argv=0x7fff9a063228) at test.cpp:127

Thread 5 (process 9536):
#0  0x00000038fa40ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000035b62a2091 in qpid::client::StateManager::waitFor (this=<value optimized out>, desired=<value optimized out>)
    at ../include/qpid/sys/posix/Condition.h:63
#2  0x00000035b6259411 in qpid::client::ConnectionHandler::waitForOpen (this=<value optimized out>) at qpid/client/ConnectionHandler.cpp:144
#3  0x00000035b6266648 in qpid::client::ConnectionImpl::open (this=<value optimized out>) at qpid/client/ConnectionImpl.cpp:282
#4  0x00000035b6258115 in qpid::client::Connection::open (this=<value optimized out>, settings=<value optimized out>)
    at qpid/client/Connection.cpp:125
#5  0x00000035b627f4f1 in qpid::client::FailoverManager::attempt (this=<value optimized out>, c=<value optimized out>, s=<value optimized out>)
    at qpid/client/FailoverManager.cpp:121
#6  0x00000035b628047c in qpid::client::FailoverManager::attempt (this=<value optimized out>, c=<value optimized out>, s=<value optimized out>, 
    urls=<value optimized out>) at qpid/client/FailoverManager.cpp:111
#7  0x00000035b6280af5 in qpid::client::FailoverManager::connect (this=<value optimized out>, brokers=<value optimized out>)
    at qpid/client/FailoverManager.cpp:82
#8  0x00000035b628128b in qpid::client::FailoverManager::execute (this=<value optimized out>, c=<value optimized out>)
    at qpid/client/FailoverManager.cpp:44
#9  0x00000000004067bd in Test::run (this=0x11d9ba50) at test.cpp:97
#10 0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#11 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#12 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 9552):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000035b5d2c33f in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000035b5d2cd22 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 10221):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x00000035b5d2c33f in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000035b5d2cd22 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 8382):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000035b5d2c33f in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x00000035b5d2cd22 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 9569):
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038fccbec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x00000038fccbcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000038fccbcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x00000038fccbd2ef in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
#6  0x00000035b6260e76 in fire (this=<value optimized out>) at qpid/client/ConnectionImpl.cpp:152
#7  0x00000035b5dfaf60 in qpid::sys::Timer::run (this=<value optimized out>) at qpid/sys/Timer.cpp:119
#8  0x00000035b5d238da in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#9  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb)
Comment 11 Gordon Sim 2010-06-09 10:05:09 EDT
*** Bug 602268 has been marked as a duplicate of this bug. ***
Comment 12 Gordon Sim 2010-06-09 11:19:31 EDT
Fixed on trunk (r953032) and in release repo (http://mrg1.lab.bos.redhat.com/git/?p=qpid.git;a=commitdiff;h=079143676f0881d15138059d328a5531ef6a307e).
Comment 13 Gordon Sim 2010-06-11 05:32:22 EDT
Further fix applied to trunk (953610) and release repo (http://mrg1.lab.bos.redhat.com/git/?p=qpid.git;a=commitdiff;h=6227acefa2c27ad6b66ae7616e0b738ba3f9a754).
Comment 14 Frantisek Reznicek 2010-06-17 06:25:10 EDT
The issue has been fixed, tested on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qmf-0.7.946106-3.el5
python-qpid-0.7.946106-1.el5
qmf-0.7.946106-3.el5
qmf-devel-0.7.946106-3.el5
qpid-cpp-client-0.7.946106-3.el5
qpid-cpp-client-devel-0.7.946106-3.el5
qpid-cpp-client-devel-docs-0.7.946106-3.el5
qpid-cpp-client-rdma-0.7.946106-3.el5
qpid-cpp-client-ssl-0.7.946106-3.el5
qpid-cpp-mrg-debuginfo-0.7.946106-3.el5
qpid-cpp-server-0.7.946106-3.el5
qpid-cpp-server-cluster-0.7.946106-3.el5
qpid-cpp-server-devel-0.7.946106-3.el5
qpid-cpp-server-rdma-0.7.946106-3.el5
qpid-cpp-server-ssl-0.7.946106-3.el5
qpid-cpp-server-store-0.7.946106-3.el5
qpid-cpp-server-xml-0.7.946106-3.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tools-0.7.946106-4.el5
rh-qpid-cpp-tests-0.7.946106-3.el5
ruby-qmf-0.7.946106-3.el5

-> VERIFIED
Comment 15 Martin Prpič 2010-10-08 07:42:44 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, on a 2 node MRG cluster, one of the clients running on it could core after shutting down a single network interface on one of the broker nodes. With this update, clients no longer core.
Comment 17 errata-xmlrpc 2010-10-14 12:07:31 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html

Note You need to log in before you can comment on or make changes to this bug.