Bug 767344

Summary:	Memory corruption and crash after client connection destruction
Product:	Red Hat Enterprise MRG	Reporter:	Jason Dillaman <jdillama>
Component:	qpid-cpp	Assignee:	Andrew Stitcher <astitcher>
Status:	CLOSED DUPLICATE	QA Contact:	MRG Quality Engineering <mrgqe-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.0	CC:	iboverma, jross, lzhaldyb
Target Milestone:	3.0
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-04-25 15:59:39 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	698367, 803771

Description Jason Dillaman 2011-12-13 20:41:59 UTC

Description of problem:
It appears that after destructing an open Qpid client connection, an AsynchIO callback will attempt to close the associated socket and callback the associated destructed connection.  Closing the socket will write a "-1" to the area of memory that used to hold the socket file descriptor and attempting to invoke the callback resulted in a pure virtual function call.

Version-Release number of selected component (if applicable):
qpid-cpp-client-0.12-6_ptc_hotfix_3.el6.x86_64

How reproducible:
Rare

Steps to Reproduce:
1. Destruct an open connection which initiates the close socket process
2. Witness that 'qpid::sys::posix::AsynchIO::close' is invoked when both 'socket' and the close callback function contains a reference/pointer back to the destructed connection
  
Actual results:
The heap was corrupted resulting in a segmentation fault in an unrelated thread allocating memory.  Additionally, the AIO thread was about to terminate the process due to invoking a pure virtual method.

Expected results:
The connection closed and the client application continued to function.

Additional info:

Comment 1 Jason Dillaman 2011-12-13 20:51:56 UTC

Poller thread backtrace:
------------------------

#0  0x00007f0326497d10 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#1  0x00007f032649bf40 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#2  0x00007f03264a2625 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#3  0x00007f032551ea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#4  0x00007f032551cc06 in ?? () from /usr/lib64/libstdc++.so.6
#5  0x00007f032551cc33 in std::terminate() () from /usr/lib64/libstdc++.so.6
#6  0x00007f032551d55f in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
#7  0x00007f0325bf0b56 in operator() (this=0x7f031010d520, h=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
#8  qpid::sys::posix::AsynchIO::close (this=0x7f031010d520, h=<value optimized out>) at qpid/sys/posix/AsynchIO.cpp:574
#9  0x00007f0325bf241f in qpid::sys::posix::AsynchIO::disconnected (this=0x7f031010d520, h=...) at qpid/sys/posix/AsynchIO.cpp:564
#10 0x00007f0325cb7ad3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#11 0x00007f0325cb4955 in qpid::sys::DispatchHandle::processEvent (this=0x7f031010d528, type=qpid::sys::Poller::DISCONNECTED) at qpid/sys/DispatchHandle.cpp:291
#12 0x00007f0325bfda9d in process (this=0x28dd840) at qpid/sys/Poller.h:131
#13 qpid::sys::Poller::run (this=0x28dd840) at qpid/sys/epoll/EpollPoller.cpp:519
#14 0x00007f0325bf540a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#15 0x00007f0324a107f1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f0324d0a70d in clone () from /lib64/libc.so.6


Segmentation fault thread: (unrelated connection)
--------------------------

#0  0x00007f0324c9a1d2 in malloc_consolidate () from /lib64/libc.so.6
#1  0x00007f0324c9d062 in _int_malloc () from /lib64/libc.so.6
#2  0x00007f0324c9e48d in malloc () from /lib64/libc.so.6
#3  0x00007f032551d0bd in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#4  0x00007f0325fb2700 in qpid::client::ConnectionImpl::create (version=<value optimized out>, settings=...) at qpid/client/ConnectionImpl.cpp:178
#5  0x00007f0325fa69f6 in qpid::client::Connection::open (this=0x29414a0, settings=...) at qpid/client/Connection.cpp:124
#6  0x00007f0325fa744e in qpid::client::Connection::open (this=0x29414a0, url=..., settings=...) at qpid/client/Connection.cpp:81
#7  0x00007f032625ae2e in qpid::client::amqp0_10::ConnectionImpl::tryConnect (this=0x29413e0) at qpid/client/amqp0_10/ConnectionImpl.cpp:279
#8  0x00007f032625b85d in qpid::client::amqp0_10::ConnectionImpl::connect (this=0x29413e0, started=...) at qpid/client/amqp0_10/ConnectionImpl.cpp:249
#9  0x00007f032625baf3 in qpid::client::amqp0_10::ConnectionImpl::open (this=0x29413e0) at qpid/client/amqp0_10/ConnectionImpl.cpp:223
#10 0x00007f0326258266 in qpid::client::amqp0_10::ConnectionImpl::reopen (this=<value optimized out>) at qpid/client/amqp0_10/ConnectionImpl.cpp:234
#11 0x00007f0326267528 in qpid::client::amqp0_10::SessionImpl::nextReceiver (this=0x7f02f4000b60, receiver=..., timeout=...) at qpid/client/amqp0_10/SessionImpl.cpp:356

Comment 2 Andrew Stitcher 2013-04-25 15:58:46 UTC

Looking at the stack trace here, Id say this bug has the same underlying cause as BZ883469

The common symptom is the pure virtual call with qpid::sys::DispatchHandle::processEvent(..., qpid::sys::Poller::DISCONNECTED) in the stack trace.

So closing as duplicate in favour of that bug as it has a reproducer.

Comment 3 Andrew Stitcher 2013-04-25 15:59:39 UTC


*** This bug has been marked as a duplicate of bug 883469 ***