Bug 1687409 - sporadic segfault in qpid::sys::posix::AsynchIO::disconnected in qpid-cpp-client when closing connection
Summary: sporadic segfault in qpid::sys::posix::AsynchIO::disconnected in qpid-cpp-cli...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 3.2
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: ---
Assignee: messaging-bugs
QA Contact: Messaging QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-11 12:58 UTC by Pavel Moravec
Modified: 2025-02-10 03:59 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-02-10 03:59:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Pavel Moravec 2019-03-11 12:58:53 UTC
Description of problem:
Trying to find out reproducer for https://bugzilla.redhat.com/show_bug.cgi?id=1687401 , I found a different segfault with BT:

(gdb) bt
#0  0x00007fe8980048e0 in ?? ()
#1  0x00007fe95382ddf9 in qpid::sys::posix::AsynchIO::close (this=0x7fe91c01a900, h=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/posix/AsynchIO.cpp:615
#2  0x00007fe95382f0b1 in qpid::sys::posix::AsynchIO::disconnected (this=0x7fe91c01a900, h=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/posix/AsynchIO.cpp:607
#3  0x00007fe9538b6723 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#4  0x00007fe9538b5047 in qpid::sys::DispatchHandle::processEvent (this=0x7fe91c01a908, type=qpid::sys::Poller::DISCONNECTED)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/DispatchHandle.cpp:291
#5  0x00007fe95385595d in process (this=0x7fe940001670) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/Poller.h:131
#6  qpid::sys::Poller::run (this=0x7fe940001670) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/epoll/EpollPoller.cpp:522
#7  0x00007fe953849d5a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/posix/Thread.cpp:35
#8  0x00000037c9607aa1 in start_thread (arg=0x7fe92abfd700) at pthread_create.c:301
#9  0x00000037c92e8bdd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) 


Version-Release number of selected component (if applicable):
qpid-cpp 1.36.0-20+hf1


How reproducible:
100% within a hour


Steps to Reproduce:
1. randomly restart qpidd (and re-create some queue for the reproducer):
while true; do
	echo "$(date): restarting qpidd"
	service qpidd restart > /dev/null 2>&1
	sleep 0.1
	qpid-config add queue TestQueue --limit-policy=ring --max-queue-count=1000000
	sleep 10
	sleep $((RANDOM%20))
done

2. simulate random network failures:
iptables -F
while true; do
	prob=$((50+RANDOM%50))
	echo "$(date): dropping ${prob}% packets"
	iptables -A OUTPUT -p tcp --destination-port 5672 -m statistic --mode random --probability 0.${prob} -j DROP
	sleep 30;
	sleep $((RANDOM%30))
	echo "$(date): clearing iptables"
	iptables -F
	sleep $((RANDOM%30))
done

3. have this client program that concurrently opens a connection and sender, sends few messages and closes everything - in a loop for each thread:

#include <qpid/messaging/Address.h>
#include <qpid/messaging/Connection.h>
#include <qpid/messaging/Message.h>
#include <qpid/messaging/Receiver.h>
#include <qpid/messaging/Sender.h>
#include <qpid/messaging/Session.h>
#include <sys/time.h>

#include <cstdlib>
#include <iostream>

#include <sstream>
#include <thread>
#include <unistd.h>
#include <time.h>

using namespace qpid::messaging;

using std::stringstream;
using std::string;
int exitFlag=0;

#define BROKER_AND_PORT                     "amqp:tcp:127.0.0.1:5672"
#define SENDMSG_ARRAY_THREAD_COUNT          500
#define SENDMSG_COUNT                       50
#define SENDING_QUEUE                       "TestQueue"
std::thread sendMessageArrayThread[SENDMSG_ARRAY_THREAD_COUNT];
std::mutex mtx;  

void connection_send_dont_close(int j){
  std::thread::id this_id=std::this_thread::get_id();
  Connection connection(BROKER_AND_PORT,"");
  try {
    int iter=0;
    while(1){
      connection.open();
      Session session = connection.createSession();
      Sender sender = session.createSender(SENDING_QUEUE);
      int i=0; iter++;
      while(i++<=SENDMSG_COUNT){
        Message request;
        request.setContent("Test Message");
        request.setSubject("SOMESUBJECT");
        sender.send(request);
      }
      std::cout << "thread " << j << " before close in iteration " << iter << std::endl;
      connection.close();
    }
  }
  catch(const std::exception& error) {
        std::cout << "exception caught: LocalThread Number " << j << " Thread 0x" << std::hex << this_id << ": Error: " <<  error.what() << std::endl;
        connection.close();
        mtx.unlock();
  }
}

void init_send_msg_threads(){
  int i=0;
  std::cout << "Creating " << SENDMSG_ARRAY_THREAD_COUNT <<" array thread" << std::endl;
  for(i=0;i<SENDMSG_ARRAY_THREAD_COUNT;i++){
    sendMessageArrayThread[i]=std::thread(connection_send_dont_close,i);
    usleep(100);
  }
}

void wait_for_send_msg_threads(){
  int i=0;
  for(i=0;i<SENDMSG_ARRAY_THREAD_COUNT;i++){
    sendMessageArrayThread[i].join();
  }
}

int main(int argc, char** argv) {
  init_send_msg_threads();
  wait_for_send_msg_threads();
  return 1;
}


4. compile it:
g++ -g qpid_reproducer.cpp -lqpidmessaging -std=c++0x -o qpid-reproducer

5. Run it (in a loop such that more segfaults can be observed):
ulimit -c unlimited
while true; do ./qpid-reproducer; done


Actual results:
5. generates a coredump per hour, with above backtrace


Expected results:
5. no such coredumps


Additional info:

Comment 10 Red Hat Bugzilla 2025-02-10 03:59:41 UTC
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.


Note You need to log in before you can comment on or make changes to this bug.