.. and here is reproducer:

In Satellite (havent tested): have >1000 content hosts with goferd. Restart goferd frequently (every 10-30 seconds, say).

Outside Satellite:

1) Add some queues mimicking pulp.agent.<UUID> ones:

for i in $(seq 0 200); do qpid-config --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 add queue pulp.agent.$i & done


2) Run below script (modification of the one from bz1398377):

for i in $(seq 1 20); do python just_consume_to_segfault.py 10.0 10 & sleep 0.95; done

for i in $(seq 1 20); do python just_consume_to_segfault.py 10.0 10 & sleep 0.95; done

(it is worth running it once on some system, and once on another - it adds some load to the system..)


3) Wait up to 30 minutes (I got it in 10 mniutes but it is non-deterministic like a dice)


4) segfault!

Script itself:

#!/usr/bin/python

from time import sleep
from uuid import uuid4

from proton import ConnectionException, Timeout
from proton import SSLDomain, SSLException
#from proton import Message

from proton.utils import BlockingConnection

import threading
import traceback
import os, sys
import random

SSL = True
ROUTER_ADDRESS_NOSSL = "proton+amqp://pmoravec-sat62-rhel7.gsslab.brq.redhat.com:5648"
ROUTER_ADDRESS_SSL = "amqps://pmoravec-sat62-rhel7.gsslab.brq.redhat.com:5647"
ROUTER_ADDRESS = ROUTER_ADDRESS_SSL if SSL else ROUTER_ADDRESS_NOSSL
#ROUTER_ADDRESS = "proton+amqp://toledo-capsule.gsslab.brq.redhat.com:5648"
ADDRESS = "pulp.agent"
#ADDRESS = "queue"
HEARTBEAT = 10
SLEEP = float(sys.argv[1])
THREADS = int(sys.argv[2])
START_PERIOD = 10.0


class ReceiverThread(threading.Thread):
    def __init__(self, _id, address=ADDRESS, domain=None):
        super(ReceiverThread, self).__init__()
        self._id = _id
        self.address = address
        print self.address
        self.domain = domain
        self.running = True
        self.conn = None

    def connect(self):
        try:
            self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=self.domain, heartbeat=HEARTBEAT)
            self.conn.create_receiver(self.address, name=str(uuid4()), dynamic=False, options=None)
        except Exception:
            self.conn = None

    def run(self):
        while self.running:
            while self.conn == None:
                self.connect()
            sleep(SLEEP)
            try:
                print "%s: reconnecting.." % self.address
                self.conn.close()
            except Exception, e:
                print e
                pass
            self.conn = None

    def stop(self):
        self.running = False

ca_certificate='/etc/rhsm/ca/katello-server-ca.pem'
client_certificate='/etc/pki/consumer/bundle.pem'
client_key=None

domain = SSLDomain(SSLDomain.MODE_CLIENT)
domain.set_trusted_ca_db(ca_certificate)
domain.set_credentials(client_certificate, client_key or client_certificate, None)
domain.set_peer_authentication(SSLDomain.VERIFY_PEER)

random.seed()
threads = []
for i in range(THREADS):
  threads.append(ReceiverThread(i, "%s.%s" % (ADDRESS, i), domain if SSL else None))
  threads[i].start()
  sleep(START_PERIOD/THREADS)

while True:
  sleep(10)
_in = raw_input("Press Enter to exit:")

for i in range(THREADS):
    threads[i].stop()
for i in range(THREADS):
    threads[i].join()

btw I am not further able to reproduce the segfault on reinstalled Satellite. So verification steps shall be rather smoke test based - run the reproducer and check new version of qdrouterd doesnt fail.

/me failed to reliably reproduce on 0.4-20. Running the original reproducer against 0.4-21, and/or running scaled reproducer (more client connections, more often reconnections), no segfault was hit.

** HOTFIX AVAILABLE **

This set of hotfix packages are currently undergoing testing by Red Hat Quality Assurance but are available for users who want to get early access. This hotfix includes fixes for bugs 1398377 and 1398536

Instructions for hotfixing:

1) Download to your Satellite : http://people.redhat.com/~mmccune/hotfix/HOTFIX-1398377-1398536.tar.gz

2) Expand the archive, tar xvf HOTFIX-1398377-1398536.tar.gz

3) cd to directory depending on version of base OS you are using

4) Upgrade RPMS:

rpm -Uvh qpid-dispatch-router-0.4-21.el7sat.x86_64.rpm libqpid-dispatch-0.4-21.el7sat.x86_64.rpm

(optionally include qpid-dispatch-tools if you get dependency errors and have the older version of this package already installed)

5) katello-service restart

proceed with operations.

Verified in Satellite 6.2.4 async based on #4 and #6 as well as the no-break automation test results.

# rpm -qa | grep qpid-dispatch
libqpid-dispatch-0.4-21.el6sat.x86_64
qpid-dispatch-debuginfo-0.4-21.el6sat.x86_64
qpid-dispatch-router-0.4-21.el6sat.x86_64
qpid-dispatch-tools-0.4-21.el6sat.x86_64

# rpm -qa | grep qpid-dispatch
qpid-dispatch-debuginfo-0.4-21.el7sat.x86_64
qpid-dispatch-tools-0.4-21.el7sat.x86_64
libqpid-dispatch-0.4-21.el7sat.x86_64
qpid-dispatch-router-0.4-21.el7sat.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2855