Description of problem:
Under unknown scenario within Satellite, qdrouterd on Sat segfaults with backtrace matching https://issues.jboss.org/browse/ENTMQ-1162 / https://access.redhat.com/solutions/2042333 .

When running that scenario (outside any Satellite bits) reproducing that bug, I get the same crash on qdrouterd 0.4-22 / proton 0.9-16.


Version-Release number of selected component (if applicable):
qpid-proton-c-0.9-16.el7.x86_64
qpid-dispatch-router-0.4-22.el7sat.x86_64


How reproducible:
100% within few minutes


Steps to Reproduce:
(reproducer outside Satellite; for Sat based reproducer, in a loop, concurrently restart goferd frequently on few hundreds of Content Hosts)

1. Add many queues to qpidd:
for j in $(seq 0 10); do for i in $(seq 0 50); do qpid-config add queue pulp.pmoravec.${j}.${i}; done; echo $j; done

2. Start qdrouterd that link routes pulp.* prefix to qpidd

3. Run below script (use proper ROUTER_ADDRESS) - the script just concurrently creates receivers to addresses pulp.pmoravec.[0-10].[0-50] :


import random
import threading
import traceback

from proton import ConnectionException
from proton.utils import BlockingConnection
from proton import SSLDomain

from time import sleep
from uuid import uuid4
 
ROUTER_ADDRESS = "proton+amqp://dispatch.router.fqdn:5648"
ADDRESS = "pulp.pmoravec"
HEARTBEAT = 2
SLEEP_MIN = 1.9
SLEEP_MAX = 2.1
THREADS = 10
 
class ReceiverThread(threading.Thread):
    def __init__(self, _id, address=ADDRESS, domain=None):
        super(ReceiverThread, self).__init__()
        self._id = _id
        self.address = address
        self.domain = domain
        self.running = True
        self.nr = 1
 
    def connect(self):
        self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=self.domain, heartbeat=HEARTBEAT)
        self.conn.create_receiver('%s.%s' %(self.address, 0), name=str(uuid4()), dynamic=False, options=None)
 
    def reconnect(self):
        print "(%s): something got broken, reconnecting.." % self._id
        try:
            self.conn.close()
        except:
            print "(%s): receiver thread: failed to close connection" % self._id
            pass
        self.connect()
 
    def run(self):
        self.connect()
        while self.running:
            sleep(random.uniform(SLEEP_MIN,SLEEP_MAX))
            self.nr += 1
            try:
                self.recv = self.conn.create_receiver('%s.%s' %(self.address, self.nr), name=str(uuid4()), dynamic=False, options=None)
            except Exception as e:
                print "(%s): receiver failed, retrying.." % self._id
                self.reconnect()
 
    def stop(self):
        self.running = False
 
threads = []
for i in range(THREADS):
  threads.append(ReceiverThread(i, '%s.%s' %(ADDRESS, i)))
  threads[i].start()
 
_in = raw_input("Press Enter to exit:")
 
for i in range(THREADS):
    threads[i].stop()
for i in range(THREADS):
    threads[i].join()


4. Wait few minutes to qdrouterd segfault.


Actual results:
qdrouterd segfaults


Expected results:
no qdrouterd segfault


Additional info:
Backtrace:

#0  pni_record_find (record=<optimized out>, record=<optimized out>, key=key@entry=0) at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/record.c:71
#1  pn_record_get (record=<optimized out>, key=key@entry=0) at /usr/src/debug/qpid-proton-0.9/proton-c/src/object/record.c:120
#2  0x00007f9fb52e1593 in pn_connection_get_context (conn=<optimized out>) at /usr/src/debug/qpid-proton-0.9/proton-c/src/engine/engine.c:184
#3  0x00007f9fb5525e61 in qd_link_connection (link=<optimized out>) at /usr/src/debug/qpid-dispatch-0.4/src/container.c:937
#4  0x00007f9fb5533c95 in router_link_attach_handler (context=0x8de8d0, link=<optimized out>) at /usr/src/debug/qpid-dispatch-0.4/src/router_node.c:1686
#5  0x00007f9fb552511c in handle_link_open (container=<optimized out>, pn_link=0x7f9f7895ecb0) at /usr/src/debug/qpid-dispatch-0.4/src/container.c:217
#6  process_handler (unused=<optimized out>, qd_conn=0x7f9f88025fe0, container=0x836cf0) at /usr/src/debug/qpid-dispatch-0.4/src/container.c:481
#7  handler (handler_context=0x836cf0, conn_context=<optimized out>, event=event@entry=QD_CONN_EVENT_PROCESS, qd_conn=0x7f9f88025fe0) at /usr/src/debug/qpid-dispatch-0.4/src/container.c:643
#8  0x00007f9fb553795c in process_connector (cxtr=0x7f9f880188e0, qd_server=0x8f06f0) at /usr/src/debug/qpid-dispatch-0.4/src/server.c:398
#9  thread_run (arg=<optimized out>) at /usr/src/debug/qpid-dispatch-0.4/src/server.c:626
#10 0x00007f9fb55389c0 in qd_server_run (qd=0x604030) at /usr/src/debug/qpid-dispatch-0.4/src/server.c:971
#11 0x0000000000401cd8 in main_process (config_path=config_path@entry=0x7ffc5a7c2f2d "/etc/qpid-dispatch/qdrouterd.conf", 
    python_pkgdir=python_pkgdir@entry=0x402401 "/usr/lib/qpid-dispatch/python", fd=fd@entry=2) at /usr/src/debug/qpid-dispatch-0.4/router/src/main.c:135
#12 0x0000000000401950 in main (argc=3, argv=0x7ffc5a7c1688) at /usr/src/debug/qpid-dispatch-0.4/router/src/main.c:335