I can confirm qpid-dispatch 0.4-11 fixes the deadlock.

To the reproducer: the above one has lower probability to hit the bug. An improved version is:

1) Have just_many_consumers.2.py script per #c0
2) Run the script 10 times in parallel and re-start if stopped (update "-lt 10" accordingly to other python processes running on the host):

while true; do while [ $(pgrep python | wc -l) -lt 10 ]; do python just_many_consumers.2.py & sleep 1; done; sleep 1; date; done

3) Sometimes, kill all the scripts - this greatly improves reliability of the reproducer, to invoke faster link dropping in parallel (together with prompt link establishment later on):

while true; do kill $(ps aux | grep just | grep python | awk '{ print $2 }'); sleep 10; sleep $(($((RANDOM))%10)); date; done

Hi,

    can you please provide the explanation for step 1 and step 2 :
    I did not get them . is there any command for same ?

      1. Start qpid broker with queues "queue.0", "queue.1" to "queue.9"
      2. Start qdrouterd with link routing queue. to the qpid router

thanks and regards,
Tazim

(In reply to Tazim Kolhar from comment #3)
> Hi,
> 
>     can you please provide the explanation for step 1 and step 2 :
>     I did not get them . is there any command for same ?
> 
>       1. Start qpid broker with queues "queue.0", "queue.1" to "queue.9"
>       2. Start qdrouterd with link routing queue. to the qpid router
> 
> thanks and regards,
> Tazim

Sure.

>       1. Start qpid broker with queues "queue.0", "queue.1" to "queue.9"

Just add the queued to the running broker on Satellite:

for i in $(seq 0 9); do
  qpid-config add queue queue.${i} --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671
done

(ensure the queues are present by:

qpid-stat --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 -q | grep queue

)

>       2. Start qdrouterd with link routing queue. to the qpid router

Add to /etc/qpid-dispatch/qdrouterd.conf this section:

linkRoutePattern {
    prefix: queue.
    connector: broker
}

and restart qdrouterd service to apply it.


The reproducer is little bit artificial (not following steps a Satellite user would do to get qdrouterd stuck), but I suppose it is sufficient (it just mimicks such activity done on Content Hosts via the reproducer script).

VERIFIED:

# rpm -qa | grep foreman
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch
foreman-vmware-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3.5-1.el7sat.noarch
foreman-selinux-1.7.2.17-1.el7sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.14-1.el7sat.noarch
foreman-ovirt-1.7.2.50-1.el7sat.noarch
foreman-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman_docker-1.2.0.24-1.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.15.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_docker-0.0.3.10-1.el7sat.noarch
foreman-debug-1.7.2.50-1.el7sat.noarch
foreman-proxy-1.7.2.8-1.el7sat.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch
foreman-discovery-image-3.0.5-3.el7sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch
foreman-libvirt-1.7.2.50-1.el7sat.noarch
foreman-gce-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman-0.1.4.15-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.23-1.el7sat.noarch
foreman-postgresql-1.7.2.50-1.el7sat.noarch
foreman-compute-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman-redhat_access-0.2.4-1.el7sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.10-1.el7sat.noarch


steps:
>       1. Start qpid broker with queues "queue.0", "queue.1" to "queue.9"

Just add the queued to the running broker on Satellite:

for i in $(seq 0 9); do
  qpid-config add queue queue.${i} --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671
done

(ensure the queues are present by:

qpid-stat --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 -q | grep queue

)

>       2. Start qdrouterd with link routing queue. to the qpid router

Add to /etc/qpid-dispatch/qdrouterd.conf this section:

linkRoutePattern {
    prefix: queue.
    connector: broker
}

and restart qdrouterd service to apply it.

Script itself:

#!/usr/bin/python

from time import sleep
from uuid import uuid4

from proton import ConnectionException, Timeout

from proton.utils import BlockingConnection

import threading
import traceback
import os

ROUTER_ADDRESS = "proton+amqp://10.34.84.156:5672"
ADDRESS = "queue"
HEARTBEAT = 2
SLEEP = 2.0
THREADS = 10

class ReceiverThread(threading.Thread):
    def __init__(self, _id, address=ADDRESS):
        super(ReceiverThread, self).__init__()
        self._id = _id
        self.address = address
        print self.address
        self.running = True
        self.conn = None

    def connect(self):
        try:
            self.conn = BlockingConnection(ROUTER_ADDRESS, ssl_domain=None, heartbeat=HEARTBEAT)
            self.conn.create_receiver(self.address, name=str(uuid4()), dynamic=False, options=None)
        except Exception:
            self.conn = None

    def run(self):
        while self.running:
            while self.conn == None:
                self.connect()
            sleep(SLEEP)
            try:
                print "%s: reconnecting " % self.address
                self.conn.close()
            except Exception, e:
                print e
                pass
            self.conn = None

threads = []
for i in range(THREADS):
  threads.append(ReceiverThread(i, '%s.%s' %(ADDRESS, i)))
  sleep(SLEEP/THREADS)
  threads[i].start()

while True:
  sleep(10)


4. Run that script 7 times in parallel (there is a chance it will coredump - if so re-run it 7 times again)

# qdstat -b 0.0.0.0:5647 --ssl-certificate /etc/pki/katello/qpid_router_client.crt --ssl-key /etc/pki/katello/qpid_router_client.key --ssl-trustfile /etc/pki/katello/certs/katello-default-ca.crt -g
Router Statistics
  attr           value
  ======================================================
  Mode           interior
  Area           0
  Router Id      hp-dl180g6-01.rhts.eng.bos.redhat.com
  Address Count  12
  Link Count     2
  Node Count     0

No deadlock / missing response. 
Confirmed with the developers too. got reply
that getting response is more important 

moving it to Verified

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0052