Bug 1546236

After the initial attempt to send an event to Qpid fails, HornetQ will periodically resend the message to Qpid until it has been received. When the attempt to redeliver the event message fails, candlepin's connections to Qpid grow after each attempt until the max number of permissible connections have been made.

Once this occurs, this will prevent further communication with Qpidd; even from qpid tooling such as qpid-stat or qpid-config.


Steps To Reproduce
==================

1. Deploy candlepin with Qpid support.
deploy -gtaq

2. Stop Tomcat once deployed
sudo systemctl stop tomat

3. Disable candlepin's suspend mode /etc/candlepin/candlepin.conf
candlepin.suspend_mode_enabled=false

4. Disable SSL requirement for Qpidd so we can use the 'drain' tool
# Disable SSL in /etc/qpid/qpidd.conf
ssl-require-client-authentication=no

$ sudo systemctl restart qpidd

5. Re-configure the event queue and exchanges
# Delete the event queue and create a new one with configuration to enable flow stop (max-queue-size=16384).
# NOTE:  Run from the base candlepin checkout dir.
sudo qpid-config --ssl-certificate=server/bin/qpid/keys/candlepin.crt --ssl-key server/bin/qpid/keys/candlepin.key -b amqps://localhost:5671 del queue allmsg --force
sudo qpid-config --ssl-certificate=server/bin/qpid/keys/candlepin.crt --ssl-key server/bin/qpid/keys/candlepin.key -b amqps://localhost:5671 add queue allmsg --durable --max-queue-size=16384
sudo qpid-config --ssl-certificate=server/bin/qpid/keys/candlepin.crt --ssl-key server/bin/qpid/keys/candlepin.key -b amqps://localhost:5671 bind event allmsg '#'

6. Clean out any existing hornetQ data
$ sudo rm -rf /var/lib/candlepin/hornetq/*

7. Restart candlepin
sudo systemctl restart tomcat

8. Register a system to this candlepin (I used candlepin's test data)
sudo subscription-manager register -u admin -p admin --org admin

9. List the available subscriptions for this system, and make note of your favorite pool ID
$ sudo subscription-manager list --avail --all

10. List the details of the event queue. You should see that after registration a couple of events were put on the 'allmsg' queue.
$ sudo qpid-stat --ssl-certificate=server/bin/qpid/keys/candlepin.crt --ssl-key server/bin/qpid/keys/candlepin.key -b amqps://localhost:5671 -q allmsg

11. In a separate terminal, tail the candlepin log so that it can be monitored.
$ tail -f /var/log/candlepin/candlepin.log

12. Trigger an endless attach/unattach pattern for your registed system. Sub in the pool_id from above.
# while true; do subscription-manager remove --all; subscription-manager attach --pool=YOUR_FAV_POOL_ID; date; sleep 1; done

13. Watch the log file until you start seeing errors. This can take a few minutes to fill the Qpid queue to its configured max.

14. Once you start seeing the errors, check to make sure that HornetQ's queue starts to store the failed messages. You should see the message counts start to rise.
$ curl -k -u admin:admin https://localhost:8443/candlepin/admin/queues

16. Let the errros continue to occur, and monitor the connections that are made to Qpidd. Once HornetQ starts retrying these failed messages, you see the connection counts start to rise.

$ sudo netstat -anp | grep qpidd | grep -c ESTAB


Result:

Letting the failures continue to occur for a long period will eventually exhaust all connections, requiring a qpidd restart. Candlepin does not seem to be able to recover from this.

Expected Result:

Candlepin should better manange the connections that it opens to Qpid. If an exception happens while sending to Qpid, the connection should be closed, and re-opened on the next attempt.