Bug 1277292

+++ This bug was initially created as a clone of Bug #1269509 +++

Description of problem:

When you attempt to install errata(s) onto a host via
the errata managment tab of a host. If that operation
will take longer to complete than approx ~120 seconds the operation will time out.

We narrowed this down to an issue with qpid:

"""
The difference in performance between satellite and using YUM directly to install errata (as reported in
#01519801) can be attributed to a known issue [2] with Qpid and AMQP 1.0.  Further it explains why this
performance is a regression in Satellite 6.1.  In 6.0, AMQP-0-10 was used.  In 6.1, AMQP-1.0 was introduced
to support Qpid Dispatch Router on Capsules.  Using AMQP-0-10, the client can explicitly trigger the flush
by sending a sync control (or setting the sync flag on the transfer). There is no such explicit mechanism in
AMQP 1.0, so at present the flush happens after a short timeout on the broker.  During the development and
testing of python-gofer-proton, I noted an approximate 1 second delay sending messages.  After discussion
with Ted Ross, Gordon Sim and Ken Giusti, this finding was confirmed.

So, what this means to Satellite is that it takes 1 second to send every message when using AMQP 1.0 to
durable queues when Qpid persistence is enabled (which it is).  The messaging flow during an errata install
looks something like this:

Satellite           Agent
- -----------------------------
   |                  |
   | request -------->|
   |<------- accepted |
   |<------- started  |
   |<------- progress*|
   |<------- result   |
   |                  |

* A TON OF THESE

For example: 549 (progress) messages where sent on a test system when installing 191 packages associated
with an Errata install.  At 1 second per progress message this adds 549 seconds (9 minutes) to the install.
 These numbers matched test results.  With progress reporting enabled: ~18 min.  Without progress reporting:
~9 min.

Solutions:

1. Fix the AMQP-1.0 issue in Qpid.  This has been described by the Qpid teams as difficult so unlikely this
will get fixed soon.  Or, that the fixed version be ported to RHEL6.

2. Disconnect progress reporting in the katelo agent plugin.  This solution is simple but assumes Katello
can live without progress reported for agent tasks.  This progress is reported to Pulp and is included in
the Pulp task.

3. A variation on #2.  If katello needs the reported progress information we could rate-limit the reported
progress.  This mitigates the issue but not as good as #1 or #2.


The disconnect delays introduced by AMQP-1.0 heartbeats that I observed and reported earlier seems to have
no significant impact on the performance of installing Errata.

Regards,

Jeff
"""

For 6.1.z we opted for (2) from above as merged into katello-2.2's agent:

https://github.com/Katello/katello-agent/pull/32

but we need a more permanent solution for 6.2 (or we just need to carry this change forward in 6.2 if we decide to just keep it as-is).

If this bug is being looked at near the end of the 6.2 cycle and we are out of time, just get the above mentioned PR merged into 6.2 and avoid any regressions.