Bug 1159961

+++ This bug was initially created as a clone of Bug #1159303 +++

I heard this second hand, so this BZ is about investigating if this is an issue or not. The observation that there might be a problem is on an installation of Pulp where the broker has many (100+) pulp.agent.<uuid> queues, but only a few consumers registered to the system.

Expected Behavior:
I expect that when I create a consumer a pulp.agent.<uuid> queue is created, and when I delete a consumer the corresponding pulp.agent.<uuid> is also removed from the broker.

Why is this important:
Lingering, unnecessary queues could cause qpid to run out of file descriptors.

--- Additional comment from  on 2014-10-31 10:20:11 EDT ---

After some investigation it was determined that Pulp does not remove old consumer subscribers during unregistration. The deletion of the pulp.agent.<uuid> is the expected behavior, so we should fix this.

After some discussion with jortel, we determined two changes need to be made:

1) pulp.agent.<uuid> needs to be 100% managed server side. This includes the creation and deletion of the queues. The 'manager' code for the consumer can handle this responsibility. Goferd needs to be adjusted to optionally not declare queues from the consumer side, and the pulp-agent and katello-agent will need small changes so that they know the server will create and delete the consumer queues.

2) A reaper task that removes these queues needs to be added. This runs periodically with some frequency and removes "orphaned_queues" that have been orphaned after an unregister event longer than X amount of time. These two options will be specified in server.conf and will come with defaults.

3) Docs need to be written on this indicating how cleanup occurs.

The reaper design is necessary versus a "delete right now during unregistration" design because there is one final message that needs to be delivered to the consumer, and we should allow a reasonable amount of time for that to occur before the server force-deletes the queue.

--- Additional comment from Pavel Moravec on 2014-10-31 13:51:44 EDT ---

(In reply to bbouters from comment #1)
> 2) A reaper task that removes these queues needs to be added. This runs
> periodically with some frequency and removes "orphaned_queues" that have
> been orphaned after an unregister event longer than X amount of time. These
> two options will be specified in server.conf and will come with defaults.

Note you can use auto-delete queues with deletion timeout (x-declare argument qpid.auto_delete_timeout). If a queue loses its latest consumer, the qpid broker waits for the timeout seconds and then it deletes the queue itself. This might simplify the implementation. Note however that if the queue never has a consumer, it wont be deleted any time (the auto-delete timer is triggered _only_ when latest consumer unsubscribes). Also I am not sure if the same functionality is available in RabbitMQ.

--- Additional comment from  on 2014-10-31 16:20:58 EDT ---

Auto delete would be an elegant way to solve this, but I'm not sure how it would allow for consumers to be powered off for longer than the auto_delete timeout. A consumer that was running, but then had its agent service stopped or is powered off will wake up at some point in the future to find its queue missing.

Today, if an agent finds its queue missing it recreates it so it would recover, but any meaningful messages that were issued to the consumer (ie: bind or unbind) while it is turned off would be missing.

This could cause some very unexpected behaviors for the user where they remember doing an action such as binding a group of consumers to a given repo, but whoever wasn't powered on at that moment doesn't receive the config.

Suggestions for how we can make the auto_delete approach work are welcome because it is more elegant but I'm not sure how to resolve this one lingering problem.

--- Additional comment from Pavel Moravec on 2014-11-03 05:48:14 EST ---

(In reply to bbouters from comment #3)
> Auto delete would be an elegant way to solve this, but I'm not sure how it
> would allow for consumers to be powered off for longer than the auto_delete
> timeout. A consumer that was running, but then had its agent service stopped
> or is powered off will wake up at some point in the future to find its queue
> missing.

Good point. What about auto_delete queue (optionally with some timeout) that has alternate exchange set, as well as the exchange routing messages to these queues has the same alt.exchange set. Plus having one durable auxiliary queue that gets all messages routed via the alt.exchange. Then:

- when deleting the queue due to auto-delete parameter, all messages in the queue are redirected via the alt.exchange to the aux.queue
- when a consumer is off and broker should deliver a message to its (deleted) queue, it finds the original exchange does not have a matching binding (to the deleted queue) so the broker re-routes the message to the alt.exchange, i.e. to the aux.queue

When a consumer is starting, if it detects its queue is gone, it would have to call QMF method "queueMoveMessages" that moves messages from aux.queue to the proper newly created pulp.agent.<uuid> queue, with some proper filter (to move just messages relevant to the pulp consumer).

Gotchas:
- needs some more testing and probably bigger code changes
- if some default exchange (like "" one) is used for distributing the messages to pulp.agent.<uuid> queues, you cant set alternate exchange for such an exchange. But you can create a new exchange for this traffic.
- this solution does not cover situations when a pulp consumer wont power on at all (not sure if allowed scenario). Then the aux.queue would keep messages for this consumer forever. But IMHO this problem is common to any solution - is there a way how to identify a consumer wont ever power on?