This wasn't seen as an actual bug but is more the effect of some conversations and investigations I've been doing. From a conversation with jconnor it's definitely an issue, but at very least I hope this bug causes a deeper investigation than I've done so far.

= Background =
In the task subsystem, the AsyncTask subclass of Task handles gofer's asynchronous with correlated reply setup. The task kernel-level thread ends quicky once the request is made into gofer. When the consumer replies back from the operation, the gofer callback updates the task with the response. This way, we can leverage the task subsystem for the auditing/results of async consumer operations.

The task remains in a "running" state until the callback is received, even though the kernel-level thread ends. That part is key.

= Issue =
The thread for the task ends quickly once gofer acknowledges the invocation. That's good.

The bad part is that the fact that the task is set at "running" means it's effectively taking up a spot in our running task queue. The number of running spots is defaulted to 4 and applies to _all_ Pulp task-based operations (repo sync, cds sync).

The issue can be easily seen in the install package to consumer group implementation. Each consumer install is tracked as an individual task in the task subsystem (that makes sense). That means that any consumer group of over 4 consumers will completely take over Pulp.

This is especially a problem in the consumer package install case. The majority of the work there is done on the consumer itself (barring the bandwidth on Pulp for the consumer to download the packages), which means the Pulp server is sitting idle during those times.

Keeping in mind the desired consumer scale of Pulp, we can't have a consumer group of 1000 consumers blocking all Pulp operations (repo sync, cds sync, etc) while each consumer takes the time to install packages. Since the queue is FIFO, all syncs will be backed up until the install is complete on all 1000 consumers (ok, 997 of them).

= Solutions =

== Introduce an Idle State ==
Instead of leaving tasks in the "running" state while waiting on the callback, they go into an idle state. This would allow other threads to enter the running state while waiting on the callback.

I don't know enough of the details of the task subsystem, but the prospect of having that many idle tasks sitting at once just feels wrong.

We will also probably need to add logic to cap the number of idle threads and block more from happening. So when 500 consumer installs are going on and a task comes in attempting to trigger the 501st, the task subsystem detects that the next state of that new task would be idle and prevents it from running until there's room for it to transition to idle.

Of course, that sort of look ahead logic isn't trivial.  :)

== Split Pulp-centric and Consumer-centric Tasks into Two Queues ==
We're in this position because of the nature of tasks that require a gofer callback. That implies that the actual work of the task is being done off of the Pulp server. Therefore, we may be able to track those tasks separate from ones that will hammer the server.

For example, we could continue to allow 4 concurrent Pulp server tasks but, say, 100 concurrent consumer tasks, under the rationale that those 100 are not actively working to melt the Pulp server. That leaves the bulk of the current task infrastructure in place; tasks remain as running but the kernel-level thread dies.

The only complication is in the manager layer to determine which queue to use both when adding a task to the queue and potentially when looking up a task status. I suspect this is simpler than adding in an idle state and all of the rules that have to go in place for idle tasks, but again, I'm not closely familiar with the task code.