Bug 884683

With the current scheduler implementation, there is a race condition which allows a job to "jump the queue" when a system is released during a scheduling pass. This may result in deadlock if two recipe sets are competing for systems with limited availability, and recipe set A acquires a system or systems needed by recipe set B and vice-versa.

This can be resolved by making the scheduling process more event driven, rather than relying entirely on the current scheduling loop.

Recipe-based scheduling: when a job is first sent to the Beaker server, all recipes are checked against the current pool of idle systems (including dynamic virt resources). If an appropriate system is available, it is assigned immediately, otherwise the recipe enters the recipe queue.

System-based scheduling: when a system becomes free (first added to Beaker, completes execution of a recipe, switched from Broken or Manual back to Automated), Beaker checks the recipe queue for suitable work. If a recipe is found, then it is assigned directly to the system and the system never actually enters the idle pool.

The scheduling loop remains in place to perform cleanup of the recipe queue (e.g. aborting dead recipes that can never be completed because all suitable systems have been removed from Beaker, checking for free resources in the dynamic virtualisation pool)

*** This bug has been marked as a duplicate of bug 872187 ***