Description of problem:
So far we've had two incidents where the job queue got into a state where it wasn't keeping up with the inflow of Rules Engine jobs and we had to manually delete all jobs from the queue.

Diagnosis showed that the queue contained a large number of jobs that were blocked because the queue contained older jobs for the same bugs.  The job queue processor was stuck because it kept choosing the blocked jobs (and having to decline them) rather than choosing unblocked jobs.

This happens because the decision of which job to choose from the queue is amde only based on the priority of the job (useless, as all jobs currently have the same priority) and whether the job's grab_until and run_after times are in the past.  This effectively turns the job queue into a pool of jobs where most jobs are "runnable", and because of the way MySQL works, the same set of jobs keep getting selected (and declined), ad infinitum.

The solution:
The job queue needs to be an actual queue.  The oldest eligible jobs need to be processed first, and if a job needs to be declined it must go to the end of the queue, either by resetting the insert_time to the current time or discarding the job and creating a new one.  The former option may be easier, as jobs have other attributes that need to be preserved if the job is declined (e.g. the retry count).