If multiple requests are made to start an async hypervisor update to the same owner/org, candlepin will start the first, and queue up the other jobs to be run once the first has completed (WAITING state). Once the first job finishes, ALL other hypervisor update jobs that are in the WAITING state get triggered and run at the same time despite having the restriction of one checkin job can run per owner.

This can potentially allow these intensive jobs to queue up and upon execution and impact candlepin performance.


Steps to reproduce for candlepin:

1) To simulate a long checkin, put a sleep at the top of HypervisorUpdateJob.toExecute with start/end log before/after. Otherwise, make sure that the host/guest mappings in the curl command contain enough data to make the request take a bit of time to complete.

2) Tail the logs:
tail -f /var/log/candlepin/candlepin.log | grep "job=hypervisor_update"

3) Issue multiple simultaneous requests to start an async checkin via curl:

for i in {1..10}; do curl -k -u admin:admin -X POST -H "Content-Type: text/plain" -d '{"hypervisors":[]}' https://localhost:8443/candlepin/hypervisors/snowwhite; done


Results:

The log will show that the initial job will start. Once the job finishes, the log will show that the other 9 have been started together.

Expected Results:
The jobs should be executed one at a time in order of being created. The queuing  of the jobs should be allowed, but per owner, they should get run one at a time.