Bug 1497248 - HypervisorUpdateJob should only ever run one instance per Org at a time
Summary: HypervisorUpdateJob should only ever run one instance per Org at a time
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Candlepin
Classification: Community
Component: candlepin
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 2.1
Assignee: candlepin-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: 1497707
TreeView+ depends on / blocked
 
Reported: 2017-09-29 14:56 UTC by Michael Stead
Modified: 2017-10-10 13:31 UTC (History)
3 users (show)

Fixed In Version: candlepin-2.1.6-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1497707 (view as bug list)
Environment:
Last Closed: 2017-10-10 13:31:25 UTC


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github candlepin candlepin pull 1728 0 None None None 2017-10-02 13:50:49 UTC

Description Michael Stead 2017-09-29 14:56:55 UTC
If multiple requests are made to start an async hypervisor update to the same owner/org, candlepin will start the first, and queue up the other jobs to be run once the first has completed (WAITING state). Once the first job finishes, ALL other hypervisor update jobs that are in the WAITING state get triggered and run at the same time despite having the restriction of one checkin job can run per owner.

This can potentially allow these intensive jobs to queue up and upon execution and impact candlepin performance.


Steps to reproduce for candlepin:

1) To simulate a long checkin, put a sleep at the top of HypervisorUpdateJob.toExecute with start/end log before/after. Otherwise, make sure that the host/guest mappings in the curl command contain enough data to make the request take a bit of time to complete.

2) Tail the logs:
tail -f /var/log/candlepin/candlepin.log | grep "job=hypervisor_update"

3) Issue multiple simultaneous requests to start an async checkin via curl:

for i in {1..10}; do curl -k -u admin:admin -X POST -H "Content-Type: text/plain" -d '{"hypervisors":[]}' https://localhost:8443/candlepin/hypervisors/snowwhite; done


Results:

The log will show that the initial job will start. Once the job finishes, the log will show that the other 9 have been started together.

Expected Results:
The jobs should be executed one at a time in order of being created. The queuing  of the jobs should be allowed, but per owner, they should get run one at a time.


Note You need to log in before you can comment on or make changes to this bug.