Bug 1127574

Summary: Dispatch multiple requests to OpenStack in parallel
Product: [Retired] Beaker Reporter: Nick Coghlan <ncoghlan>
Component: schedulerAssignee: matt jia <mjia>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: medium    
Version: developCC: dcallagh, dowang, ebaak, mjia, rjoost, tpelka
Target Milestone: 24.0Keywords: FutureFeature, Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-21 18:51:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nick Coghlan 2014-08-07 07:50:05 UTC 
The initial OpenStack integration only makes a single VM creation request at a time. This clearly doesn't scale in systems which may be attempting to schedule dozens or hundreds of recipes in parallel.

This needs to be changed so the requests can be dispatched in parallel.

Constraints:
- the current MySQL client library is known to be incompatible with gevent, so that can't be used to provide asynchronous IO for the requests
- we've had issues with MySQL deadlocks when attempting to run recipe state updates in parallel, so we're wary of switching back to that configuration

Proposal:
- use the backported concurrent.futures module from Python 3: https://pypi.python.org/pypi/futures

This provides a nice abstraction for submitting work to a thread pool as callables and then waiting for the results: https://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecutor-example

Alternatively, we could use the threading and Queue modules directly, but I think the futures abstraction is a better choice.
Comment 1 matt jia 2016-10-19 04:06:12 UTC 
I have tried using concurrent.futures to dispatch multiple requests to OpenStack. Everything seems working fine when I run five jobs in my testing environment. I can see that the requests were dispatched asynchronously to OpenStack.

On Gerrit:

   https://gerrit.beaker-project.org/#/c/5332/

Note: python-futures is not available on RHEL6 so we have to build it by ourselves.
Comment 4 Dan Callaghan 2017-02-21 18:51:02 UTC 
Beaker 24.0 has been released.