Red Hat Bugzilla – Bug 1257401
Extended delay between calls to create nova instances from heat-engine
Last modified: 2016-04-26 10:44:09 EDT
Description of problem:
It does not appear that heat is honoring the value of max_concurrent_builds set in the nova.conf file. Yes I know its heat and not nova.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.modify the value of max_concurrent_builds in nova.conf
Not all of the requested build happen concurrently
The level of concurrency should be honored
I did some profiling today, and discovered that most of the time is getting eaten up doing recursive calculation of the number of resources. This is something we made optional (can be disabled via config file) during Liberty via the Unlimited option max_resources_per_stack - see this patch:
Unfortunately that isn't yet backported to kilo (I'm working on a backport, few issues to sort out with the tests but otherwise should be straight forward), so to prove this is the bottleneck, it's possible to comment the validation in the code:
In my testing I observed a 500% performance penalty for having the resource number check enabled (admittedly something of a worst-case due to tailoring a template to magnify the issue).
The root_stack and total resource count thing has long been acknowledged as a problem in the heat community, but it wasn't until my testing today that I fully understood the severity of the impact, particularly for highly nested deployments like tripleo.
I raised an upstream bug to discuss disabling the check by default (this was rejected previously, but worth revisiting I think, this discussion shouldn't influence the viabillty of the backport of the unlimited option mentioned above);
Stable/kilo backport proposed for the unlimited option mentioned in comment #4: