Bug 1421052 - Adjusting OSP 10 Director components settings for scaling 100+ nodes
Summary: Adjusting OSP 10 Director components settings for scaling 100+ nodes
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Angus Thomas
QA Contact: Amit Ugol
Depends On:
TreeView+ depends on / blocked
Reported: 2017-02-10 09:06 UTC by Pablo Caruana
Modified: 2020-04-15 15:15 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-05-01 00:41:46 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Article) 3597351 0 None None None 2019-02-08 19:08:41 UTC
Red Hat Knowledge Base (Solution) 2918611 0 None None None 2017-02-10 09:16:05 UTC

Description Pablo Caruana 2017-02-10 09:06:15 UTC
Currently there is no official documentation for providing the hints tweaks for reaching deployments with 3 controllers + 97 or more computes.

Some of the bottleneck detected were sorted by increasing the  rpc_response_timeout  to at least 3600 in heat, ironic and nova configuration, enabling the memcached cached, increasing the heat engine rpc workers (48) as the default one where too low (2) and in this way it was able to  enlarge cloud to 80-90 compute node then reaching some haproxy timeouts at the controllers above the default ones.
With all those information we could create a basic solution articles explaining some of the tunables that can be used, but still, As more details can appear this should be taken as a whole, and be included in the standard documentation as it's not uncommon to have customers coming and asking about scaling the platform (specially cloud providers/ partners having their own products at scale and thus, we expect this to become even more common in the future.

Comment 2 Joe Talerico 2017-02-10 17:02:55 UTC
Can you share what is failing here? 

Any sort of "tweaks" needed should be pushed back into the product vs having some sort of one-off documentation somewhere.

Comment 18 Sai Sindhur Malleni 2019-05-01 00:41:46 UTC
There is a general OSP 10 scale guide now at https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/recommendations_for_large_deployments/index

Closing this BZ, please reopen if needed.

Note You need to log in before you can comment on or make changes to this bug.