Bug 1781265 - Time to run “Render hieradata from template” is slow and not linear when double the number of overcloud nodes.
Summary: Time to run “Render hieradata from template” is slow and not linear when doub...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 16.0 (Train on RHEL 8.1)
Assignee: James Slagle
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-09 16:35 UTC by Martin Schuppert
Modified: 2020-02-06 14:44 UTC (History)
3 users (show)

Fixed In Version: tripleo-ansible-0.4.1-0.20191212203202.62bce8e.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-06 14:43:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 698414 0 None MERGED Move localhost_address to net_ip_map hieradata 2020-09-21 06:12:40 UTC
OpenStack gerrit 698415 0 None MERGED Copy all_nodes.json instead of rendering template 2020-09-21 06:12:37 UTC
Red Hat Product Errata RHEA-2020:0283 0 None None None 2020-02-06 14:44:05 UTC

Description Martin Schuppert 2019-12-09 16:35:46 UTC
Description of problem:

Scale from 0 to 50 computes: ~35min
2019-12-03 23:43:12,078 p=1027190 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-04 00:18:25,265 p=1027190 u=mistral |  TASK [Server deployments] ******************************************************
Scale from 50 to 100 computes: ~2h7min
2019-12-04 08:22:04,248 p=12775 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************              
2019-12-04 10:29:10,426 p=12775 u=mistral |  TASK [Server deployments] ******************************************************             
Scale from 100 to 150 computes: ~7h7min
2019-12-03 13:58:37,355 p=950621 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-03 21:05:42,175 p=950621 u=mistral |  TASK [Server deployments] ******************************************************

This BZ is to track the following patches for downstream inclusing since they speed up the rendering
process immense:

https://review.opendev.org/#/c/697363/
https://review.opendev.org/#/c/697364/ 

This patch switches to using include_tasks instead of import_tasks for some tasks, which is much faster since ansible does not spend as much time having to skip lots of tasks:
https://review.opendev.org/697510


Version-Release number of selected component (if applicable):
tripleo-ansible-0.4.1-0.20191114131847.0c76b58.el8ost


Results with the above mentioned patches:
100 node cell update run with all 3 patches applied:
2019-12-09 11:28:05,728 p=357809 u=mistral |  TASK [tripleo-hieradata : Copy overcloud.json to all_nodes.json] ***************
2019-12-09 11:28:15,567 p=357809 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-09 11:29:03,631 p=357809 u=mistral |  TASK [Server deployments] ******************************************************
2019-12-09 11:29:25,834 p=357809 u=mistral |  TASK [Lookup deployment UUID] **************************************************
=> previously 2h
**************************************************

Scale out from 100-150 computs with all 3 patches applied:
2019-12-09 16:07:55,233 p=537767 u=mistral |  TASK [tripleo-hieradata : Copy overcloud.json to all_nodes.json] ***************
2019-12-09 16:08:08,331 p=537767 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-09 16:09:26,715 p=537767 u=mistral |  TASK [Server deployments] ******************************************************
2019-12-09 16:09:56,899 p=537767 u=mistral |  TASK [Lookup deployment UUID] **************************************************
=> previously 7h

Comment 4 Jad Haj Yahya 2020-01-21 11:47:57 UTC
Went through all files modified on above commits and made sure changes are present

Verified on RHOS_TRUNK-16.0-RHEL-8-20200113.n.0

Comment 8 errata-xmlrpc 2020-02-06 14:43:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283


Note You need to log in before you can comment on or make changes to this bug.