Bug 1781265

Summary: Time to run “Render hieradata from template” is slow and not linear when double the number of overcloud nodes.
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: tripleo-ansibleAssignee: James Slagle <jslagle>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 16.0 (Train)CC: jhajyahy, lshort, ramishra
Target Milestone: rcKeywords: Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-0.4.1-0.20191212203202.62bce8e.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-06 14:43:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Schuppert 2019-12-09 16:35:46 UTC 
Description of problem:

Scale from 0 to 50 computes: ~35min
2019-12-03 23:43:12,078 p=1027190 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-04 00:18:25,265 p=1027190 u=mistral |  TASK [Server deployments] ******************************************************
Scale from 50 to 100 computes: ~2h7min
2019-12-04 08:22:04,248 p=12775 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************              
2019-12-04 10:29:10,426 p=12775 u=mistral |  TASK [Server deployments] ******************************************************             
Scale from 100 to 150 computes: ~7h7min
2019-12-03 13:58:37,355 p=950621 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-03 21:05:42,175 p=950621 u=mistral |  TASK [Server deployments] ******************************************************

This BZ is to track the following patches for downstream inclusing since they speed up the rendering
process immense:

https://review.opendev.org/#/c/697363/
https://review.opendev.org/#/c/697364/ 

This patch switches to using include_tasks instead of import_tasks for some tasks, which is much faster since ansible does not spend as much time having to skip lots of tasks:
https://review.opendev.org/697510


Version-Release number of selected component (if applicable):
tripleo-ansible-0.4.1-0.20191114131847.0c76b58.el8ost


Results with the above mentioned patches:
100 node cell update run with all 3 patches applied:
2019-12-09 11:28:05,728 p=357809 u=mistral |  TASK [tripleo-hieradata : Copy overcloud.json to all_nodes.json] ***************
2019-12-09 11:28:15,567 p=357809 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-09 11:29:03,631 p=357809 u=mistral |  TASK [Server deployments] ******************************************************
2019-12-09 11:29:25,834 p=357809 u=mistral |  TASK [Lookup deployment UUID] **************************************************
=> previously 2h
**************************************************

Scale out from 100-150 computs with all 3 patches applied:
2019-12-09 16:07:55,233 p=537767 u=mistral |  TASK [tripleo-hieradata : Copy overcloud.json to all_nodes.json] ***************
2019-12-09 16:08:08,331 p=537767 u=mistral |  TASK [tripleo-hieradata : Render hieradata from template] **********************
2019-12-09 16:09:26,715 p=537767 u=mistral |  TASK [Server deployments] ******************************************************
2019-12-09 16:09:56,899 p=537767 u=mistral |  TASK [Lookup deployment UUID] **************************************************
=> previously 7h
Comment 4 Jad Haj Yahya 2020-01-21 11:47:57 UTC 
Went through all files modified on above commits and made sure changes are present

Verified on RHOS_TRUNK-16.0-RHEL-8-20200113.n.0
Comment 8 errata-xmlrpc 2020-02-06 14:43:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283