Bug 1927356

Description of problem:

Hello,

For large clouds with more than 130+ nodes the stack create/update is either complete erroneously or hangs without any output or generation of overcloudrc file.

Part 1

The issue reported earlier looked like:

+--------------------------------------+-----------------+--------+------------------------+----------------+-----------+
(undercloud) $ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 9b4c9e80-d605-442a-8945-3691d60e3eb9 | overcloud  | 06823da80cc04fab952e0a0d03f5b344 | CREATE_COMPLETE | 2021-01-27T21:45:54Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
(undercloud) $ openstack overcloud status
+-----------+-------------------+
| Plan Name | Deployment Status |
+-----------+-------------------+
| overcloud |   DEPLOY_SUCCESS  |
+-----------+-------------------+

so, in actual the deployment is complete but it ends with:

Timed out waiting for messages from Execution (ID: 9a5e1227-be08-4902-a17d-4ee31a392ae3, State: RUNNING). The WebSocket timed out before the Workflow completed.
Host <> not found in /home/stack/.ssh/known_hosts
Overcloud Endpoint: https://<url>:13000
Overcloud Horizon Dashboard URL: https://<url>:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed with error

The openstack overcloud failures command listed:

 $ openstack overcloud failures
|-> Failures for host: undercloud
|--> Task: Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log
|---> censored: "the output has been hidden due to the fact that 'no_log: true' was specified for this result"
|---> changed: true

We later found that there were a lot of stale executions in ERROR & RUNNING state which were cleaned up before running another deploy.

~Timed out waiting for messages from Execution (ID: d8029597-3fe9-4425-8d37-d85e5003c574, State: RUNNING). The WebSocket timed out before the Workflow completed.
Overcloud Endpoint: https://<>:13000
Overcloud Horizon Dashboard URL: https://<>:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed with error

Part 2:

We then increased the keystone, zaqar and overcloud deploy timeout as per [1] and ran the update again with respective values of 86400s, 86400s and 1440m.

This time however the ansible got completed but the overcloud deploy command was in hung state when I last checked the status on call, 24 hours timeout was yet to be complete.

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-11.3.2-1.20200914170177.el8ost.noarch