Created attachment 1477543 [details] undercloud install lag Description of problem: While trying to manually deploy OPS13 UC on my host I now* hit some timeout issue, see sctual results section. *On a fresh server on which I had uninstalled puppet which was preinstalled by Theforeman and caused this bug https://bugzilla.redhat.com/show_bug.cgi?id=1618992#c8 Version-Release number of selected component (if applicable): rhel7.5 rpm -qa | grep triple puppet-tripleo-8.3.4-5.el7ost.noarch openstack-tripleo-common-containers-8.6.3-10.el7ost.noarch openstack-tripleo-ui-8.3.2-1.el7ost.noarch python-tripleoclient-9.2.3-4.el7ost.noarch ansible-tripleo-ipsec-8.1.1-0.20180308133440.8f5369a.el7ost.noarch openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch openstack-tripleo-puppet-elements-8.0.1-1.el7ost.noarch openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch openstack-tripleo-validations-8.4.2-1.el7ost.noarch openstack-tripleo-common-8.6.3-10.el7ost.noarch How reproducible: Unsure first time I hit this. Steps to Reproduce: 1. Follow guide and hit issue while running: openstack undercloud install 2. 3. Actual results: Fail to complete UC deployment: 2018-08-21 16:33:07,795 INFO: Generated new ssh key in ~/.ssh/id_rsa 2018-08-21 16:34:57,942 INFO: Created flavor "baremetal" with profile "None" 2018-08-21 16:35:25,230 INFO: Created flavor "control" with profile "control" 2018-08-21 16:35:25,659 INFO: Created flavor "compute" with profile "compute" 2018-08-21 16:35:26,035 INFO: Created flavor "ceph-storage" with profile "ceph-storage" 2018-08-21 16:35:26,601 INFO: Created flavor "block-storage" with profile "block-storage" 2018-08-21 16:35:27,109 INFO: Created flavor "swift-storage" with profile "swift-storage" 2018-08-21 16:35:27,111 INFO: Configuring Mistral workbooks 2018-08-21 16:37:08,371 INFO: Mistral workbooks configured successfully 2018-08-21 16:43:14,799 ERROR: TIMEOUT waiting for execution 42f95343-e6a3-4afb-bf59-40c20cb61527 to finish. State: RUNNING 2018-08-21 16:43:14,799 DEBUG: An exception occurred Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1852, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: TIMEOUT waiting for execution 42f95343-e6a3-4afb-bf59-40c20cb61527 to finish. State: RUNNING 2018-08-21 16:43:14,802 ERROR: ############################################################################# Undercloud install failed. Reason: TIMEOUT waiting for execution 42f95343-e6a3-4afb-bf59-40c20cb61527 to finish. State: RUNNING See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1852, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: TIMEOUT waiting for execution 42f95343-e6a3-4afb-bf59-40c20cb61527 to finish. State: RUNNING Command 'instack-install-undercloud' returned non-zero exit status 1 Expected results: Should complete UC deployment. Additional info:
Created attachment 1477568 [details] sosrepot
Is there some context as to why this is a HardwareProvisioning issue?
Hello Bob, Sorry, my fault - reallocating to Workflows as it's apparently a timeout in Mistral.
FYI, suspect timeout was indeed the source of problem. On same host bumped up to 20 min, rebooted host. Managed to complete undercloud deploy this time around: 2018-08-27 16:39:05,421 INFO: ############################################################################# Undercloud install complete. The file containing this installation's passwords is at /home/stack/undercloud-passwords.conf. There is also a stackrc file at /home/stack/stackrc. These files are needed to interact with the OpenStack services, and should be secured. ############################################################################# All that remain to verify is check that 20 min code update lands in RPM.
FYI I'd reinstalled same system, bumped to 600 still hit timeout. Bumped to figure up to 700 plus restarted hosts before it managed to complete UC deployment. Not sure what did it the extra +100 or host reboot. Don't use my system as a reference as it's geriatric HW, least 5Y old :)
upstream patch not backported to stable/queens, moving back to ASSIGNED
(In reply to Tzach Shefi from comment #6) > FYI, suspect timeout was indeed the source of problem. > On same host bumped up to 20 min, rebooted host. > Managed to complete undercloud deploy this time around: > > 2018-08-27 16:39:05,421 INFO: > ############################################################################# > Undercloud install complete. > > The file containing this installation's passwords is at > /home/stack/undercloud-passwords.conf. > > There is also a stackrc file at /home/stack/stackrc. > > These files are needed to interact with the OpenStack services, and should be > secured. > > ############################################################################# > > All that remain to verify is check that 20 min code update lands in RPM. Hi Tzach, I hit this exact issue. Please let me know how I can modify the timeout to 20 mins since I'm new to openstack. Thanks, Linh
2018-11-05 17:22:44,451 INFO: Created flavor "ceph-storage" with profile "ceph-storage" 2018-11-05 17:22:44,500 INFO: Created flavor "block-storage" with profile "block-storage" 2018-11-05 17:22:44,551 INFO: Created flavor "swift-storage" with profile "swift-storage" 2018-11-05 17:22:44,552 INFO: Configuring Mistral workbooks 2018-11-05 17:22:58,774 INFO: Mistral workbooks configured successfully 2018-11-05 17:29:01,368 ERROR: TIMEOUT waiting for execution 8eead3f0-2bf8-45c4-aac9-4f75ff81b462 to finish. State: RUNNING 2018-11-05 17:29:01,368 DEBUG: An exception occurred Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1852, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: TIMEOUT waiting for execution 8eead3f0-2bf8-45c4-aac9-4f75ff81b462 to finish. State: RUNNING 2018-11-05 17:29:01,368 ERROR: ############################################################################# Undercloud install failed. Reason: TIMEOUT waiting for execution 8eead3f0-2bf8-45c4-aac9-4f75ff81b462 to finish. State: RUNNING See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1852, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: TIMEOUT waiting for execution 8eead3f0-2bf8-45c4-aac9-4f75ff81b462 to finish. State: RUNNING Command 'instack-install-undercloud' returned non-zero exit status 1 [stack@undercloud-director ~]$
I increased the timeout, but now having a different error. 2018-11-06 00:48:05,280 INFO: Configuring Mistral workbooks 2018-11-06 00:48:19,476 INFO: Mistral workbooks configured successfully 2018-11-06 01:02:54,085 ERROR: ERROR error creating the default Deployment Plan overcloud Check the create_default_deployment_plan execution in Mistral with openstack workflow execution list Mistral execution ID: 63224e63-d0fc-4793-a1d6-be39a6c31c6b 2018-11-06 01:02:54,085 DEBUG: An exception occurred Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1844, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: ERROR error creating the default Deployment Plan overcloud Check the create_default_deployment_plan execution in Mistral with openstack workflow execution list Mistral execution ID: 63224e63-d0fc-4793-a1d6-be39a6c31c6b 2018-11-06 01:02:54,086 ERROR: ############################################################################# Undercloud install failed. Reason: ERROR error creating the default Deployment Plan overcloud Check the create_default_deployment_plan execution in Mistral with openstack workflow execution list Mistral execution ID: 63224e63-d0fc-4793-a1d6-be39a6c31c6b See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2337, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2029, in _post_config _post_config_mistral(instack_env, mistral, swift) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1965, in _post_config_mistral _create_default_plan(mistral, plans) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1907, in _create_default_plan fail_on_error=True) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1844, in _wait_for_mistral_execution raise RuntimeError(error_message) RuntimeError: ERROR error creating the default Deployment Plan overcloud Check the create_default_deployment_plan execution in Mistral with openstack workflow execution list Mistral execution ID: 63224e63-d0fc-4793-a1d6-be39a6c31c6b Command 'instack-install-undercloud' returned non-zero exit status 1 [stack@undercloud-director ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0068