Bug 2239130
| Summary: | During scale up of computes, stack fails with ResourceGroup "Compute" Stack "overcloud" Timed out | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Asma Syed Hameed <asyedham> |
| Component: | python-tripleoclient | Assignee: | OSP Team <rhos-maint> |
| Status: | CLOSED NOTABUG | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 17.1 (Wallaby) | CC: | jraju, jslagle, mburns, pweeks, ramishra |
| Target Milestone: | --- | Keywords: | Scale, Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-12-13 15:22:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Description of problem: We have 3 controllers, 7 ceph and 350 virtual computes deployed. During scale up of compute nodes by 100 the stack failed. 2023-09-13 16:49:43Z [overcloud.Compute]: CREATE_FAILED CREATE aborted (Task create from ResourceGroup "Compute" Stack "overcloud" [b9e181ea-0881-4785-8f45-065c54ba8391] Timed out) 2023-09-13 16:49:43Z [overcloud.Compute]: UPDATE_FAILED Stack UPDATE cancelled 2023-09-13 16:49:43Z [overcloud]: CREATE_FAILED Timed out 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 375, in _try_overcloud_deploy_with_compat_yaml 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self._heat_deploy(stack_name, overcloud_yaml, 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 156, in _heat_deploy 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.deploy_without_plan( 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 80, in deploy_without_plan 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise exceptions.DeploymentError("Heat Stack create failed.") 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud tripleoclient.exceptions.DeploymentError: Heat Stack create failed. 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred: 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1268, in take_action 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1206, in take_action 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.deploy_tripleo_heat_templates( 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 339, in deploy_tripleo_heat_templates 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self._try_overcloud_deploy_with_compat_yaml( 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 384, in _try_overcloud_deploy_with_compat_yaml 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise ValueError(messages) 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: Failed to deploy: Heat Stack create failed. 2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2023-09-14 00:04:36.074 936354 ERROR openstack [-] Failed to deploy: Heat Stack create failed.^[[00m Failed to deploy: Heat Stack create failed. 2023-09-14 00:04:36.074 936354 DEBUG osc_lib.shell [-] clean_up DeployOvercloud: Failed to deploy: Heat Stack create failed. clean_up /usr/lib/python3.9/site-packages/osc_lib/shell.py:496^[[00m 2023-09-14 00:04:36.094 936354 INFO osc_lib.shell [-] END return value: 1^[[00m Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230802.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy 350+ overcloud nodes 2. Perform scale up of computes Actual results: Stack create failed Expected results: Stack create successful Additional info: time openstack overcloud deploy \ --timeout 500 \ --templates /usr/share/openstack-tripleo-heat-templates \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \ --stack overcloud \ --libvirt-type kvm \ --ntp-server x.x.x.x \ --deployed-server \ -e /home/stack/templates/overcloud-vip-deployed.yaml \ -e /home/stack/templates/overcloud-networks-deployed.yaml \ -e /home/stack/templates/overcloud-baremetal-deployed.yaml \ -e /home/stack/templates/overcloud-ceph-deployed.yaml \ --networks-file /home/stack/virt/network/network_data_v2.yaml \ -e /home/stack/virt/internal.yaml \ -e /home/stack/virt/network/network-environment_v2.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e ~/containers-prepare-parameter.yaml \ --stack-only