2239130 – During scale up of computes, stack fails with ResourceGroup "Compute" Stack "overcloud" Timed out

Bug 2239130 - During scale up of computes, stack fails with ResourceGroup "Compute" Stack "overcloud" Timed out

Summary: During scale up of computes, stack fails with ResourceGroup "Compute" Stack "...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-tripleoclient
Sub Component:
Version:	17.1 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	OSP Team
QA Contact:	David Rosenfeld
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-09-15 11:54 UTC by Asma Syed Hameed
Modified:	2024-12-13 15:22 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2024-12-13 15:22:33 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-28754	0	None	None	None	2023-09-15 11:55:46 UTC

Description Asma Syed Hameed 2023-09-15 11:54:33 UTC

Description of problem:
We have 3 controllers, 7 ceph and 350 virtual computes deployed. During scale up of compute nodes by 100 the stack failed.

2023-09-13 16:49:43Z [overcloud.Compute]: CREATE_FAILED  CREATE aborted (Task create from ResourceGroup "Compute" Stack "overcloud" [b9e181ea-0881-4785-8f45-065c54ba8391] Timed out)
2023-09-13 16:49:43Z [overcloud.Compute]: UPDATE_FAILED  Stack UPDATE cancelled
2023-09-13 16:49:43Z [overcloud]: CREATE_FAILED  Timed out


2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 375, in _try_overcloud_deploy_with_compat_yaml
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self._heat_deploy(stack_name, overcloud_yaml,
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 156, in _heat_deploy
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.deploy_without_plan(
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 80, in deploy_without_plan
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise exceptions.DeploymentError("Heat Stack create failed.")
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud tripleoclient.exceptions.DeploymentError: Heat Stack create failed.
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1268, in take_action
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.set_deployment_status(
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1206, in take_action
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.deploy_tripleo_heat_templates(
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 339, in deploy_tripleo_heat_templates
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self._try_overcloud_deploy_with_compat_yaml(
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 384, in _try_overcloud_deploy_with_compat_yaml
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise ValueError(messages)
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ValueError: Failed to deploy: Heat Stack create failed.
2023-09-14 00:04:36.071 936354 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2023-09-14 00:04:36.074 936354 ERROR openstack [-] Failed to deploy: Heat Stack create failed.^[[00m
Failed to deploy: Heat Stack create failed.
2023-09-14 00:04:36.074 936354 DEBUG osc_lib.shell [-] clean_up DeployOvercloud: Failed to deploy: Heat Stack create failed. clean_up /usr/lib/python3.9/site-packages/osc_lib/shell.py:496^[[00m
2023-09-14 00:04:36.094 936354 INFO osc_lib.shell [-] END return value: 1^[[00m

Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230802.n.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy 350+ overcloud nodes
2. Perform scale up of computes

Actual results:
Stack create failed

Expected results:
Stack create successful

Additional info:
time openstack overcloud deploy \
--timeout 500 \
--templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server x.x.x.x \
--deployed-server \
-e /home/stack/templates/overcloud-vip-deployed.yaml \
-e /home/stack/templates/overcloud-networks-deployed.yaml \
-e /home/stack/templates/overcloud-baremetal-deployed.yaml \
-e /home/stack/templates/overcloud-ceph-deployed.yaml \
--networks-file /home/stack/virt/network/network_data_v2.yaml \
-e /home/stack/virt/internal.yaml \
-e /home/stack/virt/network/network-environment_v2.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
--stack-only

Note You need to log in before you can comment on or make changes to this bug.