Bug 2228077

Summary:	[OSP 16] heat stack fails while nova continues to create vms in the background
Product:	Red Hat OpenStack	Reporter:	Flavio Piccioni <fpiccion>
Component:	openstack-nova	Assignee:	OSP DFG:Compute <osp-dfg-compute>
Status:	NEW ---	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	16.2 (Train)	CC:	dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, tkajinam, vromanso
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Flavio Piccioni 2023-08-01 10:32:39 UTC

Description of problem:
Customer is trying to onboard NFV workloads creating vms using heat.
Expected target is to create 100vms at once, but currently the stack is failing trying to create 50vms at once


2023-08-01 08:59:02Z [centos_temp_50_VMs.vm_20]: CREATE_FAILED  ClientException: resources.vm_20: Unknown Error (HTTP 504)
2023-08-01 08:59:02Z [centos_temp_50_VMs]: CREATE_FAILED  Resource CREATE failed: ClientException: resources.vm_20: Unknown Error (HTTP 504)


Version-Release number of selected component (if applicable):
RHOSP 16.2.5 (Juniper's Contrail env)

How reproducible:
Run a stack that creates 50vms at time

Actual results:
The stack fails, but the vms building process continue in background 


Expected results:
100vms stack to succeed

Additional info:
Trying to create a smaller stack (30vms) stack succeed.
We tried tuning haproxy's "timeout  check" from 10 to 30 seconds with no luck