Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2097551

Summary:	DCN: A new site with DistributedComputeHCIScaleOut nodes fails to deploy
Product:	Red Hat OpenStack	Reporter:	Marian Krcmarik <mkrcmari>
Component:	openstack-tripleo-heat-templates	Assignee:	Brendan Shephard <bshephar>
Status:	CLOSED ERRATA	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	high	Docs Contact:
Priority:	high
Version:	17.0 (Wallaby)	CC:	bshephar, cjeanner, mburns, oblaut, pweeks
Target Milestone:	beta	Keywords:	Regression, Triaged
Target Release:	17.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-14.3.1-0.20220706080800.feca772.el9ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-09-21 12:22:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marian Krcmarik 2022-06-16 00:00:38 UTC

Description of problem:
If a new site (stack) is deploy in DCN topology with DistributedComputeHCIScaleOut nodes It fails on stack validation and throws following error:
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud heat.common.exception.StackValidationFailed: Value.map_merge: [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     nested_stack.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = f(*args, **kwargs)
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 996, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise exception.StackValidationFailed(
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud heat.common.exception.StackValidationFailed: outputs.role_data.value.map_merge: [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     nested_stack.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = f(*args, **kwargs)
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = res.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = super(TemplateResource, self).validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.validate_nested_stack()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise exception.StackValidationFailed(
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud heat.common.exception.StackValidationFailed: resources.5<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/deployment/haproxy/haproxy-edge-container-puppet.yaml>.outputs.role_data.value.map_merge: [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 75, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     nested_stack.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = f(*args, **kwargs)
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = res.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.validate_nested_stack()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/heat/resource_chain.py", line 107, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(ResourceChain, self).validate_nested_stack()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise exception.StackValidationFailed(
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud heat.common.exception.StackValidationFailed: resources.ServiceChain<nested_stack>.resources.5<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/deployment/haproxy/haproxy-edge-container-puppet.yaml>.outputs.role_data.value.map_merge: [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred:
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/common/context.py", line 416, in wrapped
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return func(self, ctx, *args, **kwargs)
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 848, in create_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     stack = self._parse_template_and_validate_stack(
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/service.py", line 756, in _parse_template_and_validate_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     stack.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osprofiler/profiler.py", line 160, in wrapper
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = f(*args, **kwargs)
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/stack.py", line 969, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = res.validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/template_resource.py", line 267, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     result = super(TemplateResource, self).validate()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 65, in validate
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.validate_nested_stack()
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/heat/engine/resources/stack_resource.py", line 80, in validate_nested_stack
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise exception.StackValidationFailed(
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud heat.common.exception.StackValidationFailed: resources.DistributedComputeHCIScaleOutServiceChain<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/common/services/distributedcomputehciscaleout-role.yaml>.resources.ServiceChain<nested_stack>.resources.5<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/deployment/haproxy/haproxy-edge-container-puppet.yaml>.outputs.role_data.value.map_merge: [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.090 226215 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-15 21:22:41.099 226215 ERROR openstack [-] Failed to deploy: ERROR: resources.DistributedComputeHCIScaleOutServiceChain<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/common/services/distributedcomputehciscaleout-role.yaml>.resources.ServiceChain<nested_stack>.resources.5<file:///home/stack/overcloud-deploy/dcn1/tripleo-heat-templates/deployment/haproxy/haproxy-edge-container-puppet.yaml>.outputs.role_data.value.map_merge: : [1].ansible_group_vars.map_merge: [0].get_attr: The specified reference "HAPRoxyBase" (in unknown) is incorrect.

It started to happen with openstack-tripleo-heat-templates-14.3.1-0.20220607161058.ced328c.el9ost.noarch where the following patch was included - https://github.com/openstack/tripleo-heat-templates/commit/a3dd02377394c6fab4a556f0c666a04755fa905c and It seems to touch the place where the traceback points to so It may be related.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-14.3.1-0.20220607161058.ced328c.el9ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy DCN site with DistributedComputeHCIScaleOut nodes

Actual results:
Stack verification fails

Comment 2 Brendan Shephard 2022-06-16 02:53:18 UTC

Hi,

I have pushed patch to fix this now:
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846084

Comment 3 Marian Krcmarik 2022-06-16 10:40:03 UTC

(In reply to Brendan Shephard from comment #2)
> Hi,
> 
> I have pushed patch to fix this now:
> https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846084

Nice, I did not notice the typo at all.

Comment 12 errata-xmlrpc 2022-09-21 12:22:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543