Bug 1790894

Summary: Can't add service mapping from user environment during cloud update
Product: Red Hat OpenStack Reporter: shyam.biradar
Component: python-tripleoclientAssignee: Bogdan Dobrelya <bdobreli>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: aschultz, augol, bdobreli, emacchi, hbrock, jslagle, kecarter, mburns
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-12 11:37:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description shyam.biradar 2020-01-14 13:49:14 UTC
Description of problem:
Service mappings defined in environment file are not getting detected


Version-Release number of selected component (if applicable):


How reproducible:
Consistently reproducible

Steps to Reproduce:
1. Deploy 'RHOSP 15' undercloud and overcloud.
2. Introduce two new heat resources/services, add them to env file trilio_env.yaml
resource_registry:
  OS::TripleO::Services::TrilioDatamover: docker/services/trilio-datamover.yaml
  OS::TripleO::Services::TrilioDatamoverApi: docker/services/trilio-datamover-api.yaml
3. Change roles_data to add these services to a role.

4. Update overcloud with OpenStack overcloud deploy command with additional environment file. Overcloud update fails, with resource type could not found error.

Command: 
openstack overcloud deploy --templates \
--libvirt-type qemu \
--ntp-server 172.172.3.201 \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e /home/stack/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
-e /home/stack/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml \
-e /home/stack/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/shyam/triliovault-cfg-scripts/redhat-director-scripts/trilio_env.yaml \
-r /home/stack/shyam/triliovault-cfg-scripts/redhat-director-scripts/roles_data.yaml \
--log-file overcloud_deployment_88.log


Actual Result:
Newly added service mappings are not getting detected and hence stack validation is failing.

Expected Result:
New services should be mapped to heat resources and overcloud update should work fine.



Environment:
RHOSP 15 on rhel8


Additional info:


Stack trace/logs:
Execption======================================================================================

(undercloud) [stack@undercloudstein redhat-director-scripts]$ ./trilio-deploy.sh
Removing the current plan files
Uploading new plan files
Temporary Swift GET/PUT URL parameters have successfully been updated.
Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.
Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate
    resource_name=resource_name)

  File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info
    name=resource_type)

heat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped
    return func(self, ctx, *args, **kwargs)

  File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template
    result.update(stack.get_nested_parameters(filter_parameter))

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters
    params.update(nested.get_nested_parameters(filter_func))

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters
    params.update(nested.get_nested_parameters(filter_func))

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters
    for name, rsrc in six.iteritems(self.resources):

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources
    self._resources = self._resources_for_defn(self.defn)

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn
    for name in stack_defn.enabled_rsrc_names()

  File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>
    for name in stack_defn.enabled_rsrc_names()

  File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__
    resource_name=name)

  File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate
    raise exception.StackValidationFailed(message=six.text_type(exc))

heat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.

{'result': 'Failure caused by error in tasks: notify_zaqar\n\n notify_zaqar [task_ex_id=5810deb7-259f-4d10-b3a4-b3e74c752f53] -> Workflow failed due to message status. Status:FAILED Message:Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n [wf_ex_id=7b4edd20-df4d-49cf-915c-12bab21d79b6, idx=0]: Workflow failed due to message status. Status:FAILED Message:Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n', 'swift_container': 'overcloud_ceph_ansible_fetch_dir', 'status': 'FAILED', 'message': 'Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n'}
Exception updating plan: {'result': 'Failure caused by error in tasks: notify_zaqar\n\n notify_zaqar [task_ex_id=5810deb7-259f-4d10-b3a4-b3e74c752f53] -> Workflow failed due to message status. Status:FAILED Message:Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n [wf_ex_id=7b4edd20-df4d-49cf-915c-12bab21d79b6, idx=0]: Workflow failed due to message status. Status:FAILED Message:Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n', 'swift_container': 'overcloud_ceph_ansible_fetch_dir', 'status': 'FAILED', 'message': 'Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 559, in get_class_to_instantiate\n resource_name=resource_name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 538, in get_resource_info\n name=resource_type)\n\nheat.common.exception.EntityNotFound: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/heat/common/context.py", line 410, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/service.py", line 1315, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1113, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 1095, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 318, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/stack.py", line 312, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python3.6/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python3.6/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nheat.common.exception.StackValidationFailed: The Resource Type (OS::TripleO::Services::TrilioDatamoverApi) could not be found.\n'}
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.172.3.62', 60824), raddr=('172.172.3.62', 13808)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.172.3.62', 56458), raddr=('172.172.3.62', 13000)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.172.3.62', 42572), raddr=('172.172.3.62', 13004)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.172.3.62', 57022), raddr=('172.172.3.62', 13989)>
(undercloud) [stack@undercloudstein redhat-director-scripts]$

====================================================================================

Comment 1 Alex Schultz 2020-01-14 20:10:38 UTC
*** Bug 1790895 has been marked as a duplicate of this bug. ***

Comment 2 shyam.biradar 2020-01-20 06:22:06 UTC
Hi,

Team, Any update here?

Comment 3 Emilien Macchi 2020-01-29 23:49:43 UTC
would it be possible to share the whole content of /home/stack/shyam/triliovault-cfg-scripts/redhat-director-scripts and /home/stack/tripleo-heat-templates ?

Comment 4 Bogdan Dobrelya 2020-03-18 09:56:11 UTC
The issue may be a combination of the following:
1) /home/stack/shyam/triliovault-cfg-scripts/redhat-director-scripts/trilio_env.yaml is outside of /home/stack/openstack-tripleo-heat-templates
2) trilio_env.yaml's resource_registry uses relative paths for docker/services/trilio*.yaml

this may end up with the wrong paths substituted by the templates processing engine.
Please try either:
a) placing the extra env file trilio_env.yaml into the common path /home/stack/openstack-tripleo-heat-templates shared with other env files used in deployment,
b) using absolute paths for its resource_registry entries

Comment 5 Emilien Macchi 2020-08-11 17:54:29 UTC
Please look at comment #4 and report back how it works with Bogdan's proposals.

Comment 6 shyam.biradar 2020-08-12 04:24:33 UTC
Hi Team,

This is working for us. Something was wrong in our configuration.
Worked with relative paths only.

We can close this bug.

Thank you.