[OSP13] Overcloud deployment fail when attempting to call heat-templates
not from their default path (/usr/share/openstack-tripleo-heat-templates).
When deployment command is using custom path for the heat-templates directory, the overcloud deployment will attempt to search for ceratin scripts, such run-os-net-config.sh in a wrong path and that will cause overcloud deployment to fail.
(*) attempt to deploy overcloud with templates that are not on their default path.
For example, change the template path by adding this to deploy command:
openstack overcloud deploy --templates openstack-tripleo-heat-templates/
- Overcloud deployment fails.
It's attempting to search scripts in a wrong path
Cannot find : openstack-tripleo-heat-templates/usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
Omri, could you please give the full command used please? We need to see which templates / network config you're trying to use.
(In reply to Emilien Macchi from comment #2)
> Omri, could you please give the full command used please? We need to see
> which templates / network config you're trying to use.
Sure, actaully this deploy_command was used by UI:DFG team to deploy osp13 on their BM envrionment :
[stack@puma01 ~]$ cat deploy_command
openstack overcloud deploy --templates openstack-tripleo-heat-templates/ -e openstack-tripleo-heat-templates/environments/network-isolation.yaml -e openstack-tripleo-heat-templates/environments/network-environment.yaml -e openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml -e openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor.yaml -e /home/stack/virt/docker-images.yaml -e custom.yaml
(In reply to Omri Hochman from comment #3)
> [stack@puma01 ~]$ cat deploy_command
> openstack overcloud deploy --templates openstack-tripleo-heat-templates/ -e
> openstack-tripleo-heat-templates/environments/network-isolation.yaml -e
> openstack-tripleo-heat-templates/environments/network-environment.yaml -e
> openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml -e
> -e /home/stack/virt/docker-images.yaml -e custom.yaml
Could you please show the content of custom.yaml?
Udi - Can you please provide the content of the custom.yaml from what you had on puma01 for the OSP13 UI deployment.
custom.yaml was just this:
By the way, I also opened a bug on this same issue and it was closed as "not a bug" - see bug #1572990. I think it's still a bug, because paths should always be relative and not point to places outside the plans.
This seems also related to https://bugs.launchpad.net/tripleo/+bug/1762403
I've been beating on this scenario for a while against upstream tip, and I cannot reproduce the above results with a minimal environment, nor can I see any logic errors in the related code in overcloud_deploy.py.
When I introduce a debugger I see the variables looking correct, and if allowed to continue this finishes with a successful overcloud deployment.
* `import pdb; pdb.set_trace()` on line 361
* Ran deploy:
`openstack overcloud deploy --templates openstack-tripleo-heat-templates/ -e custom.yaml`
(undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates openstack-tripleo-heat-templates/ -e custom.yaml
Waiting for messages on queue 'tripleo' with no timeout.
-> plans = plan_management.list_deployment_plans(self.clients)
Namespace(answers_file=None, block_storage_flavor=None, block_storage_scale=None, ceph_storage_flavor=None, ceph_storage_scale=None, compute_flavor=None, compute_scale=None, config_download=True, control_flavor=None, control_scale=None, deployed_server=False, disable_password_generation=False, disable_validations=False, dry_run=False, environment_directories=['/home/stack/.tripleo/environments'], environment_files=[u'custom.yaml'], force_postconfig=False, libvirt_type=None, networks_file=None, no_cleanup=False, no_proxy='', ntp_server=None, output_dir=None, overcloud_ssh_key=None, overcloud_ssh_user='heat-admin', plan_environment_file=None, reg_activation_key='', reg_force=False, reg_method='satellite', reg_org='', reg_sat_url='', rhel_reg=False, roles_file=None, run_validations=False, skip_deploy_identifier=False, skip_postconfig=False, stack='overcloud', swift_storage_flavor=None, swift_storage_scale=None, templates=u'openstack-tripleo-heat-templates/', timeout=240, update_plan_only=False, validation_errors_fatal=True, validation_warnings_fatal=False)
My custom.yaml contents:
I'm planning to close this as NOTABUG unless someone can show me an error in my test or reproduce this in a pristine environment.
I think that there is no problem as long as the templates don't point to a directory on the filesystem which is outside the template's directory tree (for example pointing to /usr/share when the templates are on the user's home directory). Even if there is such a pointer, the deployment will pass if you run it from the CLI, and only GUI users will hit a problem (because the referenced yaml is not in the container). Omri - can you confirm?
I can confirm that outside references for -e files passed it never get it's path processed as it normally does for those sitting inside of --templates path.
There had been a few unit tests added upstream  illustrating this behavior. I think that is a design limitation we cannot handle. Is it documented?
Sounds like it confirmed as an issue .on comment#10 - adding require release-notes for osp13
@bogdan, Can you please elaborate, preferably with output how you confirmed this?
My non-standard path templates were under ~/openstack-tripleo-heat-templates/ and my custom.yaml was under ~/ directly, not inside the non-standard path provided.
I would like to learn what's going on here and you seem to know more about this than I do. :-)
@David, there is utility  used both with undercloud and overcloud heat based installations. Env files containing resources registry, when referenced outside of the templates dir, never get properly rewritten its content  nor redirected its paths relative to the templates path - compare  vs . The links to unit tests only illustrate that. In the end though, this may work for some cases, as well as may equally fail for another ones. I think Steven Shardy can comment on that better than me.
Specifically to those unit tests linked above, let me explain the issue as I understand it... So there is
rewritten correct from the original in-tree path of
(the relative path assumes it originally sits in /tmp/thtroot/inside.yaml, then
get copied into /twd/templates as the utility logic processes templates by new paths)
While the *relative* outside paths become broken, see the original
'OS::Foo::Bar': '../outside.yaml' # == /tmp/thtroot/../outside.yaml (correct)
Is broken as it does not exist - we do not copy files outside of the templates directory tree!
While the absolute paths, like one for your example @David, seems working fine.
The absolute outside paths are illustrated via the /tmp/thtroot42/notouch.yaml fixture.
(In reply to David Peacock from comment #12)
> @bogdan, Can you please elaborate, preferably with output how you confirmed
> My non-standard path templates were under
> ~/openstack-tripleo-heat-templates/ and my custom.yaml was under ~/
> directly, not inside the non-standard path provided.
> I would like to learn what's going on here and you seem to know more about
> this than I do. :-)
Been a while since I looked at this, but IIRC the following summary should be correct, and aligns with the tests referenced by Bogdan:
1. For any environment files which are j2 rendered (there are several in the tripleo-heat-templates tree from recent releases e.g to support custom networks), any -e reference *must* point to the same tree as --templates, e.g
~/openstack-tripleo-heat-templates/ or whatever.
2. If your -e custom.yaml is outside t-h-t, we try to handle it by translating any resource_registry paths to relative, because the file must be copied into the root of the plan container, e.g the same as ~/openstack-tripleo-heat-templates/.
The (1) case may be missing some docs and validation, but it's a known limitation of the design - we can't handle j2 rendering files in arbitrary locations.
(2) is an attempt to maintain backwards compatibility for the cases where folks have custom environment files that are outside the t-h-t tree, but historically there have been some bugs in this area, so if you encounter issues I'd say the first thing to confirm is if the same environment works in the --templates directory.
https://github.com/openstack/tripleo-heat-templates/blob/master/ci/common/net-config-simple-bridge.yaml#L45 was mentioned above, and I agree that's probably a bug, and would likely result in this kind of problem - all the paths inside t-h-t should be relative or the path mangling for (2) is likely to break when the tree is moved so it'd be good to confirm if that's the issue here (the CLI from comment #3 doesn't look like we are, but it'd be good to confirm)
Thank you both, Bogdan and Steven, for the enlightenment. :-)
I believe the issue is the same, my concern is that the use of the templates apparently conflicts with a feature that has been used for a while.