Bug 1257717 - Environment files must be passed anew on every stack update
Environment files must be passed anew on every stack update
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: z2
: 7.0 (Kilo)
Assigned To: Zane Bitter
Alexander Chuzhoy
: Triaged, ZStream
: 1264159 (view as bug list)
Depends On: 1261921
Blocks: 1255931 1256538 1265010
  Show dependency treegraph
Reported: 2015-08-27 14:39 EDT by Zane Bitter
Modified: 2018-02-08 05:52 EST (History)
12 users (show)

See Also:
Fixed In Version: openstack-heat-2015.1.1-1.el7ost
Doc Type: Enhancement
Doc Text:
On a PATCH update (using the "-x" flag in the 'heat stack-update' command), the existing environment is now retained unless explicitly overridden. This is because the Orchestration service now re-uses other parts of the environment, not just the parameters that were passed previously and not overriden. This feature was added because in the most common stack update cases, users prefer to maintain the current environment (including resource mappings and the like). This will also prevent any unintended changes in complex deployments whenever users forget to include the required environment files at stack creation time.
Story Points: ---
Clone Of:
Last Closed: 2015-10-08 08:20:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 154619 None None None Never

  None (edit)
Description Zane Bitter 2015-08-27 14:39:59 EDT
Every time a user performs a stack update, they must pass the files and the environment definition to Heat, even if none of them have changed. With the Heat client able to combine multiple environment files into a single environment to pass to Heat, collecting them all again can be a burdensome process for the user. 

One symptom of this is that making any change to and OSP Director-installed OpenStack cloud is fraught with danger, as the entire configuration (in the form of extra environment files) has to be repeated without error each time.

We already have a client flag that allows Heat to re-use existing parameter values upon a stack update by using the PATCH (instead of PUT) verb in the API. When doing PATCH, Heat should preserve *all* of the environment, not just the parameters.

An implementation of this has already merged upstream in Liberty (2015.2): https://review.openstack.org/#/c/154619 It should be relatively simple to backport.
Comment 4 Zane Bitter 2015-09-17 13:22:33 EDT
*** Bug 1264159 has been marked as a duplicate of this bug. ***
Comment 5 Alexander Chuzhoy 2015-09-18 11:29:39 EDT

The deployment command used:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 -e <yaml> --ntp-server x.x.x.x  --timeout 90 --compute-flavor compute --control-flavor control  --rhel-reg --reg-method portal --reg-org <orgid>  --reg-activation-key '<key>'

The update command (simply doesn't have the yaml file):
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1  --ntp-server x.x.x.x  --timeout 90 --compute-flavor compute --control-flavor control  --rhel-reg --reg-method portal --reg-org <orgid>  --reg-activation-key '<key>'

Results in:
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Stack failed with status: resources.Networks: resources.StorageMgmtNetwork: Conflict: resources.StorageMgmtSubnet: Unable to complete operation on subnet 521b0677-7689-4649-b81e-2ef5818e78f7. One or more ports have an IP allocation from this subnet.
ERROR: openstack Heat Stack update failed.
Comment 6 Zane Bitter 2015-09-18 12:26:23 EDT
My best guess is that we're not using the (python API equivalent of) the -x CLI option (to use PATCH instead of PUT to update the stack) in all of the "openstack overcloud ..." commands.
Comment 8 Jan Provaznik 2015-09-21 14:02:33 EDT
I was able to get a very similar error:
$ openstack overcloud deploy --templates --debug -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml 2>&1 | tee netlog.create
<stack created by OC service configuration failed - probably wrong network setup>

$ openstack overcloud deploy --templates --debug 2>&1 | tee netlog.update
Stack failed with status: resources.Networks: resources.ExternalNetwork: Conflict: resources.ExternalSubnet: Unable to complete operation on subnet ed6c6677-9eeb-4ea5-af36-5345bbbfde27. One or more ports have an IP allocation from this subnet.

This seems to be caused by same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1255931#c11 - an extra env file used on create overrides resource_registry for some resources, then when this registry env file is missing on update, resource_registry is changed which causes replacement of these resources:

[stack@instack ~]$ heat resource-list -n 5 overcloud|grep FAIL| ExternalSubnet                              | ed6c6677-9eeb-4ea5-af36-5345bbbfde27          | OS::Neutron::Subnet                               | DELETE_FAILED   | 2015-09-21T14:57:03Z | ExternalNetwork                             |
| StorageSubnet                               | ff9a3495-2e2b-4469-b7d7-c6f1898b8d9c          | OS::Neutron::Subnet                               | DELETE_FAILED   | 2015-09-21T14:57:03Z | StorageNetwork                              |
| InternalApiSubnet                           | e44f6beb-1bfd-499c-84d2-cea05cad6db7          | OS::Neutron::Subnet                               | DELETE_FAILED   | 2015-09-21T14:57:04Z | InternalNetwork                             |
| StorageMgmtSubnet                           | c0e5c0e9-a101-4546-8424-2813aa99b852          | OS::Neutron::Subnet                               | DELETE_FAILED   | 2015-09-21T14:57:04Z | StorageMgmtNetwork                          |
| TenantSubnet                                | 975a9582-fc12-4383-bc56-8f567f35d712          | OS::Neutron::Subnet                               | DELETE_FAILED   | 2015-09-21T14:57:04Z | TenantNetwork                               |
| Networks                                    | 8d0f4329-a0de-4df0-84aa-d3c3bc59c314          | OS::TripleO::Network                              | UPDATE_FAILED   | 2015-09-21T15:20:54Z |                                             |
| ExternalNetwork                             | df2f773a-6ee9-478f-81aa-8ae4d4d955bf          | OS::TripleO::Network::External                    | UPDATE_FAILED   | 2015-09-21T15:20:57Z | Networks                                    |
| TenantNetwork                               | 826dd7dd-a0ad-4e61-8a25-5af914a8f0bb          | OS::TripleO::Network::Tenant                      | UPDATE_FAILED   | 2015-09-21T15:20:57Z | Networks                                    |
| StorageMgmtNetwork                          | a42fae15-ff4e-48f2-af4c-133d63cc80f9          | OS::TripleO::Network::StorageMgmt                 | UPDATE_FAILED   | 2015-09-21T15:20:58Z | Networks                                    |
| InternalNetwork                             | d8b37fba-c722-4e64-8633-23285eb03a24          | OS::TripleO::Network::InternalApi                 | UPDATE_FAILED   | 2015-09-21T15:20:59Z | Networks                                    |
| StorageNetwork                              | 1e7eb6b1-6ea7-45d2-8eb6-86f5291ef653          | OS::TripleO::Network::Storage                     | UPDATE_FAILED   | 2015-09-21T15:20:59Z | Networks                                    |

[stack@instack ~]$ cd /usr/share/openstack-tripleo-heat-templates/
[stack@instack openstack-tripleo-heat-templates]$ grep -r TenantSubnet 
network/tenant.yaml:  TenantSubnetName:
network/tenant.yaml:  TenantSubnet:
network/tenant.yaml:      name: {get_param: TenantSubnetName}
[stack@instack openstack-tripleo-heat-templates]$ grep -r tenant.yaml
environments/network-isolation.yaml:  OS::TripleO::Network::Tenant: ../network/tenant.yaml
environments/network-isolation.yaml:  OS::TripleO::Controller::Ports::TenantPort: ../network/ports/tenant.yaml
environments/network-isolation.yaml:  OS::TripleO::Compute::Ports::TenantPort: ../network/ports/tenant.yaml
[stack@instack openstack-tripleo-heat-templates]$ grep -r OS::TripleO::Network::Tenant
overcloud-resource-registry-puppet.yaml:  OS::TripleO::Network::Tenant: network/noop.yaml
network/networks.yaml:    type: OS::TripleO::Network::Tenant
environments/network-isolation.yaml:  OS::TripleO::Network::Tenant: ../network/tenant.yaml
overcloud-resource-registry.yaml:  OS::TripleO::Network::Tenant: network/noop.yaml

related rpms:
[root@instack rdomanager_oscplugin]# rpm -qa|grep osc
[root@instack rdomanager_oscplugin]# rpm -qa|grep heat

IF default environment file is not passed on update operation at all, then update passes. To not pass the default env file, this patch for python-tuskarclient was used:
[root@instack rdomanager_oscplugin]# diff -r v1/overcloud_deploy.py.orig v1/overcloud_deploy.py.changes
<         resource_registry_path = os.path.join(tht_root, RESOURCE_REGISTRY_NAME)
>             resource_registry_path = os.path.join(tht_root, RESOURCE_REGISTRY_NAME)
>             environments = [resource_registry_path, env_path]
>         else:
>             environments = []
<         environments = [resource_registry_path, env_path]
Comment 9 Zane Bitter 2015-09-21 17:05:49 EDT
That looks right to me, it explains everything that we're seeing. We'll also need to check other commands (like "openstack overcloud update") and make sure they don't pass the default environment files on updates any more either.
Comment 10 Zane Bitter 2015-09-21 17:24:45 EDT
Since the issue we're now encountering is in a different component, I created bug 1265010 to cover that. I'm moving this openstack-heat bug back to MODIFIED because there's no issue that we know of with the Heat feature, it's purely client-side issues with rdo-manager specifically.
Comment 11 Alexander Chuzhoy 2015-10-01 12:04:27 EDT


Deployed the stack with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1  --ceph-storage-scale 0 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/network-environment.yaml --ntp-server X.X.X.X  --timeout 90 --compute-flavor compute --control-flavor control -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --neutron-bridge-mappings datacentre:br-ex  --neutron-network-vlan-ranges datacentre:<range>  --neutron-disable-tunneling --neutron-network-type vlan

Updated the stack with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1   --ntp-server X.X.X.X  --timeout 90 --compute-flavor compute --control-flavor control  --neutron-bridge-mappings datacentre:br-ex  --neutron-network-vlan-ranges datacentre:<range>  --neutron-disable-tunneling --neutron-network-type vlan

Completed successfully.
Comment 13 errata-xmlrpc 2015-10-08 08:20:45 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.