Every time a user performs a stack update, they must pass the files and the environment definition to Heat, even if none of them have changed. With the Heat client able to combine multiple environment files into a single environment to pass to Heat, collecting them all again can be a burdensome process for the user. One symptom of this is that making any change to and OSP Director-installed OpenStack cloud is fraught with danger, as the entire configuration (in the form of extra environment files) has to be repeated without error each time. We already have a client flag that allows Heat to re-use existing parameter values upon a stack update by using the PATCH (instead of PUT) verb in the API. When doing PATCH, Heat should preserve *all* of the environment, not just the parameters. An implementation of this has already merged upstream in Liberty (2015.2): https://review.openstack.org/#/c/154619 It should be relatively simple to backport.
*** Bug 1264159 has been marked as a duplicate of this bug. ***
FailedQA Environment: openstack-heat-common-2015.1.1-4.el7ost.noarch openstack-heat-api-cfn-2015.1.1-4.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-62.el7ost.noarch openstack-heat-templates-0-0.6.20150605git.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.1-4.el7ost.noarch openstack-heat-api-2015.1.1-4.el7ost.noarch python-heatclient-0.6.0-1.el7ost.noarch openstack-heat-engine-2015.1.1-4.el7ost.noarch The deployment command used: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 -e <yaml> --ntp-server x.x.x.x --timeout 90 --compute-flavor compute --control-flavor control --rhel-reg --reg-method portal --reg-org <orgid> --reg-activation-key '<key>' The update command (simply doesn't have the yaml file): openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ntp-server x.x.x.x --timeout 90 --compute-flavor compute --control-flavor control --rhel-reg --reg-method portal --reg-org <orgid> --reg-activation-key '<key>' Results in: Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates Stack failed with status: resources.Networks: resources.StorageMgmtNetwork: Conflict: resources.StorageMgmtSubnet: Unable to complete operation on subnet 521b0677-7689-4649-b81e-2ef5818e78f7. One or more ports have an IP allocation from this subnet. ERROR: openstack Heat Stack update failed.
My best guess is that we're not using the (python API equivalent of) the -x CLI option (to use PATCH instead of PUT to update the stack) in all of the "openstack overcloud ..." commands.
I was able to get a very similar error: $ openstack overcloud deploy --templates --debug -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml 2>&1 | tee netlog.create <stack created by OC service configuration failed - probably wrong network setup> $ openstack overcloud deploy --templates --debug 2>&1 | tee netlog.update ... Stack failed with status: resources.Networks: resources.ExternalNetwork: Conflict: resources.ExternalSubnet: Unable to complete operation on subnet ed6c6677-9eeb-4ea5-af36-5345bbbfde27. One or more ports have an IP allocation from this subnet. This seems to be caused by same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1255931#c11 - an extra env file used on create overrides resource_registry for some resources, then when this registry env file is missing on update, resource_registry is changed which causes replacement of these resources: [stack@instack ~]$ heat resource-list -n 5 overcloud|grep FAIL| ExternalSubnet | ed6c6677-9eeb-4ea5-af36-5345bbbfde27 | OS::Neutron::Subnet | DELETE_FAILED | 2015-09-21T14:57:03Z | ExternalNetwork | | StorageSubnet | ff9a3495-2e2b-4469-b7d7-c6f1898b8d9c | OS::Neutron::Subnet | DELETE_FAILED | 2015-09-21T14:57:03Z | StorageNetwork | | InternalApiSubnet | e44f6beb-1bfd-499c-84d2-cea05cad6db7 | OS::Neutron::Subnet | DELETE_FAILED | 2015-09-21T14:57:04Z | InternalNetwork | | StorageMgmtSubnet | c0e5c0e9-a101-4546-8424-2813aa99b852 | OS::Neutron::Subnet | DELETE_FAILED | 2015-09-21T14:57:04Z | StorageMgmtNetwork | | TenantSubnet | 975a9582-fc12-4383-bc56-8f567f35d712 | OS::Neutron::Subnet | DELETE_FAILED | 2015-09-21T14:57:04Z | TenantNetwork | | Networks | 8d0f4329-a0de-4df0-84aa-d3c3bc59c314 | OS::TripleO::Network | UPDATE_FAILED | 2015-09-21T15:20:54Z | | | ExternalNetwork | df2f773a-6ee9-478f-81aa-8ae4d4d955bf | OS::TripleO::Network::External | UPDATE_FAILED | 2015-09-21T15:20:57Z | Networks | | TenantNetwork | 826dd7dd-a0ad-4e61-8a25-5af914a8f0bb | OS::TripleO::Network::Tenant | UPDATE_FAILED | 2015-09-21T15:20:57Z | Networks | | StorageMgmtNetwork | a42fae15-ff4e-48f2-af4c-133d63cc80f9 | OS::TripleO::Network::StorageMgmt | UPDATE_FAILED | 2015-09-21T15:20:58Z | Networks | | InternalNetwork | d8b37fba-c722-4e64-8633-23285eb03a24 | OS::TripleO::Network::InternalApi | UPDATE_FAILED | 2015-09-21T15:20:59Z | Networks | | StorageNetwork | 1e7eb6b1-6ea7-45d2-8eb6-86f5291ef653 | OS::TripleO::Network::Storage | UPDATE_FAILED | 2015-09-21T15:20:59Z | Networks | [stack@instack ~]$ cd /usr/share/openstack-tripleo-heat-templates/ [stack@instack openstack-tripleo-heat-templates]$ grep -r TenantSubnet network/tenant.yaml: TenantSubnetName: network/tenant.yaml: TenantSubnet: network/tenant.yaml: name: {get_param: TenantSubnetName} [stack@instack openstack-tripleo-heat-templates]$ grep -r tenant.yaml environments/network-isolation.yaml: OS::TripleO::Network::Tenant: ../network/tenant.yaml environments/network-isolation.yaml: OS::TripleO::Controller::Ports::TenantPort: ../network/ports/tenant.yaml environments/network-isolation.yaml: OS::TripleO::Compute::Ports::TenantPort: ../network/ports/tenant.yaml [stack@instack openstack-tripleo-heat-templates]$ grep -r OS::TripleO::Network::Tenant overcloud-resource-registry-puppet.yaml: OS::TripleO::Network::Tenant: network/noop.yaml network/networks.yaml: type: OS::TripleO::Network::Tenant environments/network-isolation.yaml: OS::TripleO::Network::Tenant: ../network/tenant.yaml overcloud-resource-registry.yaml: OS::TripleO::Network::Tenant: network/noop.yaml related rpms: [root@instack rdomanager_oscplugin]# rpm -qa|grep osc python-rdomanager-oscplugin-0.0.10-5.el7ost.noarch [root@instack rdomanager_oscplugin]# rpm -qa|grep heat openstack-heat-engine-2015.1.1-4.el7ost.noarch openstack-heat-api-2015.1.1-4.el7ost.noarch openstack-heat-templates-0-0.6.20150605git.el7ost.noarch python-heatclient-0.6.0-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-64.el7ost.noarch openstack-heat-common-2015.1.1-4.el7ost.noarch openstack-heat-api-cfn-2015.1.1-4.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.1-4.el7ost.noarch heat-cfntools-1.2.8-2.el7.noarch IF default environment file is not passed on update operation at all, then update passes. To not pass the default env file, this patch for python-tuskarclient was used: [root@instack rdomanager_oscplugin]# diff -r v1/overcloud_deploy.py.orig v1/overcloud_deploy.py.changes 508c508,511 < resource_registry_path = os.path.join(tht_root, RESOURCE_REGISTRY_NAME) --- > resource_registry_path = os.path.join(tht_root, RESOURCE_REGISTRY_NAME) > environments = [resource_registry_path, env_path] > else: > environments = [] 510d512 < environments = [resource_registry_path, env_path]
That looks right to me, it explains everything that we're seeing. We'll also need to check other commands (like "openstack overcloud update") and make sure they don't pass the default environment files on updates any more either.
Since the issue we're now encountering is in a different component, I created bug 1265010 to cover that. I'm moving this openstack-heat bug back to MODIFIED because there's no issue that we know of with the Heat feature, it's purely client-side issues with rdo-manager specifically.
Verified: Environment: openstack-heat-engine-2015.1.1-5.el7ost.noarch Deployed the stack with: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml -e /home/stack/network-environment.yaml --ntp-server X.X.X.X --timeout 90 --compute-flavor compute --control-flavor control -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --neutron-bridge-mappings datacentre:br-ex --neutron-network-vlan-ranges datacentre:<range> --neutron-disable-tunneling --neutron-network-type vlan Updated the stack with: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ntp-server X.X.X.X --timeout 90 --compute-flavor compute --control-flavor control --neutron-bridge-mappings datacentre:br-ex --neutron-network-vlan-ranges datacentre:<range> --neutron-disable-tunneling --neutron-network-type vlan Completed successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1865