We tried the same workaround adding in network-enviorment.yaml 

NetworkDeploymentActions: ['CREATE','UPDATE'] 

Still same issue where same network is trying to create ..

We also tried adding following

NetworkDeploymentActions: ['CREATE']

Still the same issue errors out with network exists.

Some things I see from the templates and deploy command...

They aren't including network-isolation.yaml in the deployment but have included the bulk of the contents from network-isolation.yaml in network-environment.yaml file:
   OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml

*** This section is normally in network-isolation.yaml
  OS::TripleO::Network::External: /usr/share/openstack-tripleo-heat-templates/network/external.yaml
  OS::TripleO::Network::InternalApi: /usr/share/openstack-tripleo-heat-templates/network/internal_api.yaml
  OS::TripleO::Network::Storage: /usr/share/openstack-tripleo-heat-templates/network/storage.yaml
...

This breaks the normal inclusion of network-isolation.yaml first and network-environment.yaml but as such is probably not a problem.  The main issue is that its not possible to tell if they did a deployment first without network-isolation.yaml and then modified network-environment.yaml at a later date to include the settings for OS::TripleO::Network::External.  Based on the error messages such as these, its likely that at one point the deployment did not include the isolated network definitions:   
Conflict: resources.ExternalNetwork: Unable to create the flat network. Physical network external is in use.

This update failure message also indicates there has been a change to the VipPort network definition between deployments, most likely from non-isolated to isolated.
NotSupported: resources.VipPort: Update to properties network of VipPort (OS::Neutron::Port) is not supported.
overcloud.StorageMgmtVirtualIP:
  resource_type: OS::TripleO::Network::Ports::StorageMgmtVipPort
  physical_resource_id: 286d8922-2aaa-4db4-8542-a3cfd612d735
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted

As far as now to recover from this. From what I understand its recommended to delete the existing set of resources and redeploy, I know some guys in the field have done this previously (e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1572017).  Its best to restore the config to a known working config but I don't know if there are any backups available.

Also, it would also be useful to see the output of "openstack port list".
Based on this stack failure:

Update to properties network of VipPort (OS::Neutron::Port) is not supported.
overcloud.StorageMgmtVirtualIP:
  resource_type: OS::TripleO::Network::Ports::StorageMgmtVipPort
  physical_resource_id: 286d8922-2aaa-4db4-8542-a3cfd612d735
  status: UPDATE_FAILED
  status_reason: |
    UPDATE aborted

It looks like the StorageMgmtVipPort was created at some point and that output will confirm it.

In network-environment.yaml there is now no StorageMgmtVipPort being created:  
# Port assignments for the VIPs
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/vip.yaml

which is why the error occurs.

Daniel - I agree.

Also, since they’re not including network-isolation.yaml but have partially included the defines from network-isolation.yaml in network-environment.yaml its possible they are missing some necessary settings. At a minimum I’d recommend adding the definitions for the StorageMgmtNetwork, even though this network is not being used, i.e:

 OS::TripleO::Network::StorageMgmt: ../network/storage_mgmt.yaml
 OS::TripleO::Network::Ports::StorageMgmtVipPort: ../network/ports/storage_mgmt.yaml
 OS::TripleO::Controller::Ports::StorageMgmtPort: ../network/ports/storage_mgmt.yaml

This should help with the error:
Update to properties network of VipPort (OS::Neutron::Port) is not supported.
overcloud.StorageMgmtVirtualIP:

Following that, its likely there could be other issues with network already in use.  As indicated earlier, its very hard to tell the history here and what ports were defined in network-environment.yaml in a previous deployment since they didn’t include the standard network-isolation.yaml and we don't know the revision history of network-environment.yaml.  There probably are existing neutron DB entries that need to be removed.

You can try changes in Comment 25 to add StoragMgmtVipPort since network-isolation.yaml was not used, which may help with failure from Comment 19. With no backups it may be necessary to remove existing neutron DB entries following this.

It looks like the problem was first introduced on 10/10, we can see isolated ports in the deployment being changed to non-isolated (noop.yaml), which isn’t good:

heat-engine.log-20181013.gz:2018-10-10 13:55:15.938 3545 WARNING heat.engine.environment [req-9740b97b-7163-4503-b3a8-9431a3f6678f - - - - -] Changing OS::TripleO::Compute::Ports::InternalApiPort from http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/user-files/usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml to http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/network/ports/noop.yaml
heat-engine.log-20181013.gz:2018-10-10 13:55:15.940 3545 WARNING heat.engine.environment [req-9740b97b-7163-4503-b3a8-9431a3f6678f - - - - -] Changing OS::TripleO::Controller::Ports::StoragePort from http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/user-files/usr/share/openstack-tripleo-heat-templates/network/ports/storage.yaml to http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/network/ports/noop.yaml

it looks like that resulted in these errors:
| Networks                                  | 37f41c7d-458d-49e6-91b3-5c0533face2e | resources.Networks: Conflict: resources.StorageNetwork.resources.StorageSubnet: Unable to complete operation on subnet 005f5d92-82b5-4a05-9e2a-ebc8d5adf169: One or more ports have an IP allocation from this subnet. | UPDATE_FAILED      | 2018-10-10T13:56:44Z |
|                                           |                                      | Neutron server returns request_ids: ['re                                                                                                                                                                               |                    |                      |
| overcloud                                 | 6aae59cb-db4d-4781-a1d8-bc21ad0e6201 | resources.Networks: Conflict: resources.StorageNetwork.resources.StorageSubnet: Unable to complete operation on subnet 005f5d92-82b5-4a05-9e2a-ebc8d5adf169: One or more ports have an IP allocation from this subnet. | UPDATE_FAILED      | 2018-10-10T13:56:44Z |


Its not clear what caused the isolated ports to change to noop.yaml, whether it was the"openstack overcloud node delete" or if the templates changed.


On 10/11 we see those ports changed back from noop.yaml to an isolated port:

heat-engine.log-20181013.gz:2018-10-11 14:01:10.635 3545 WARNING heat.engine.environment [req-e442a727-b2c0-48eb-bf70-a90e09dcd8bd - - - - -] Changing OS::TripleO::Compute::Ports::InternalApiPort from http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/network/ports/noop.yaml to http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/user-files/usr/share/openstack-tripleo-heat-templates/network/ports/internal_api.yaml
heat-engine.log-20181013.gz:2018-10-11 14:01:10.636 3545 WARNING heat.engine.environment [req-e442a727-b2c0-48eb-bf70-a90e09dcd8bd - - - - -] Changing OS::TripleO::Controller::Ports::StoragePort from http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/network/ports/noop.yaml to http://100.71.249.40:8080/v1/AUTH_b45f28b402d54025bf6a9d023ce96d58/overcloud/user-files/usr/share/openstack-tripleo-heat-templates/network/ports/storage.yaml

which caused:
| Networks                                  | df2caf81-3999-401f-95be-3a867472495a | resources.Networks: resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.                                                     | UPDATE_FAILED      | 2018-10-11T14:07:14Z |
|                                           |                                      | Neutron server returns request_ids: ['req-4cbe16be-5511-454a-b010-a8304a57bbd1']                                                                                                                                       |                    |                      |
| overcloud                                 | 7316fd54-b2da-439e-81ea-c38e69aad139 | resources.Networks: resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.                                                     | UPDATE_FAILED      | 2018-10-11T14:07:14Z |
|                                           |                                      | Neutron server returns request_ids: ['req-4cbe16be-5511-454a-b010-a8304a57bbd1'] 


BTW - hold off on making the change to StoragMgmtVipPort in network-environment.yaml for now.

So however it happened, the heat database is in a problematic state for the InternalApi, External, and Storage networks since Oct. 10, which is why deploys since then are failing. We are working on a plan to recover the heat database and will have more shortly.

Thomas has updated the case:

============================================================

I had  a look at the database dump. It looks indeed like a "classic" environment change. I crafted the following queries, which basically cancels out the updates done. To be run in the undercloud against the heat database:

# Remove old internal subnet
DELETE from resource WHERE id=2013;

# Update new one
UPDATE resource set nova_instance='4296e4d2-8494-439f-9c01-6d4f5b616d80', action='CREATE' WHERE id=3339;

# Remove old external subnet
DELETE from resource WHERE id=2015;

# Update new one
UPDATE resource set nova_instance='d9f95eaf-6271-4022-9ff7-0cc293e8fc2c', action='CREATE' WHERE id=3345;

# Remove old storage subnet
DELETE from resource WHERE id=2017;

# Update new one
UPDATE resource set nova_instance='005f5d92-82b5-4a05-9e2a-ebc8d5adf169', action='CREATE' WHERE id=3342;

# Remove old internal network
DELETE from resource WHERE id=2014;

# Update new one
UPDATE resource set nova_instance='3cfad9eb-3306-44ab-8319-a5cea6cecb00', status='COMPLETE' WHERE id=3380;

# Remove old external network
DELETE from resource WHERE id=2016;

# Update new one
UPDATE resource set nova_instance='d443be1e-7196-48d6-8bce-f16fb0d330d5', status='COMPLETE' WHERE id=3381;

# Remove old storage network
DELETE from resource WHERE id=2018;

# Update new one
UPDATE resource set nova_instance='11c15afb-f2f6-46cd-978c-4ace5f2538cc', status='COMPLETE' WHERE id=3382;


Once done, restart heat-engine, and retry to deploy.

Regarding RCA, from the case which has been closed:
Trying to retrace the issue. The problem for heat appears here:

2018-10-10 13:55:15.938 3545 WARNING heat.engine.environment: Changing OS::TripleO::Compute::Ports::InternalApiPort from internal_api.yaml to noop.yaml

From mistral engine logs we can see:

2018-10-10 13:53:28.682 1540 INFO workflow_trace [-] Starting workflow [name=tripleo.plan_management.v1.update_deployment_plan..]
2018-10-10 13:54:53.811 1540 INFO workflow_trace [-] Starting workflow [name=tripleo.scale.v1.delete_node..]

node delete doesn't call update_deployment_plan, node deploy does.

In the mistral api logs:

2018-10-10 13:53:39.456 3515 INFO mistral.api.controllers.v2.environment [-] Update environment ... u'environments': [{u'path': u'overcloud-resource-registry-puppet.yaml'}]}

user-environment is missing here.

There is no corresponding deploy_plan workflow calls, so it's like that they called "openstack overcloud deploy --update-plan-only --templates", without the environment files.

========

Closing this bug.