Bug 1371442 - Stack deployment done twice with different network-environment files
Summary: Stack deployment done twice with different network-environment files
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Steve Baker
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-30 08:46 UTC by Robin Cernin
Modified: 2019-12-16 06:32 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-07 21:23:32 UTC
Target Upstream Version:


Attachments (Terms of Use)
recovery.py script used to diagnose and fix this issue (6.09 KB, text/x-python)
2016-08-30 23:03 UTC, Steve Baker
no flags Details

Description Robin Cernin 2016-08-30 08:46:02 UTC
Description of problem:

We have two network-environment files, stage and prod. Originally the overcloud was deployed using the prod network-environment files.

Then someone has run the overcloud deployment on top of prod environment with stage network-environment file. Which created new subnets and networks related to stage environment.

Undercloud then see duplicated neutron networks from both stage and prod environment:

$ neutron subnet-list
+--------------------------------------+---------------------+-----------------+-----------------------------------------------------+
| id                                   | name                | cidr            | allocation_pools                                    |
+--------------------------------------+---------------------+-----------------+-----------------------------------------------------+
| 22dc1807-5669-4a3b-9578-506bd3f7619a | storage_mgmt_subnet | 192.168.202.0/24 | {"start": "192.168.202.11", "end": "192.168.202.250"} |
| 32f483ec-9306-4bb5-a5e6-037c09bfda91 | tenant_subnet       | 192.168.201.0/24 | {"start": "192.168.201.11", "end": "192.168.201.250"} |
| 41802666-6d63-4611-bef4-e5b364553565 | internal_api_subnet | 192.168.204.0/24 | {"start": "192.168.204.11", "end": "192.168.204.250"} |
| 45876116-a72b-4f05-92f7-091a2fd68ef8 | storage_subnet      | 192.168.103.0/24 | {"start": "192.168.103.11", "end": "192.168.103.250"} |
| 47f66366-221d-4ca1-81d7-736c4dc62650 | internal_api_subnet | 192.168.104.0/24 | {"start": "192.168.104.11", "end": "192.168.104.250"} |
| 4bd8c967-f99c-49a1-9ecb-a5559e0779d4 |                     | 10.1.38.0/24    | {"start": "10.1.38.20", "end": "10.1.38.120"}       |
| 6cc46f60-5a23-45b2-9d98-fb88075a886b | storage_mgmt_subnet | 192.168.102.0/24 | {"start": "192.168.102.11", "end": "192.168.102.250"} |
| a7ce3bfe-0779-4b0d-9188-0d025e022042 | external_subnet     | 10.2.188.0/24   | {"start": "10.2.188.11", "end": "10.2.188.253"}     |
| b2bdc779-c946-4782-9977-21e789fd2be9 | external_subnet     | 10.2.188.0/24   | {"start": "10.2.188.11", "end": "10.2.188.253"}     |
| c83303c7-571e-4933-9d4b-eaf4f4cd519b | tenant_subnet       | 192.168.101.0/24 | {"start": "192.168.101.11", "end": "192.168.101.250"} |
+--------------------------------------+---------------------+-----------------+-----------------------------------------------------+

We have deleted all the duplicate subnets that matched the incorrect (staging) settings, and attempted a full stack update, unfortunately we hit the following error:

Deploying templates in the directory /home/stack/centralci-openstack-templates/templates                                                         
Stack failed with status: resources.Networks: resources.StorageNetwork: Conflict: resources.StorageSubnet: Unable to complete operation on subnet
45876116-a72b-4f05-92f7-091a2fd68ef8. One or more ports have an IP allocation from this subnet.                                      


Version-Release number of selected component (if applicable):

openstack-heat-api-2015.1.4-1.el7ost.noarch
openstack-heat-api-cfn-2015.1.4-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.4-1.el7ost.noarch
openstack-heat-common-2015.1.4-1.el7ost.noarch
openstack-heat-engine-2015.1.4-1.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-128.el7ost.noarch


How reproducible:

Deploy overcloud with two different environment files on top of each other.

Steps to Reproduce:
1. See above.
2.
3.

We need an fix to Heat DB to make sure when we run the update next time, it will not try to recreate the networks and will allow us to complete the update.

Thank you.

Comment 8 Steve Baker 2016-08-30 23:03:41 UTC
Created attachment 1196098 [details]
recovery.py script used to diagnose and fix this issue


Note You need to log in before you can comment on or make changes to this bug.