Bug 1353735 - rhel-osp-director: 7.3GA +ipv6->7.3Async update fails; resources.TenantNetwork: Conflict: resources.TenantSubnet: Unable to complete operation on subnet 6971cc5c-8a13-4fb9-8043-67586b708d74. One or more ports have an IP allocation from this subnet.
Summary: rhel-osp-director: 7.3GA +ipv6->7.3Async update fails; resources.TenantNetwo...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: async
: 7.0 (Kilo)
Assignee: Angus Thomas
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-07 20:40 UTC by Alexander Chuzhoy
Modified: 2018-05-02 10:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-25 00:02:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
heat logs (3.24 MB, application/x-gzip)
2016-07-07 20:46 UTC, Alexander Chuzhoy
no flags Details

Description Alexander Chuzhoy 2016-07-07 20:40:08 UTC
rhel-osp-director:  7.3GA +ipv6->7.3Async update fails; resources.TenantNetwork: Conflict: resources.TenantSubnet: Unable to complete operation on subnet 6971cc5c-8a13-4fb9-8043-67586b708d74. One or more ports have an IP allocation from this subnet.


Environment:
openstack-puppet-modules-2015.1.8-51.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-127.el7ost.noarch
instack-undercloud-2.1.2-39.el7ost.noarch


Steps to reproduce:
1. Depoy 7.3GA with this command:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --swift-storage-scale 0 --block-storage-scale 0 --neutron-disable-tunneling --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:118:143 --neutron-bridge-mappings datacentre:br-ex  --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans-v6.yaml -e network-environment.yaml --ceph-storage-scale 

2. Attempt to update to 7.3Async with this command:
openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans-v6.yaml -e network-environment.yaml


Result:
the update fails with:
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+                                                                                                                                                                                            
| resource_name                                 | physical_resource_id                          | resource_type                                     | resource_status | updated_time         | parent_resource                               |                                                                                                                                                                                            
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+                                                                                                                                                                                            
| Networks                                      | 6764d73d-b305-42e3-8169-3c968f00e29e          | OS::TripleO::Network                              | UPDATE_FAILED   | 2016-07-07T20:14:11Z |                                               |                                                                                                                                                                                            
| TenantNetwork                                 | b98c7955-7232-4f08-9b17-1cfb8b6c3ddf          | OS::TripleO::Network::Tenant                      | UPDATE_FAILED   | 2016-07-07T20:14:18Z | Networks                                      |                                                                                                                                                                                            
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+   

 resource_status_reason | resources.Networks: resources.TenantNetwork: Conflict: resources.TenantSubnet: Unable to complete operation on subnet 6971cc5c-8a13-4fb9-8043-67586b708d74. One or more ports have an IP allocation from this subnet.

As seen from the below output this is the IPV6 tenant_subnet:

[stack@instack ~]$ neutron subnet-list
+--------------------------------------+---------------------+--------------------------+----------------------------------------------------------------------------------------+
| id                                   | name                | cidr                     | allocation_pools                                                                       |
+--------------------------------------+---------------------+--------------------------+----------------------------------------------------------------------------------------+
| 265b09d2-2c10-4dd1-a482-21f320b8cf8f | storage_subnet      | fd00:fd00:fd00:3000::/64 | {"start": "fd00:fd00:fd00:3000::10", "end": "fd00:fd00:fd00:3000:ffff:ffff:ffff:fffe"} |
| 27a58536-311e-4b53-8f42-c6d58c91b7d0 | tenant_subnet       | 172.16.0.0/24            | {"start": "172.16.0.4", "end": "172.16.0.250"}                                         |
| 6971cc5c-8a13-4fb9-8043-67586b708d74 | tenant_subnet       | fd00:fd00:fd00:5000::/64 | {"start": "fd00:fd00:fd00:5000::10", "end": "fd00:fd00:fd00:5000:ffff:ffff:ffff:fffe"} |
| 7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3 |                     | 192.0.2.0/24             | {"start": "192.0.2.5", "end": "192.0.2.24"}                                            |
| e6284d0c-cee5-4eab-8e60-fd64a2990c69 | external_subnet     | 2001:db8:fd00:1000::/64  | {"start": "2001:db8:fd00:1000::10", "end": "2001:db8:fd00:1000:ffff:ffff:ffff:fffe"}   |
| f44dc1ab-e482-4eda-b2e4-649c6f57dcfc | storage_mgmt_subnet | fd00:fd00:fd00:4000::/64 | {"start": "fd00:fd00:fd00:4000::10", "end": "fd00:fd00:fd00:4000:ffff:ffff:ffff:fffe"} |
| f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0 | internal_api_subnet | fd00:fd00:fd00:2000::/64 | {"start": "fd00:fd00:fd00:2000::10", "end": "fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe"} |
+--------------------------------------+---------------------+--------------------------+----------------------------------------------------------------------------------------+



[stack@instack ~]$ neutron port-list
+--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------------------------+
| id                                   | name                          | mac_address       | fixed_ips                                                                                      |
+--------------------------------------+-------------------------------+-------------------+------------------------------------------------------------------------------------------------+
| 0b82c7ad-0dfc-4b6a-9488-1cde338afbe8 |                               | fa:16:3e:39:29:16 | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::12"} |
| 11f7b6ca-72a6-4972-acbf-21e866cb255c | public_virtual_ip             | fa:16:3e:f5:a8:b3 | {"subnet_id": "e6284d0c-cee5-4eab-8e60-fd64a2990c69", "ip_address": "2001:db8:fd00:1000::10"}  |
| 191df6c4-3377-4cd1-9d6c-607d001c416a |                               | fa:16:3e:e7:26:9b | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.5"}               |
| 1a5782fd-27f3-41e6-b29a-d00924e83fb9 |                               | fa:16:3e:63:23:48 | {"subnet_id": "e6284d0c-cee5-4eab-8e60-fd64a2990c69", "ip_address": "2001:db8:fd00:1000::12"}  |
| 26d6695c-ce88-4261-8ae7-6ccd4bfce4db | storage_virtual_ip            | fa:16:3e:b5:f8:5b | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::10"} |
| 28be991a-0415-4371-b75c-672a9516ed14 |                               | fa:16:3e:f2:6b:45 | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::14"} |
| 29b6601a-6718-406a-9107-e7f2ac8f8aa8 |                               | fa:16:3e:ed:ca:05 | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::12"} |
| 305d92b2-e2be-4393-a654-8a96aa227eec |                               | 00:26:ab:42:9c:1f | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.10"}              |
| 30b00f0a-2ad1-4c49-a982-e8970e737939 |                               | 00:b6:cf:1d:34:38 | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.11"}              |
| 34cefcbb-d61b-4579-ab59-649e652f667e |                               | fa:16:3e:6b:36:28 | {"subnet_id": "f44dc1ab-e482-4eda-b2e4-649c6f57dcfc", "ip_address": "fd00:fd00:fd00:4000::14"} |
| 3cc355f7-7d1a-4127-b94b-9f50fb09294a |                               | 00:60:49:69:e0:0b | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.8"}               |
| 3ce6d663-a3dc-4525-b256-598ad75c4aa2 |                               | fa:16:3e:93:3a:7c | {"subnet_id": "6971cc5c-8a13-4fb9-8043-67586b708d74", "ip_address": "fd00:fd00:fd00:5000::11"} |
| 577fda6b-c23a-4be9-90ea-bc634ff10f79 |                               | fa:16:3e:aa:ed:59 | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::13"} |
| 57af86ae-8a93-474d-b6bd-e983295d4076 |                               | fa:16:3e:b7:4d:7d | {"subnet_id": "e6284d0c-cee5-4eab-8e60-fd64a2990c69", "ip_address": "2001:db8:fd00:1000::13"}  |
| 5ba6eaf0-a369-4ca5-8d5e-b277ac233464 |                               | fa:16:3e:11:98:82 | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::11"} |
| 5ce7d5ee-92d5-4788-b9c5-0d7f5a2a9bca |                               | fa:16:3e:b6:3a:ba | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::15"} |
| 6206e842-eed1-4bee-adb2-703e4876a889 | storage_management_virtual_ip | fa:16:3e:2b:42:78 | {"subnet_id": "f44dc1ab-e482-4eda-b2e4-649c6f57dcfc", "ip_address": "fd00:fd00:fd00:4000::10"} |
| 8e761d32-6004-4014-9973-b9ed1bb19a53 | control_virtual_ip            | fa:16:3e:5f:a5:70 | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.6"}               |
| 905d3fa0-5b98-4b20-95fe-e82f733633a4 |                               | fa:16:3e:8b:1a:a0 | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::15"} |
| 9739aebf-c303-4d8f-bbe4-1201d334afe4 | redis_virtual_ip              | fa:16:3e:18:5d:03 | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::11"} |
| a5a0ded9-54ad-4d23-9eba-414b33fc4c8b |                               | fa:16:3e:c8:29:bc | {"subnet_id": "f44dc1ab-e482-4eda-b2e4-649c6f57dcfc", "ip_address": "fd00:fd00:fd00:4000::11"} |
| abee2fda-53e7-41f0-ba18-195e0b2e9a51 |                               | fa:16:3e:c3:a2:cb | {"subnet_id": "f44dc1ab-e482-4eda-b2e4-649c6f57dcfc", "ip_address": "fd00:fd00:fd00:4000::12"} |
| ad33a371-d097-4243-ba97-e2986d5e02c7 | internal_api_virtual_ip       | fa:16:3e:dd:99:42 | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::10"} |
| adc5bf78-b315-4da0-a7d0-847d6d1b39e9 |                               | 00:d7:f4:4d:3b:4e | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.9"}               |
| ae1ed021-4d89-4209-8031-cdaf2681c0c6 |                               | fa:16:3e:08:7c:a8 | {"subnet_id": "6971cc5c-8a13-4fb9-8043-67586b708d74", "ip_address": "fd00:fd00:fd00:5000::13"} |
| b0f9c27e-a510-4fa2-8133-778dc974edd5 |                               | fa:16:3e:65:0e:9f | {"subnet_id": "f44dc1ab-e482-4eda-b2e4-649c6f57dcfc", "ip_address": "fd00:fd00:fd00:4000::13"} |
| b5a66e48-83c1-4ab4-be87-7f8693475396 |                               | fa:16:3e:09:26:54 | {"subnet_id": "6971cc5c-8a13-4fb9-8043-67586b708d74", "ip_address": "fd00:fd00:fd00:5000::10"} |
| b95005ef-ca85-44a6-b6d0-dc3ba62e6d3e |                               | 00:2d:fc:12:85:3a | {"subnet_id": "7ec59561-11ec-4fd1-a3a8-9c8424a2a7d3", "ip_address": "192.0.2.7"}               |
| cb6af24e-2e29-4695-a428-a77d606bb3cf |                               | fa:16:3e:a8:c3:a9 | {"subnet_id": "f87d64e5-7bd1-4e21-b734-b50ac6dcb4d0", "ip_address": "fd00:fd00:fd00:2000::13"} |
| cd71327c-555e-4f33-a686-9cf51ac85f4c |                               | fa:16:3e:71:a4:f6 | {"subnet_id": "e6284d0c-cee5-4eab-8e60-fd64a2990c69", "ip_address": "2001:db8:fd00:1000::11"}  |
| e2c81cff-b038-4bd1-bd36-7503639940db |                               | fa:16:3e:60:94:1e | {"subnet_id": "6971cc5c-8a13-4fb9-8043-67586b708d74", "ip_address": "fd00:fd00:fd00:5000::12"} |
| fdd4a047-0eab-4050-b007-83fc173e7ff3 |                               | fa:16:3e:c5:c4:5b | {"subnet_id": "265b09d2-2c10-4dd1-a482-21f320b8cf8f", "ip_address": "fd00:fd00:fd00:3000::14"} |




Expected result:
Successful update.

Comment 2 Alexander Chuzhoy 2016-07-07 20:46:58 UTC
Created attachment 1177453 [details]
heat logs

Comment 4 Dan Sneddon 2016-07-07 21:12:29 UTC
IPv6 for the Tenant network is not compatible with VXLAN/GRE tunneling. The Tenant network should have only an IPv4 subnet, but I see an IPv6 subnet as well in the output above.

Perhaps the resource registry might have been pointing at tenant_v6.yaml instead of tenant.yaml for the Tenant network before the upgrade?

A workaround may be to delete the Tenant IPv6 subnet (if it exists). Any ports on that subnet would have to be deleted first. I think that may allow the upgrade to replace the IPv6 subnet with an IPv4-only subnet.

Comment 5 Alexander Chuzhoy 2016-07-07 21:19:49 UTC
I deleted the IPv6 subnet and resumed the update:
FAILED                                                                                                                                                                                 
update finished with status FAILED                                                                                                                                                     
[stack@instack ~]$                                                                                                                                                                     
[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+                                                                                                                                                                                            
| resource_name                                 | physical_resource_id                          | resource_type                                     | resource_status | updated_time         | parent_resource                               |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+
| Controller                                    | 035719fc-60b4-4ec4-b078-8976f5437897          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-07T21:14:03Z |                                               |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+
[stack@instack ~]$ heat resource-show overcloud Controller
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                            |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                |
|                        |   "attributes": null,                                                                                                                            |
|                        |   "refs": null                                                                                                                                   |
|                        | }                                                                                                                                                |
| description            |                                                                                                                                                  |
| links                  | http://192.0.2.1:8004/v1/c7ec9c0471564cb58e617c88762aca69/stacks/overcloud/cd3c50a5-cdcf-41de-9956-48b1bb00c975/resources/Controller (self)      |
|                        | http://192.0.2.1:8004/v1/c7ec9c0471564cb58e617c88762aca69/stacks/overcloud/cd3c50a5-cdcf-41de-9956-48b1bb00c975 (stack)                          |
|                        | http://192.0.2.1:8004/v1/c7ec9c0471564cb58e617c88762aca69/stacks/overcloud-Controller-3zarhc234x6x/035719fc-60b4-4ec4-b078-8976f5437897 (nested) |
| logical_resource_id    | Controller                                                                                                                                       |
| physical_resource_id   | 035719fc-60b4-4ec4-b078-8976f5437897                                                                                                             |
| required_by            | ControllerAllNodesValidationDeployment                                                                                                           |
|                        | ControllerClusterDeployment                                                                                                                      |
|                        | ControllerCephDeployment                                                                                                                         |
|                        | AllNodesExtraConfig                                                                                                                              |
|                        | ControllerNodesPostDeployment                                                                                                                    |
|                        | SwiftDevicesAndProxyConfig                                                                                                                       |
|                        | ControllerBootstrapNodeDeployment                                                                                                                |
|                        | ControllerAllNodesDeployment                                                                                                                     |
|                        | ControllerBootstrapNodeConfig                                                                                                                    |
|                        | ControllerIpListMap                                                                                                                              |
|                        | VipDeployment                                                                                                                                    |
|                        | CephClusterConfig                                                                                                                                |
|                        | AllNodesValidationConfig                                                                                                                         |
|                        | ControllerSwiftDeployment                                                                                                                        |
|                        | ControllerClusterConfig                                                                                                                          |
|                        | allNodesConfig                                                                                                                                   |
| resource_name          | Controller                                                                                                                                       |
| resource_status        | UPDATE_FAILED                                                                                                                                    |
| resource_status_reason | resources.Controller: Property error: resources[2].properties: Property MysqlClustercheckPassword not assigned                                   |
| resource_type          | OS::Heat::ResourceGroup                                                                                                                          |
| updated_time           | 2016-07-07T21:14:03Z                                                                                                                             |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+




The above is reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1344794

Comment 6 Marios Andreou 2016-07-13 09:21:08 UTC
(In reply to Dan Sneddon from comment #4)
> IPv6 for the Tenant network is not compatible with VXLAN/GRE tunneling. The
> Tenant network should have only an IPv4 subnet, but I see an IPv6 subnet as
> well in the output above.
> 
> Perhaps the resource registry might have been pointing at tenant_v6.yaml
> instead of tenant.yaml for the Tenant network before the upgrade?
> 
> A workaround may be to delete the Tenant IPv6 subnet (if it exists). Any
> ports on that subnet would have to be deleted first. I think that may allow
> the upgrade to replace the IPv6 subnet with an IPv4-only subnet.

o/ Dan, Sasha:

I *think* this may be related/explain what Dan is seeing: I noticed the original deploy command is specifying vlan for the ovs agent: --neutron-disable-tunneling --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:118:143 --neutron-bridge-mappings datacentre:br-ex

But isn't specifying these for the update. For example the NeutronNetworkType defaults to 'vxlan' in the heat templates:

./puppet/services/neutron-plugin-ml2.yaml:50:  NeutronNetworkType:
./puppet/services/neutron-plugin-ml2.yaml-51-    default: 'vxlan'

So vxlan was used for the update? It is probably safer to override those options via parameter_defaults in an environment file Sasha (I *think* they are deprecated on the clientside already if I'm not mistaken),

thanks.

Comment 10 Sofer Athlan-Guyot 2017-01-25 00:02:11 UTC
Hi Sasha,

I'm going to close this one.  The explanation from Marios' comment makes sense.  Adding the missing parameter to the update command should do it.  If it's still an issue don't hesitate to re-open it.

Comment 11 Amit Ugol 2018-05-02 10:49:56 UTC
closed, no need for needinfo.


Note You need to log in before you can comment on or make changes to this bug.