Bug 1478818 - OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conf...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity urgent
: beta
: 12.0 (Pike)
Assigned To: Sofer Athlan-Guyot
Marius Cornea
: Triaged
Depends On:
Blocks: 1399762 1501515
  Show dependency treegraph
 
Reported: 2017-08-07 03:56 EDT by Marius Cornea
Modified: 2017-12-13 16:49 EST (History)
14 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1501515 (view as bug list)
Environment:
Last Closed: 2017-12-13 16:49:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1709105 None None None 2017-08-07 11:05 EDT
OpenStack gerrit 491510 None None None 2017-08-07 11:06 EDT

  None (edit)
Description Marius Cornea 2017-08-07 03:56:23 EDT
Description of problem:
OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170803094851.el7ost.noarch
openstack-heat-api-9.0.0-0.20170803122112.c80011c.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170803122112.c80011c.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Upgrade undercloud to OSP12
3. Update images
4. Run major-upgrade-composable-steps-docker.yaml:

#!/bin/bash

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \

Note: the network-environment.yaml file hasn't been changed

Actual results:

017-08-05 21:17:07Z [overcloud-Networks-5ofo6ltrxp7n.InternalApiNetwork]: CREATE_FAILED  Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n.InternalApiSubnet]: DELETE_FAILED  Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109']
2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066-4
2017-08-05 21:18:17Z [overcloud-Networks-5ofo6ltrxp7n.InternalNetwork]: DELETE_FAILED  Conflict: resources.InternalNetwork.resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066
2017-08-05 21:18:18Z [overcloud-Networks-5ofo6ltrxp7n]: UPDATE_FAILED  Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:18Z [Networks]: UPDATE_FAILED  resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:19Z [overcloud]: UPDATE_FAILED  resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']

 Stack overcloud UPDATE_FAILED 

overcloud.Networks.InternalApiNetwork.InternalApiNetwork:
  resource_type: OS::Neutron::Net
  physical_resource_id: 
Heat Stack update failed.
Heat Stack update failed.
  status: CREATE_FAILED
  status_reason: |
    Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
    Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
overcloud.Networks.InternalNetwork.InternalApiSubnet:
  resource_type: OS::Neutron::Subnet
  physical_resource_id: b88764b8-9a29-4bd1-80d8-c0d10a16f9e7
  status: DELETE_FAILED
  status_reason: |
    Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
    Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109']


Expected results:
The upgrade process doesn't try to recreate networks as there haven't been any changes made to the networking environment.

Additional info:

Attaching the undercloud sosreport.
Comment 1 Marius Cornea 2017-08-07 08:34:45 EDT
Digging a bit into the issue I notice that the InternalApiNetwork nested stack is CREATE_FAILED while the InternalNetwork nested stack is DELETE_FAILED:


(undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -i network | grep -i internal
| 3d38bf53-6f78-4c5e-a5cc-304771eb3be4 | overcloud-Networks-5ofo6ltrxp7n-InternalApiNetwork-32zdrw34ckj2                                                                              | 0b9f9afc1f5a450da7c50b911c3e7f7f | CREATE_FAILED   | 2017-08-07T11:50:21Z | None                 | 2d36d135-16c4-4e99-ab65-c1cac483725d |
| 697c7941-46fc-4172-ba90-a059ab72e6c4 | overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n                                                                                 | 0b9f9afc1f5a450da7c50b911c3e7f7f | DELETE_FAILED   | 2017-08-05T20:08:25Z | None                 | 2d36d135-16c4-4e99-ab65-c1cac483725d |

The following change seems to be related:
https://review.openstack.org/#/c/475612/22/network/networks.j2.yaml
Comment 2 Yurii Prokulevych 2017-08-31 04:16:01 EDT
I got this error while trying to setup stack's output.

openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml --setup-heat-outputs

Env was updated from 2017-08-18.3 to 2017-08-28.10

Package: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch

Change from upstream review already present in tht.
Comment 4 Yurii Prokulevych 2017-08-31 04:20:27 EDT
Follow up attempt to delete overcloud failed:

openstack stack delete --wait overcloud
Are you sure you want to delete this stack(s) [y/N]? y
2017-08-31 08:19:08Z [overcloud]: DELETE_IN_PROGRESS  Stack DELETE started
2017-08-31 08:19:24Z [overcloud.EndpointMapData]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.DeployedServerEnvironment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ServerIdMap]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:25Z [overcloud.AllNodesDeploySteps]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:25Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.DeployedServerEnvironment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.EndpointMapData]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ServerIdMap]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ServerOsCollectConfigData]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:26Z [overcloud.ServerOsCollectConfigData]: DELETE_COMPLETE  state changed
2017-08-31 08:19:27Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:29Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:29Z [overcloud.SshKnownHostsConfig]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:31Z [overcloud.SshKnownHostsConfig]: DELETE_COMPLETE  state changed
2017-08-31 08:19:59Z [overcloud.AllNodesDeploySteps]: DELETE_FAILED  NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404)
2017-08-31 08:19:59Z [overcloud]: DELETE_FAILED  Resource DELETE failed: NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404)

 Stack overcloud DELETE_FAILED 

Unable to delete 1 of the 1 stacks.
Comment 12 errata-xmlrpc 2017-12-13 16:49:38 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.