Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1478818 - OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conf...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity urgent
: beta
: 12.0 (Pike)
Assigned To: Sofer Athlan-Guyot
Marius Cornea
: Triaged
Depends On:
Blocks: 1399762 1501515
  Show dependency tree / graph
 
Reported: 2017-08-07 03:56 EDT by Marius Cornea
Modified: 2018-02-05 14:10 EST (History)
14 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1501515 (view as bug list)
Environment:
Last Closed: 2017-12-13 16:49:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1709105 None None None 2017-08-07 11:05 EDT
OpenStack gerrit 491510 None None None 2017-08-07 11:06 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Marius Cornea 2017-08-07 03:56:23 EDT 
Description of problem:
OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170803094851.el7ost.noarch
openstack-heat-api-9.0.0-0.20170803122112.c80011c.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170803122112.c80011c.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Upgrade undercloud to OSP12
3. Update images
4. Run major-upgrade-composable-steps-docker.yaml:

#!/bin/bash

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \

Note: the network-environment.yaml file hasn't been changed

Actual results:

017-08-05 21:17:07Z [overcloud-Networks-5ofo6ltrxp7n.InternalApiNetwork]: CREATE_FAILED  Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n.InternalApiSubnet]: DELETE_FAILED  Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109']
2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066-4
2017-08-05 21:18:17Z [overcloud-Networks-5ofo6ltrxp7n.InternalNetwork]: DELETE_FAILED  Conflict: resources.InternalNetwork.resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-cd1a8fe5-e066
2017-08-05 21:18:18Z [overcloud-Networks-5ofo6ltrxp7n]: UPDATE_FAILED  Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:18Z [Networks]: UPDATE_FAILED  resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
2017-08-05 21:18:19Z [overcloud]: UPDATE_FAILED  resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']

 Stack overcloud UPDATE_FAILED 

overcloud.Networks.InternalApiNetwork.InternalApiNetwork:
  resource_type: OS::Neutron::Net
  physical_resource_id: 
Heat Stack update failed.
Heat Stack update failed.
  status: CREATE_FAILED
  status_reason: |
    Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
    Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75']
overcloud.Networks.InternalNetwork.InternalApiSubnet:
  resource_type: OS::Neutron::Subnet
  physical_resource_id: b88764b8-9a29-4bd1-80d8-c0d10a16f9e7
  status: DELETE_FAILED
  status_reason: |
    Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet.
    Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109']


Expected results:
The upgrade process doesn't try to recreate networks as there haven't been any changes made to the networking environment.

Additional info:

Attaching the undercloud sosreport.
Comment 1 Marius Cornea 2017-08-07 08:34:45 EDT 
Digging a bit into the issue I notice that the InternalApiNetwork nested stack is CREATE_FAILED while the InternalNetwork nested stack is DELETE_FAILED:


(undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -i network | grep -i internal
| 3d38bf53-6f78-4c5e-a5cc-304771eb3be4 | overcloud-Networks-5ofo6ltrxp7n-InternalApiNetwork-32zdrw34ckj2                                                                              | 0b9f9afc1f5a450da7c50b911c3e7f7f | CREATE_FAILED   | 2017-08-07T11:50:21Z | None                 | 2d36d135-16c4-4e99-ab65-c1cac483725d |
| 697c7941-46fc-4172-ba90-a059ab72e6c4 | overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n                                                                                 | 0b9f9afc1f5a450da7c50b911c3e7f7f | DELETE_FAILED   | 2017-08-05T20:08:25Z | None                 | 2d36d135-16c4-4e99-ab65-c1cac483725d |

The following change seems to be related:
https://review.openstack.org/#/c/475612/22/network/networks.j2.yaml
Comment 2 Yurii Prokulevych 2017-08-31 04:16:01 EDT 
I got this error while trying to setup stack's output.

openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml --setup-heat-outputs

Env was updated from 2017-08-18.3 to 2017-08-28.10

Package: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch

Change from upstream review already present in tht.
Comment 4 Yurii Prokulevych 2017-08-31 04:20:27 EDT 
Follow up attempt to delete overcloud failed:

openstack stack delete --wait overcloud
Are you sure you want to delete this stack(s) [y/N]? y
2017-08-31 08:19:08Z [overcloud]: DELETE_IN_PROGRESS  Stack DELETE started
2017-08-31 08:19:24Z [overcloud.EndpointMapData]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.DeployedServerEnvironment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:24Z [overcloud.ServerIdMap]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:25Z [overcloud.AllNodesDeploySteps]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:25Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.DeployedServerEnvironment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.EndpointMapData]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ServerIdMap]: DELETE_COMPLETE  state changed
2017-08-31 08:19:25Z [overcloud.ServerOsCollectConfigData]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:26Z [overcloud.ServerOsCollectConfigData]: DELETE_COMPLETE  state changed
2017-08-31 08:19:27Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:29Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_COMPLETE  state changed
2017-08-31 08:19:29Z [overcloud.SshKnownHostsConfig]: DELETE_IN_PROGRESS  state changed
2017-08-31 08:19:31Z [overcloud.SshKnownHostsConfig]: DELETE_COMPLETE  state changed
2017-08-31 08:19:59Z [overcloud.AllNodesDeploySteps]: DELETE_FAILED  NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404)
2017-08-31 08:19:59Z [overcloud]: DELETE_FAILED  Resource DELETE failed: NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404)

 Stack overcloud DELETE_FAILED 

Unable to delete 1 of the 1 stacks.
Comment 12 errata-xmlrpc 2017-12-13 16:49:38 EST 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.