Bug 1478818
| Summary: | OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Sofer Athlan-Guyot <sathlang> | |
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | high | |||
| Version: | 12.0 (Pike) | CC: | bfournie, dbecker, dsneddon, lbezdick, mbracho, mbultel, mburns, morazi, rhel-osp-director-maint, sathlang, sclewis, tvignaud, yprokule | |
| Target Milestone: | beta | Keywords: | Triaged | |
| Target Release: | 12.0 (Pike) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1501515 (view as bug list) | Environment: | ||
| Last Closed: | 2017-12-13 21:49:38 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1399762, 1501515 | |||
Digging a bit into the issue I notice that the InternalApiNetwork nested stack is CREATE_FAILED while the InternalNetwork nested stack is DELETE_FAILED: (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -i network | grep -i internal | 3d38bf53-6f78-4c5e-a5cc-304771eb3be4 | overcloud-Networks-5ofo6ltrxp7n-InternalApiNetwork-32zdrw34ckj2 | 0b9f9afc1f5a450da7c50b911c3e7f7f | CREATE_FAILED | 2017-08-07T11:50:21Z | None | 2d36d135-16c4-4e99-ab65-c1cac483725d | | 697c7941-46fc-4172-ba90-a059ab72e6c4 | overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n | 0b9f9afc1f5a450da7c50b911c3e7f7f | DELETE_FAILED | 2017-08-05T20:08:25Z | None | 2d36d135-16c4-4e99-ab65-c1cac483725d | The following change seems to be related: https://review.openstack.org/#/c/475612/22/network/networks.j2.yaml I got this error while trying to setup stack's output. openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml --setup-heat-outputs Env was updated from 2017-08-18.3 to 2017-08-28.10 Package: openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch Change from upstream review already present in tht. Follow up attempt to delete overcloud failed: openstack stack delete --wait overcloud Are you sure you want to delete this stack(s) [y/N]? y 2017-08-31 08:19:08Z [overcloud]: DELETE_IN_PROGRESS Stack DELETE started 2017-08-31 08:19:24Z [overcloud.EndpointMapData]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.DeployedServerEnvironment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:24Z [overcloud.ServerIdMap]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:25Z [overcloud.AllNodesDeploySteps]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:25Z [overcloud.BlockStorageSshKnownHostsDeployment]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.ObjectStorageSshKnownHostsDeployment]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.CephStorageSshKnownHostsDeployment]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.DeployedServerEnvironment]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.EndpointMapData]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.ServerIdMap]: DELETE_COMPLETE state changed 2017-08-31 08:19:25Z [overcloud.ServerOsCollectConfigData]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:26Z [overcloud.ServerOsCollectConfigData]: DELETE_COMPLETE state changed 2017-08-31 08:19:27Z [overcloud.ComputeSshKnownHostsDeployment]: DELETE_COMPLETE state changed 2017-08-31 08:19:29Z [overcloud.ControllerSshKnownHostsDeployment]: DELETE_COMPLETE state changed 2017-08-31 08:19:29Z [overcloud.SshKnownHostsConfig]: DELETE_IN_PROGRESS state changed 2017-08-31 08:19:31Z [overcloud.SshKnownHostsConfig]: DELETE_COMPLETE state changed 2017-08-31 08:19:59Z [overcloud.AllNodesDeploySteps]: DELETE_FAILED NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404) 2017-08-31 08:19:59Z [overcloud]: DELETE_FAILED Resource DELETE failed: NotFound: resources.AllNodesDeploySteps.resources.WorkflowTasks_Step2: Not Found (HTTP 404) Stack overcloud DELETE_FAILED Unable to delete 1 of the 1 stacks. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
Description of problem: OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-7.0.0-0.20170803094851.el7ost.noarch openstack-heat-api-9.0.0-0.20170803122112.c80011c.el7ost.noarch openstack-heat-engine-9.0.0-0.20170803122112.c80011c.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 2. Upgrade undercloud to OSP12 3. Update images 4. Run major-upgrade-composable-steps-docker.yaml: #!/bin/bash timeout 100m openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \ -e /home/stack/docker-osp12.yaml \ Note: the network-environment.yaml file hasn't been changed Actual results: 017-08-05 21:17:07Z [overcloud-Networks-5ofo6ltrxp7n.InternalApiNetwork]: CREATE_FAILED Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75'] 2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n.InternalApiSubnet]: DELETE_FAILED Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet. Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109'] 2017-08-05 21:18:16Z [overcloud-Networks-5ofo6ltrxp7n-InternalNetwork-4xmk6d4rys2n]: DELETE_FAILED Resource DELETE failed: Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet. Neutron server returns request_ids: ['req-cd1a8fe5-e066-4 2017-08-05 21:18:17Z [overcloud-Networks-5ofo6ltrxp7n.InternalNetwork]: DELETE_FAILED Conflict: resources.InternalNetwork.resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet. Neutron server returns request_ids: ['req-cd1a8fe5-e066 2017-08-05 21:18:18Z [overcloud-Networks-5ofo6ltrxp7n]: UPDATE_FAILED Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75'] 2017-08-05 21:18:18Z [Networks]: UPDATE_FAILED resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75'] 2017-08-05 21:18:19Z [overcloud]: UPDATE_FAILED resources.Networks: Conflict: resources.InternalApiNetwork.resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75'] Stack overcloud UPDATE_FAILED overcloud.Networks.InternalApiNetwork.InternalApiNetwork: resource_type: OS::Neutron::Net physical_resource_id: Heat Stack update failed. Heat Stack update failed. status: CREATE_FAILED status_reason: | Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use. Neutron server returns request_ids: ['req-9d29f1fa-a7e3-4495-b972-a3baf0605b75'] overcloud.Networks.InternalNetwork.InternalApiSubnet: resource_type: OS::Neutron::Subnet physical_resource_id: b88764b8-9a29-4bd1-80d8-c0d10a16f9e7 status: DELETE_FAILED status_reason: | Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet b88764b8-9a29-4bd1-80d8-c0d10a16f9e7: One or more ports have an IP allocation from this subnet. Neutron server returns request_ids: ['req-cd1a8fe5-e066-4180-a4be-d41ed1772109'] Expected results: The upgrade process doesn't try to recreate networks as there haven't been any changes made to the networking environment. Additional info: Attaching the undercloud sosreport.