Bug 1589811

Created attachment 1450015 [details]
Console output

Created attachment 1450030 [details]
stack logs and generated yamls

Comment on attachment 1450030 [details]
stack logs and generated yamls

ignore - adding a new tar

Created attachment 1450034 [details]
stack logs input yamls and output yamls

Storage subnet issue: 

(undercloud) [stack@undercloud-0 ~]$ openstack subnet list 
+--------------------------------------+---------------------+--------------------------------------+-----------------+
| ID                                   | Name                | Network                              | Subnet          |
+--------------------------------------+---------------------+--------------------------------------+-----------------+
| a5402848-5098-4e82-9376-7d8525aa3521 | external_subnet     | 799544d7-fad3-4eaf-b166-263191606f0f | 10.0.0.0/24     |
| c46261e4-089d-4db2-953d-1973c83b2e5c | internal_api_subnet | cd871fee-7be8-4997-bb5c-49044b97f7b9 | 172.17.1.0/24   |
| d5b03ff4-c900-4dd8-8c05-69edd63d5865 | ctlplane-subnet     | b201278b-dc47-42bf-8a5c-e215695bbb1f | 192.168.24.0/24 |
| e16df453-0624-4e01-b30b-cf3fdaa04c45 | tenant_subnet       | f0cf8dc9-8900-4645-8d19-bdae5eff37d6 | 172.17.2.0/24   |
| f707196e-afaa-41c4-bf93-54314a563d7c | storage_subnet      | 51a29287-3c74-48bf-9d37-64e44ab515b5 | 172.17.3.0/24   |
| fed6d862-bfe4-4455-9310-4c6d92e00fd3 | storage_mgmt_subnet | 34e0cd1d-e381-4052-a280-d5ba407dbffa | 172.17.4.0/24   |
+--------------------------------------+---------------------+--------------------------------------+-----------------+

2018-06-11 10:31:02Z [overcloud-Networks-u4ywx7vadzhx-StorageNetwork-iby3dmmkyxv5.StorageSubnet]: DELETE_FAILED  Conflict: resources.StorageSubnet: Unable to complete operation on subnet f707196e-afaa-41c4-bf93-54314a563d7c: One or more ports have an IP allocation from this subnet.

Trying to manually delete the storage_mgmt_subnet fails with:

GET call to network for http://192.168.24.1:9696/v2.0/subnets/fed6d862-bfe4-4455-9310-4c6d92e00fd3 used request id req-0a10ab85-2f1a-4493-bf4e-993d5396da47

Manager unknown ran task network.GET.subnets in 1.28573608398s

Manager unknown running task network.DELETE.subnets

REQ: curl -g -i -X DELETE http://192.168.24.1:9696/v2.0/subnets/fed6d862-bfe4-4455-9310-4c6d92e00fd3 -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "Accept: " -H "X-Auth-Token: {SHA1}8545d6678baff9751f20335cf55bb238c90bc6bf"
http://192.168.24.1:9696 "DELETE /v2.0/subnets/fed6d862-bfe4-4455-9310-4c6d92e00fd3 HTTP/1.1" 409 204
RESP: [409] Content-Type: application/json Content-Length: 204 X-Openstack-Request-Id: req-d8760e3f-aa30-4d6c-9384-d7a0c68f9485 Date: Tue, 12 Jun 2018 12:06:55 GMT Connection: keep-alive
RESP BODY: {"NeutronError": {"message": "Unable to complete operation on subnet fed6d862-bfe4-4455-9310-4c6d92e00fd3: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}}


Attaching delete subnet log.

Created attachment 1450496 [details]
subnet manual delete attempt failed

(In reply to Noam Manos from comment #0)
> 
> Steps to Reproduce:
> 
> (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update prepare \
>    --templates \
>    --container-registry-file ~/update-container-params.yaml \
>    -e /home/stack/virt/docker-images.yaml \
>    -e /home/stack/virt/config_lvm.yaml \
>    -e /home/stack/virt/network/network-environment.yaml \
>    -e
> /usr/share/openstack-tripleo-heat-templates/environments/services-docker/
> octavia.yaml \
>    -e /home/stack/virt/hostnames.yml \
>    -e /home/stack/virt/nodes_data.yaml
> 
> 

By visual inspection of the update prepare command it looks to me that you missed the following environment file:

-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \

Can you confirm that this was used during the initial deployment? Please note that all environment files used for initial deployment are require for the subsequent deploy/update/upgrade commands.

(In reply to Marius Cornea from comment #11)
> 
> By visual inspection of the update prepare command it looks to me that you
> missed the following environment file:
> 
> -e
> /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.
> yaml \
> 
> Can you confirm that this was used during the initial deployment? Please
> note that all environment files used for initial deployment are require for
> the subsequent deploy/update/upgrade commands.


Indeed.

Updating overcloud with all environment files as listed in ~/overcloud_deploy.sh (from existing deployment), has passed:

$ openstack overcloud update prepare \
  --templates \
  --container-registry-file ~/update-container-params.yaml \
  -e /home/stack/virt/config_lvm.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/virt/network/network-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
  -e /home/stack/virt/hostnames.yml \
  -e /home/stack/virt/nodes_data.yaml \
  -e /home/stack/virt/extra_templates.yaml \
  -e /home/stack/virt/docker-images.yaml


...

2018-06-17 09:42:49Z [overcloud-AllNodesDeploySteps-svgtgygyysoa]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-06-17 09:42:50Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2018-06-17 09:43:00Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: 9421ed88-1574-4d84-bd0d-b5af9a6602b4
Success
Update init on stack overcloud complete.