Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1443090

Summary:	Downscale compute nodes from 1 to 0 fails on neutron errors
Product:	Red Hat OpenStack	Reporter:	Mike Abrams <mabrams>
Component:	openstack-tripleo	Assignee:	James Slagle <jslagle>
Status:	CLOSED NOTABUG	QA Contact:	Arik Chernetsky <achernet>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	11.0 (Ocata)	CC:	amuller, aschultz, chrisw, cschwede, mabrams, mburns, nyechiel, owalsh, rhel-osp-director-maint, srevivo, svanders
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-01 19:43:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1437499, 1465731

Description Mike Abrams 2017-04-18 12:56:13 UTC

Description of problem:
overcloud deploy swift scale out fails on neutron errors

Version-Release number of selected component (if applicable):
openstack-neutron-common-10.0.0-18.el7ost.noarch
RHOS11

How reproducible:
always

Steps to Reproduce:
1. install rhel 7.3
2. install rhos11:  1 controller, 1 compute (i used jenkins)
3. openstack overcloud deploy --templates --compute-scale 0 --swift-storage-scale 0

Actual results:
-----------------------------------------------------------------
[stack@undercloud-0 ~]$ . ./stackrc 
[stack@undercloud-0 ~]$ nova list
/usr/lib/python2.7/site-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks               |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| 6946ffd8-df46-4e33-add6-b0f7d6358dd3 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.168.24.11 |
| 908c26be-af59-4eb2-a54e-9679d773e1bf | controller-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.13 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
[stack@undercloud-0 ~]$ openstack overcloud deploy --templates --compute-scale 0 --swift-storage-scale 0
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: e665450e-2867-4efc-862c-7c3a8d5f9790
Plan updated
Deploying templates in the directory /tmp/tripleoclient-wMzADb/tripleo-heat-templates
Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: 6fac548b-feb7-4bca-ac2d-e04ca554d126
2017-04-18 12:33:34Z [Networks]: UPDATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-Networks-fliltytuxeie]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-04-18 12:33:34Z [ServiceNetMap]: UPDATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-Networks-fliltytuxeie.TenantNetwork]: CREATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-ServiceNetMap-p6ze7eyfsnig]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-04-18 12:33:34Z [DefaultPasswords]: UPDATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-Networks-fliltytuxeie.StorageMgmtNetwork]: CREATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-Networks-fliltytuxeie.StorageNetwork]: CREATE_IN_PROGRESS  state changed
2017-04-18 12:33:34Z [overcloud-ServiceNetMap-p6ze7eyfsnig]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-04-18 12:33:34Z [overcloud-Networks-fliltytuxeie.InternalNetwork]: CREATE_IN_PROGRESS  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.ExternalNetwork]: CREATE_IN_PROGRESS  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.StorageMgmtNetwork]: CREATE_COMPLETE  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.TenantNetwork]: CREATE_COMPLETE  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.StorageNetwork]: CREATE_COMPLETE  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.ExternalNetwork]: CREATE_COMPLETE  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie.InternalNetwork]: CREATE_COMPLETE  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie-StorageMgmtNetwork-a6ikyqz3czqv]: DELETE_IN_PROGRESS  Stack DELETE started
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie-StorageMgmtNetwork-a6ikyqz3czqv.StorageMgmtSubnet]: DELETE_IN_PROGRESS  state changed
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie-StorageNetwork-mjtz4jweypnp]: DELETE_IN_PROGRESS  Stack DELETE started
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie-StorageNetwork-mjtz4jweypnp.StorageSubnet]: DELETE_IN_PROGRESS  state changed
2017-04-18 12:33:36Z [ServiceNetMap]: UPDATE_COMPLETE  state changed
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-ExternalNetwork-xip7ijof6cem]: DELETE_IN_PROGRESS  Stack DELETE started
2017-04-18 12:33:36Z [DefaultPasswords]: UPDATE_COMPLETE  state changed
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-TenantNetwork-ilctq7h6inez]: DELETE_IN_PROGRESS  Stack DELETE started
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-ExternalNetwork-xip7ijof6cem.ExternalSubnet]: DELETE_IN_PROGRESS  state changed
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-TenantNetwork-ilctq7h6inez.TenantSubnet]: DELETE_IN_PROGRESS  state changed
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-InternalNetwork-icsbyjtdndvt]: DELETE_IN_PROGRESS  Stack DELETE started
2017-04-18 12:33:36Z [overcloud-Networks-fliltytuxeie-InternalNetwork-icsbyjtdndvt.InternalApiSubnet]: DELETE_IN_PROGRESS  state changed
2017-04-18 12:34:45Z [overcloud-Networks-fliltytuxeie-StorageMgmtNetwork-a6ikyqz3czqv.StorageMgmtSubnet]: DELETE_FAILED  Conflict: resources.StorageMgmtSubnet: Unable to complete operation on subnet eab1ff24-674f-4525-b500-0004a4be9e5f: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-fc8eb755-ce51-43c9-a04a-86b7c6b9b3f7']
2017-04-18 12:34:45Z [overcloud-Networks-fliltytuxeie-StorageMgmtNetwork-a6ikyqz3czqv]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.StorageMgmtSubnet: Unable to complete operation on subnet eab1ff24-674f-4525-b500-0004a4be9e5f: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-fc8eb755-ce51-4
2017-04-18 12:34:47Z [overcloud-Networks-fliltytuxeie-TenantNetwork-ilctq7h6inez.TenantSubnet]: DELETE_FAILED  Conflict: resources.TenantSubnet: Unable to complete operation on subnet c29e780e-9f28-4026-be30-1e568ec4dfa9: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-2fafa817-744d-4dd5-bbc3-db72f20d4f64']
2017-04-18 12:34:47Z [overcloud-Networks-fliltytuxeie-TenantNetwork-ilctq7h6inez]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.TenantSubnet: Unable to complete operation on subnet c29e780e-9f28-4026-be30-1e568ec4dfa9: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-2fafa817-744d-4dd5-b
2017-04-18 12:34:48Z [overcloud-Networks-fliltytuxeie-InternalNetwork-icsbyjtdndvt.InternalApiSubnet]: DELETE_FAILED  Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet 4bdbbe3d-e9ae-41cd-8c10-b5f92180d3ea: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-1e2b5d99-7d89-4fa6-9952-8e04275b8750']
2017-04-18 12:34:48Z [overcloud-Networks-fliltytuxeie-InternalNetwork-icsbyjtdndvt]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.InternalApiSubnet: Unable to complete operation on subnet 4bdbbe3d-e9ae-41cd-8c10-b5f92180d3ea: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-1e2b5d99-7d89-4
2017-04-18 12:34:49Z [overcloud-Networks-fliltytuxeie-StorageNetwork-mjtz4jweypnp.StorageSubnet]: DELETE_FAILED  Conflict: resources.StorageSubnet: Unable to complete operation on subnet 74d572e5-14c3-4f7b-96d3-cb411d1d55f3: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-c07de286-0239-43a3-9c81-ca8b260a82d8']
2017-04-18 12:34:49Z [overcloud-Networks-fliltytuxeie-StorageNetwork-mjtz4jweypnp]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.StorageSubnet: Unable to complete operation on subnet 74d572e5-14c3-4f7b-96d3-cb411d1d55f3: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-c07de286-0239-43a3-
2017-04-18 12:34:50Z [overcloud-Networks-fliltytuxeie-ExternalNetwork-xip7ijof6cem.ExternalSubnet]: DELETE_FAILED  Conflict: resources.ExternalSubnet: Unable to complete operation on subnet eb3d05f6-be75-4ed2-b17a-abd3fc42a1ff: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-f205e502-761a-4f7e-9c01-c1f480c099f8']
2017-04-18 12:34:50Z [overcloud-Networks-fliltytuxeie-ExternalNetwork-xip7ijof6cem]: DELETE_FAILED  Resource DELETE failed: Conflict: resources.ExternalSubnet: Unable to complete operation on subnet eb3d05f6-be75-4ed2-b17a-abd3fc42a1ff: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-f205e502-761a-4f7e
2017-04-18 12:34:50Z [overcloud-Networks-fliltytuxeie]: UPDATE_FAILED  Conflict: resources.StorageMgmtNetwork.resources.StorageMgmtSubnet: Unable to complete operation on subnet eab1ff24-674f-4525-b500-0004a4be9e5f: One or more ports have an IP allocation from this subnet.
Neutron server returns request_ids: ['req-fc8eb755-c
2017-04-18 12:34:51Z [Networks]: UPDATE_FAILED  resources.Networks: Conflict: resources.StorageMgmtNetwork.resources.StorageMgmtSubnet: Unable to complete operation on subnet eab1ff24-674f-4525-b500-0004a4be9e5f: One or more ports have an IP allocation from this subnet.
Neutron server returns request_i
2017-04-18 12:34:51Z [overcloud]: UPDATE_FAILED  resources.Networks: Conflict: resources.StorageMgmtNetwork.resources.StorageMgmtSubnet: Unable to complete operation on subnet eab1ff24-674f-4525-b500-0004a4be9e5f: One or more ports have an IP allocation from this subnet.
Neutron server returns request_i

 Stack overcloud UPDATE_FAILED 

Heat Stack update failed.
[stack@undercloud-0 ~]$ 
-----------------------------------------------------------------

Expected results:
command should succeed and a swift VM should be created

Additional info:

Comment 2 Sven Anderson 2017-09-19 14:39:39 UTC

Why exactly would you expect a swift VM running, after you scale it down to 0 with "--swift-storage-scale 0"? What is the reason for scaling the compute nodes to 0 as well? What is the use case for that?

Comment 3 Ollie Walsh 2017-09-20 09:58:16 UTC

NB: that is not how to scale down: https://docs.openstack.org/tripleo-docs/latest/install/post_deployment/delete_nodes.html

Comment 4 Ollie Walsh 2017-12-01 13:46:00 UTC

It looks like the networks are being deleted:
2017-04-18 12:33:35Z [overcloud-Networks-fliltytuxeie-StorageMgmtNetwork-a6ikyqz3czqv]: DELETE_IN_PROGRESS  Stack DELETE started

And the network deletes fail because the controller is still running.

Were the same environment files used for the initial deployment and the scale down? I expect a network environment file is missing.

Comment 5 Alex Schultz 2017-12-01 19:43:10 UTC

Networks start getting deleted if you don't include the same templates in subsequent 'openstack overcloud deploy' commands. Additionally as owalsh pointed out, that is not how you scale down.