Bug 1245226

Summary: 'openstack overcloud node delete/ or update' fails when the stack is deployed with network isolation
Product: Red Hat OpenStack Reporter: Ronelle Landy <rlandy>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: hbrock, jprovazn, mandreou, mburns, mcornea, ohochman, rhel-osp-director-maint, rlandy, sclewis, srevivo, whayutin, zbitter
Target Milestone: z1Keywords: Automation, Triaged, ZStream
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-20 16:30:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ronelle Landy 2015-07-21 14:07:14 UTC
Description of problem:

When the initial stack is deployed with network isolation attempting to delete a node fails with the following error:

UPDATE_FAILED ... ResourceUnknownStatus: Resource failed - Unknown status  FAILED due to "Conflict: Unable to complete operation on subnet e9bba091-81e8-4e89-9be2-37f70ea5165b. One or more ports  have an IP allocation from this subnet."" 

Version-Release number of selected component (if applicable):

[stack@instack ~]$ rpm -qa | grep openstack
openstack-nova-console-2015.1.0-16.el7ost.noarch
openstack-neutron-2015.1.0-12.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch
openstack-ironic-conductor-2015.1.0-9.el7ost.noarch
openstack-ceilometer-api-2015.1.0-8.el7ost.noarch
openstack-ironic-api-2015.1.0-9.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tuskar-ui-0.3.0-12.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch
openstack-ceilometer-notification-2015.1.0-8.el7ost.noarch
openstack-nova-compute-2015.1.0-16.el7ost.noarch
openstack-nova-conductor-2015.1.0-16.el7ost.noarch
openstack-swift-account-2.3.0-1.el7ost.noarch
python-django-openstack-auth-1.2.0-3.el7ost.noarch
openstack-ironic-discoverd-1.1.0-5.el7ost.noarch
openstack-neutron-openvswitch-2015.1.0-12.el7ost.noarch
openstack-nova-api-2015.1.0-16.el7ost.noarch
openstack-nova-common-2015.1.0-16.el7ost.noarch
openstack-heat-common-2015.1.0-4.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-6.el7ost.noarch
python-openstackclient-1.0.3-2.el7ost.noarch
openstack-tuskar-0.4.18-3.el7ost.noarch
openstack-dashboard-theme-2015.1.0-10.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-tempest-kilo-20150708.2.el7ost.noarch
openstack-swift-2.3.0-1.el7ost.noarch
openstack-neutron-ml2-2015.1.0-12.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-16.el7ost.noarch
openstack-nova-scheduler-2015.1.0-16.el7ost.noarch
openstack-swift-object-2.3.0-1.el7ost.noarch
openstack-nova-cert-2015.1.0-16.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-neutron-common-2015.1.0-12.el7ost.noarch
openstack-heat-engine-2015.1.0-4.el7ost.noarch
openstack-ceilometer-common-2015.1.0-8.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-44.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-8.el7ost.noarch
openstack-keystone-2015.1.0-4.el7ost.noarch
openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-8.el7ost.noarch
openstack-ironic-common-2015.1.0-9.el7ost.noarch
openstack-selinux-0.6.37-1.el7ost.noarch
openstack-swift-proxy-2.3.0-1.el7ost.noarch
redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch
openstack-glance-2015.1.0-6.el7ost.noarch
openstack-tripleo-common-0.0.1.dev6-1.git49b57eb.el7ost.noarch
openstack-heat-api-2015.1.0-4.el7ost.noarch
openstack-ceilometer-central-2015.1.0-8.el7ost.noarch
openstack-puppet-modules-2015.1.8-7.el7ost.noarch
openstack-swift-container-2.3.0-1.el7ost.noarch
openstack-dashboard-2015.1.0-10.el7ost.noarch


[stack@instack ~]$ rpm -qa | grep osc
python-rdomanager-oscplugin-0.0.8-41.el7ost.noarch

How reproducible:

Always

Steps to Reproduce:
1. Installed openstack on a virt system
2. Deploy the overcloud with two compute nodes (or scale up from one to two compute nodes after deploy) also using network isolation
3. Delete one compute node

Actual results:

UPDATE _FAILED

Expected results:

UPDATE_COMPLETE

Additional info:

Comment 6 Omri Hochman 2015-07-22 15:18:56 UTC
Need to verify when attempting to update stack with network-isolation without sending the .yaml files that the stack won't get stack on UPDATE_FAILED  : 

[stack@rhos-compute-node-18 ~]$ openstack overcloud update stack overcloud --plan overcloud -i --debug

[stack@rhos-compute-node-18 ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| b9d47a7d-48e7-4902-a51c-7472275e6958 | overcloud  | UPDATE_FAILED | 2015-07-20T23:41:35Z |
+--------------------------------------+------------+---------------+----------------------+

Comment 7 chris alfonso 2015-07-24 15:27:02 UTC
Ronelle, should we move this bug to docs then?

Comment 8 Marios Andreou 2015-08-04 14:21:54 UTC
Poked at this today. I could successfully update to 2 compute nodes from 1 and also delete a node, but with the caveat that rlandy mentions (including the -e for the network isolation in the node delete commandline args). To this end, there is a docs review @ https://review.gerrithub.io/#/c/242143 that aims to help here.

Testing notes below, if useful:

DEPLOY 1/1/1:
openstack overcloud deploy --plan overcloud --debug --log-file overcloud_deployment_net_isolation.log --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml

SCALE ADD COMPUTE:
openstack management plan set -S Compute-1=2 fe58fe8a-15b9-49d6-bfb1-e85085bfae38
openstack overcloud deploy --debug --plan overcloud -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml

DELETE A COMPUTE:
openstack overcloud node delete --stack overcloud --debug --plan fe58fe8a-15b9-49d6-bfb1-e85085bfae38 27e3caf9-a7b4-4ff6-a5ce-d90b12282bbe

^^^FAIL because I didn't include the network isolation env

47:| Networks                          | d5f2bf6b-f751-4f48-8c3c-c4cef78014c3          | OS::TripleO::Network                              | UPDATE_FAILED   | 2015-08-04T12:02:13Z |

DELETE STACK AND DEPLOY AGAIN WITH 2 COMPUTE:
openstack overcloud deploy --plan overcloud --debug --log-file overcloud_deployment_net_isolation.log --control-scale 1 --compute-scale 2 --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml

DELETE A COMPUTE:
openstack overcloud node delete --stack overcloud --debug --plan fe58fe8a-15b9-49d6-bfb1-e85085bfae38  5e96f8bc-424f-4a32-be7d-5a4d59050d37 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml

~10mins later UPDATE_COMPLETE and node gone

Comment 9 Ronelle Landy 2015-08-10 13:56:59 UTC
(In reply to chris alfonso from comment #7)
> Ronelle, should we move this bug to docs then?

Yes - was basically a user error