Bug 1683357

Summary:	After deleting compute nodes, instance uuid does not get removed from ironic node-list
Product:	Red Hat OpenStack	Reporter:	vivek koul <vkoul>
Component:	openstack-ironic	Assignee:	Steve Baker <sbaker>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	mlammon
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	10.0 (Newton)	CC:	bfournie, dasmith, dtantsur, eglynn, ipetrova, jhakimra, jkreger, kchamart, mburns, sbaker, sbauza, sgordon, vromanso, ykulkarn
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-16 21:32:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description vivek koul 2019-02-26 16:33:56 UTC

Description of problem:

After deleting compute node. Instance uuid remain attach to ironic nodes. It doesn't get removed automatically. manual intervention is always required.

Version-Release number of selected component (if applicable):
RHOSP 10

instack-5.1.0-1.el7ost.noarch                               Fri Jan 25 09:16:09 2019
instack-undercloud-5.3.7-6.el7ost.noarch                    Fri Feb  1 11:44:23 2019


openstack-ironic-api-6.2.4-3.el7ost.noarch                  Fri Jan 25 09:23:48 2019
openstack-ironic-common-6.2.4-3.el7ost.noarch               Fri Jan 25 09:23:03 2019
openstack-ironic-conductor-6.2.4-3.el7ost.noarch            Fri Jan 25 09:23:40 2019
openstack-ironic-inspector-4.2.2-4.el7ost.noarch            Fri Jan 25 09:23:44 2019
puppet-ironic-9.5.0-3.el7ost.noarch                         Fri Jan 25 09:16:15 2019
python-ironic-inspector-client-1.10.0-1.el7ost.noarch       Fri Jan 25 09:16:20 2019
python-ironic-lib-2.1.3-2.el7ost.noarch                     Fri Feb  1 11:48:21 2019
python-ironicclient-1.7.1-1.el7ost.noarch                   Fri Jan 25 09:16:20 2019

How reproducible:


Steps to Reproduce:
1.openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID]

2. nova list

3. ironic node-list

Actual results:

Deleted node has instance uuid attached with it 

Expected results:

Deleted node shouldn't be having instance uuid attached it should be in available state

Additional info:

[stack@director-computezone ~]$ openstack stack list
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| e797e8a5-dfaf-40cc-80a4-74815dd38121 | overcloud  | UPDATE_COMPLETE | 2019-01-25T09:58:16Z | 2019-02-26T14:02:17Z |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
[stack@director-computezone ~]$ openstack server list
+--------------------------------------+------------------+--------+------------------------+------------------------+
| ID                                   | Name             | Status | Networks               | Image Name             |
+--------------------------------------+------------------+--------+------------------------+------------------------+
| 74e88e1f-8704-4d80-8b32-449596009d82 | cpt-5-m640v5dpdk | ACTIVE | ctlplane=172.31.150.20 | overcloud-compute-dpdk |
| b139fb5d-ff66-45c1-8252-443cf5df508b | ceph-2           | ACTIVE | ctlplane=172.31.150.24 | overcloud-full         |
| c0ef7334-77ed-40bf-845c-8c15b629c689 | cpt-2-m630v4dpdk | ACTIVE | ctlplane=172.31.150.23 |                        |
| 6c9b0e4f-2813-4a73-9482-5ebb7cc24691 | cpt-3-m630v4dpdk | ACTIVE | ctlplane=172.31.150.11 |                        |
| c976dddf-1ef6-4959-a579-ed89a789a448 | ceph-0           | ACTIVE | ctlplane=172.31.150.18 | overcloud-full         |
| cd3c8733-ef58-4bb8-bc9c-2ce49ca84850 | ceph-1           | ACTIVE | ctlplane=172.31.150.22 | overcloud-full         |
| ce1d72da-85e2-4634-8d2d-bdc593b9c635 | cpt-4-m640v5dpdk | ACTIVE | ctlplane=172.31.150.14 |                        |
+--------------------------------------+------------------+--------+------------------------+------------------------+
[stack@director-computezone ~]$ openstack baremetal node list
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name             | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
| ea093024-b8d9-4e75-bb6f-663e4c755ac4 | cpt-1-m630v4     | 0cec9b1d-f56c-422e-b645-bcc3ffcc5c70 | power on    | active             | False       |  <==== This is deleted node still having instance uuid
| d6345dd0-1a49-42e0-92b6-23c30bffe3f5 | cpt-2-m630v4dpdk | c0ef7334-77ed-40bf-845c-8c15b629c689 | power on    | active             | False       |
| 166d7560-f63c-402d-920d-a69b3db861f3 | cpt-3-m630v4dpdk | 6c9b0e4f-2813-4a73-9482-5ebb7cc24691 | power on    | active             | False       |
| 308d49ca-e707-473d-ace9-86fbe44e85c4 | cpt-4-m640v5dpdk | ce1d72da-85e2-4634-8d2d-bdc593b9c635 | power on    | active             | False       |
| 0f7b8bd9-4f1c-420b-a1f9-7f4dcbb8d195 | ceph-0           | c976dddf-1ef6-4959-a579-ed89a789a448 | power on    | active             | False       |
| eaf79118-051c-457d-adc7-b17aabca9fe7 | ceph-1           | cd3c8733-ef58-4bb8-bc9c-2ce49ca84850 | power on    | active             | False       |
| 7efac84a-9be0-4c24-83bf-0733959f8624 | ceph-2           | b139fb5d-ff66-45c1-8252-443cf5df508b | power on    | active             | False       |
| dcfa3cf5-acf2-4bb4-b235-eb8b13472245 | cpt-5-m640v5dpdk | 74e88e1f-8704-4d80-8b32-449596009d82 | power on    | active             | False       |
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
--------------------

Comment 2 Bob Fournier 2019-02-26 19:18:02 UTC

I see these errors related to the the baremetal node being deleted (ea093024-b8d9-4e75-bb6f-663e4c755ac4) in ironic-api.log:

NodeAssociated: Node ea093024-b8d9-4e75-bb6f-663e4c755ac4 is associated with instance 0cec9b1d-f56c-422e-b645-bcc3ffcc5c70.
 format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222
2019-02-21 14:22:25.205 5019 INFO eventlet.wsgi.server [req-1b24bb6f-0827-4f3d-98a8-f09d88b7d430 admin admin - - -] 172.31.150.2 "DELETE /v1/nodes/ea093024-b8d9-4e75-bb6f-663e4c755ac4 HTTP/1.1" status: 409  len: 526 time: 0.1746671
2019-02-21 14:22:27.258 5019 DEBUG wsme.api [req-fcd6f847-e355-4712-9817-a80937f6f59b admin admin - - -] Client-side error: Node ea093024-b8d9-4e75-bb6f-663e4c755ac4 is associated with instance 0cec9b1d-f56c-422e-b645-bcc3ffcc5c70.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/ironic/conductor/manager.py", line 1472, in destroy_node
    instance=node.instance_uuid)

Comment 3 Julia Kreger 2019-02-26 19:46:39 UTC

Looks like the client is requesting to completely delete the node, so this is actually returning an error as designed. It almost looks like there was an attempt to delete the node as opposed to unprovision the node. The node must be unprovided before being deleted and it is the unprovision action that removes the associated instance UUID.