Bug 1389515

Summary: [Inquiry] Avoid redeployment / deletion of node with OSP Director
Product: Red Hat OpenStack Reporter: Andreas Karis <akaris>
Component: openstack-tripleoAssignee: James Slagle <jslagle>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: akaris, aschultz, jjoyce, jschluet, jslagle, mburns, mcornea, rhel-osp-director-maint, rhos-maint, slinaber, tonishim, tvignaud
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: aschultz: needinfo? (akaris)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-17 19:05:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Andreas Karis 2016-10-27 18:55:11 UTC 
Description of problem:
Looking for a way to avoid a node from being deleted in OSP Director. Either from user mistakes (such as deploying with a too low number of computes nodes, thus forcing a scale down) or from possible bugs in Director.

I tried to change IPMI credentials. The good thing is that the node cannot be reinstalled, because it's flagged as in maintenance mode and also it cannot be reached by ironic. But nova can still delete it.

Is there a way to prohibit deletion via nova? Is there a more elegant way to prohibit redeployment / a reinstallation of a node?

Version-Release number of selected component (if applicable):
OSP 9


Additional info:

I tried this:
~~~
for i in rhosp-4 rhops-5 rhosp-3 rhosp-2;do ironic node-update $i replace driver_info/ipmi_username=rcuser_disabled;done
~~~
After a while
~~~
[stack@undercloud-1 ~]$ ironic node-list
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
~~~
Scaling out
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=2, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 1 of 2 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Configuration has 2 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 16:14:26 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| e3fb81e5-3759-4e47-8ce8-e0a59561c3ee | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | e3fb81e5-3759-4e47-8ce8-e0a59561c3ee | power on    | active             | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
~~~

Running a scale down by 1 for the last node
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
(reverse-i-search)`': ^C
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=1, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Configuration has 1 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 17:42:14 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
2016-10-27 17:58:54 [CephStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
2016-10-27 17:58:54 [overcloud-BlockStorageNodesPostDeployment-rsohfqlgdc5d]: UPDATE_COMPLETE Stack UPDATE completed successfully
2016-10-27 17:58:55 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T17:42:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
~~~

Running a scale down by 1 more node to remove the last node (simulate accidental deletion)
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=0, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Configuration has 1 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 18:16:14 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
2016-10-27 18:33:07 [CephStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
2016-10-27 18:33:08 [overcloud-BlockStorageNodesPostDeployment-rsohfqlgdc5d]: UPDATE_COMPLETE Stack UPDATE completed successfully
2016-10-27 18:33:09 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | None                                 | None        | error              | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T18:16:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
~~~

The node is deleted, but at least at is still online:
~~~
[stack@undercloud-1 ~]$ ping 192.0.2.18
PING 192.0.2.18 (192.0.2.18) 56(84) bytes of data.
64 bytes from 192.0.2.18: icmp_seq=1 ttl=64 time=0.228 ms
~~~

Trying another scale out by 2 nodes
~~~
[stack@undercloud-1 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T18:16:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
[stack@undercloud-1 ~]$ templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=2, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 1 of 2 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Configuration has 2 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deployment failed:  Not enough nodes - available: 4, requested: 5
[stack@undercloud-1 ~]$ 
~~~
Comment 11 Alex Schultz 2017-02-28 23:55:51 UTC 
Andreas, Is there anything else you need for the bug or can we close it out? Thanks