Bug 1389515 - [Inquiry] Avoid redeployment / deletion of node with OSP Director [NEEDINFO]
Summary: [Inquiry] Avoid redeployment / deletion of node with OSP Director
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: James Slagle
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-27 18:55 UTC by Andreas Karis
Modified: 2020-02-14 18:04 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-17 19:05:03 UTC
Target Upstream Version:
aschultz: needinfo? (akaris)


Attachments (Terms of Use)

Description Andreas Karis 2016-10-27 18:55:11 UTC
Description of problem:
Looking for a way to avoid a node from being deleted in OSP Director. Either from user mistakes (such as deploying with a too low number of computes nodes, thus forcing a scale down) or from possible bugs in Director.

I tried to change IPMI credentials. The good thing is that the node cannot be reinstalled, because it's flagged as in maintenance mode and also it cannot be reached by ironic. But nova can still delete it.

Is there a way to prohibit deletion via nova? Is there a more elegant way to prohibit redeployment / a reinstallation of a node?

Version-Release number of selected component (if applicable):
OSP 9


Additional info:

I tried this:
~~~
for i in rhosp-4 rhops-5 rhosp-3 rhosp-2;do ironic node-update $i replace driver_info/ipmi_username=rcuser_disabled;done
~~~
After a while
~~~
[stack@undercloud-1 ~]$ ironic node-list
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
~~~
Scaling out
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=2, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 1 of 2 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Configuration has 2 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 16:14:26 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| e3fb81e5-3759-4e47-8ce8-e0a59561c3ee | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | e3fb81e5-3759-4e47-8ce8-e0a59561c3ee | power on    | active             | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
~~~

Running a scale down by 1 for the last node
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
(reverse-i-search)`': ^C
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=1, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Configuration has 1 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 17:42:14 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
2016-10-27 17:58:54 [CephStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
2016-10-27 17:58:54 [overcloud-BlockStorageNodesPostDeployment-rsohfqlgdc5d]: UPDATE_COMPLETE Stack UPDATE completed successfully
2016-10-27 17:58:55 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | 52c2d9c8-edcb-453e-b8a3-03f3fa3ca9cd | None        | active             | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T17:42:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
~~~

Running a scale down by 1 more node to remove the last node (simulate accidental deletion)
~~~
[stack@undercloud-1 ~]$ vim templates-akaris-rakuten/deploy.sh 
[stack@undercloud-1 ~]$ !$
templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=0, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Configuration has 1 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-10-27 18:16:14 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started
(...)
2016-10-27 18:33:07 [CephStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
2016-10-27 18:33:08 [overcloud-BlockStorageNodesPostDeployment-rsohfqlgdc5d]: UPDATE_COMPLETE Stack UPDATE completed successfully
2016-10-27 18:33:09 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://10.0.0.4:5000/v2.0
Overcloud Deployed
~~~

~~~
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| cb62daf7-7cd4-4b3d-8a56-34811b844090 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 5cddea1c-c837-4946-8150-ecd6cfaf1372 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.19 |
| 86b3c54c-5467-4cc0-90f4-ea6662909c6f | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name    | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
| 45c9bce3-fdb9-49d5-97f2-6f8aa0408220 | rhosp-4 | 86b3c54c-5467-4cc0-90f4-ea6662909c6f | None        | active             | True        |
| 4268de21-1ce8-403d-9020-e29ff192e425 | rhops-5 | 5cddea1c-c837-4946-8150-ecd6cfaf1372 | None        | active             | True        |
| 64dedaa7-d48d-4fb0-99d6-bc4d9b1689fc | rhosp-3 | cb62daf7-7cd4-4b3d-8a56-34811b844090 | None        | active             | True        |
| cccc0b18-a9b0-4979-8c2b-abc6e24bd488 | rhosp-6 | None                                 | power off   | available          | False       |
| 1c06f462-2fc5-48c5-82ec-9549dfbaafc8 | rhosp-2 | None                                 | None        | error              | True        |
+--------------------------------------+---------+--------------------------------------+-------------+--------------------+-------------+
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T18:16:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
~~~

The node is deleted, but at least at is still online:
~~~
[stack@undercloud-1 ~]$ ping 192.0.2.18
PING 192.0.2.18 (192.0.2.18) 56(84) bytes of data.
64 bytes from 192.0.2.18: icmp_seq=1 ttl=64 time=0.228 ms
~~~

Trying another scale out by 2 nodes
~~~
[stack@undercloud-1 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 30934f26-d43b-4c69-94d2-7bc1763771da | overcloud  | UPDATE_COMPLETE | 2016-10-27T00:06:59 | 2016-10-27T18:16:14 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
[stack@undercloud-1 ~]$ templates-akaris-rakuten/deploy.sh
control_scale=3, compute_scale=2, ceph_scale=0
Error: only 0 of 3 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 1 of 2 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Configuration has 2 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
Deployment failed:  Not enough nodes - available: 4, requested: 5
[stack@undercloud-1 ~]$ 
~~~

Comment 11 Alex Schultz 2017-02-28 23:55:51 UTC
Andreas, Is there anything else you need for the bug or can we close it out? Thanks


Note You need to log in before you can comment on or make changes to this bug.