Bug 1378157 - Unable to delete overcloud node
Summary: Unable to delete overcloud node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: 10.0 (Newton)
Assignee: Ryan Brady
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-21 16:09 UTC by Marius Cornea
Modified: 2017-05-31 11:21 UTC (History)
14 users (show)

Fixed In Version: python-tripleoclient-5.3.0-5.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:03:19 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 374386 None None None 2016-09-23 08:23:16 UTC
OpenStack gerrit 382707 None None None 2016-10-06 19:10:57 UTC
OpenStack gerrit 383731 None None None 2016-10-14 02:09:58 UTC
OpenStack gerrit 390590 None None None 2016-11-02 04:23:23 UTC
Launchpad 1626736 None None None 2016-09-22 20:48:17 UTC

Description Marius Cornea 2016-09-21 16:09:42 UTC
Description of problem:
Deleting an overcloud node fails with:
Two objects are equal when all of the attributes are equal, if you want to identify whether two objects are same one with same id, please use is_same_obj() function.
<urlopen error [Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml'>


Version-Release number of selected component (if applicable):
python-tripleoclient-5.0.0-0.20160907170033.b0d7ce7.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160907212643.90c852e.1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates
openstack overcloud deploy --templates \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/storage-environment.yaml \
-e ~/templates/disk-layout.yaml \
-e ~/templates/wipe-disk-env.yaml \
-e ~/templates/enable-tls.yaml \
-e ~/templates/inject-trust-anchor.yaml \
-e ~/templates/tls-endpoints-public-ip.yaml \
-e ~/templates/password-env.yaml \
--control-scale 3 \
--control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \
--compute-scale 1 \
--compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
--ceph-storage-scale 1 \
--ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 \
--ntp-server clock.redhat.com 

2. Scale out with additional compute node:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates
openstack overcloud deploy --templates \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/storage-environment.yaml \
-e ~/templates/disk-layout.yaml \
-e ~/templates/wipe-disk-env.yaml \
-e ~/templates/enable-tls.yaml \
-e ~/templates/inject-trust-anchor.yaml \
-e ~/templates/tls-endpoints-public-ip.yaml \
-e ~/templates/password-env.yaml \
--control-scale 3 \
--control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \
--compute-scale 2 \
--compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
--ceph-storage-scale 1 \
--ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 \
--ntp-server clock.redhat.com 


3. Delete one compute node:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates
openstack overcloud node delete --stack overcloud --templates \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/storage-environment.yaml \
-e ~/templates/disk-layout.yaml \
-e ~/templates/wipe-disk-env.yaml \
-e ~/templates/enable-tls.yaml \
-e ~/templates/inject-trust-anchor.yaml \
-e ~/templates/tls-endpoints-public-ip.yaml \
-e ~/templates/password-env.yaml \
6fc44adf-9a46-41ad-af33-c623011e1457


Actual results:
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
deleting nodes [u'6fc44adf-9a46-41ad-af33-c623011e1457'] from stack overcloud
Two objects are equal when all of the attributes are equal, if you want to identify whether two objects are same one with same id, please use is_same_obj() function.
<urlopen error [Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/overcloud-without-mergepy.yaml'>


Expected results:
The node gets deleted.

Additional info:

Comment 2 James Slagle 2016-09-22 16:30:37 UTC
carlos, can you have a look at this one?

Comment 3 James Slagle 2016-09-22 21:04:49 UTC
the error is due to the scale down code not using the templates from the plan.

in general, what needs to happen here is that the scale_down function in scale.py from tripleo_common needs to get updated to use the templates from the plan, and the logic moved into a workflow so that the scale down functionality is available from the API/UI as well.

Comment 4 Jason E. Rist 2016-10-20 23:02:56 UTC
This is merged upstream!
https://review.openstack.org/#/c/383731/

Comment 5 Jason E. Rist 2016-10-20 23:03:37 UTC
Moving back from POST to ON_DEV due to the need for a backport. (I think)

Comment 6 Jason E. Rist 2016-11-02 04:22:41 UTC
This is merged upstream, and we're waiting on an ACK from backport patch:
https://review.openstack.org/#/c/390590/

Comment 7 Ola Pavlenko 2016-11-06 10:30:06 UTC
(In reply to Jason E. Rist from comment #6)
> This is merged upstream, and we're waiting on an ACK from backport patch:
> https://review.openstack.org/#/c/390590/

Merged!
Changing to Modified.

Comment 12 Ana Krivokapic 2016-11-23 14:02:15 UTC
This is still failing so moving back to NEW.

Comment 14 Ana Krivokapic 2016-11-29 13:01:30 UTC
I retested this running the following steps:

1. Deploy a simple 1 controller 1 compute overcloud
2. Scale up to 1 controller 2 computes
3. Delete the original compute node - succeeds
4. Try to delete the newly added compute node - fails with timeout

Details:

[stack@instack ~]$ openstack overcloud deploy --templates --control-scale 1 --compute-scale 1
Removing the current plan files
<snip>
Overcloud Deployed
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 762a6dcd-6d77-4c31-9f94-a157f5f74855 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 5a6d6c7d-fb78-4c6f-b4c0-bc0777bbbb70 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ openstack overcloud deploy --templates --control-scale 1 --compute-scale 2
Removing the current plan files
<snip>
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 762a6dcd-6d77-4c31-9f94-a157f5f74855 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
| 0c62ec51-e559-4674-aa6a-641cab341ffb | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| 5a6d6c7d-fb78-4c6f-b4c0-bc0777bbbb70 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ openstack overcloud node delete --templates --stack overcloud  762a6dcd-6d77-4c31-9f94-a157f5f74855
deleting nodes [u'762a6dcd-6d77-4c31-9f94-a157f5f74855'] from stack overcloud
Started Mistral Workflow. Execution ID: ec4baa68-4c8d-431e-b13d-0c2bde04c9b1
[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 0c62ec51-e559-4674-aa6a-641cab341ffb | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| 5a6d6c7d-fb78-4c6f-b4c0-bc0777bbbb70 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ openstack overcloud node delete --templates --stack overcloud  0c62ec51-e559-4674-aa6a-641cab341ffb
deleting nodes [u'0c62ec51-e559-4674-aa6a-641cab341ffb'] from stack overcloud
Started Mistral Workflow. Execution ID: 0a9d1f55-a457-4d5a-98b4-56323df7ad6a
{u'execution': {u'id': u'0a9d1f55-a457-4d5a-98b4-56323df7ad6a',
                u'input': {u'container': u'overcloud',
                           u'nodes': [u'0c62ec51-e559-4674-aa6a-641cab341ffb'],
                           u'queue_name': u'2cb6790c-6369-4e9b-bf41-cd40eb07a463',
                           u'timeout': 240},
                u'name': u'tripleo.scale.v1.delete_node',
                u'params': {},
                u'spec': {u'description': u'deletes given overcloud nodes and updates the stack',
                          u'input': [u'container',
                                     u'nodes',
                                     {u'timeout': 240},
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'delete_node',
                          u'tasks': {u'delete_node': {u'action': u'tripleo.scale.delete_node nodes=<% $.nodes %> timeout=<% $.timeout %> container=<% $.container %>',
                                                      u'name': u'delete_node',
                                                      u'on-error': u'set_delete_node_failed',
                                                      u'on-success': u'send_message',
                                                      u'type': u'direct',
                                                      u'version': u'2.0'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.scale.v1.delete_node'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'set_delete_node_failed': {u'name': u'set_delete_node_failed',
                                                                 u'on-success': u'send_message',
                                                                 u'publish': {u'message': u'<% task(delete_node).result %>',
                                                                              u'status': u'FAILED'},
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0'}},
                          u'version': u'2.0'}},
 u'message': u"Failed to run action [action_ex_id=92c75275-8091-4234-a764-95af3b1d1ddc, action_cls='<class 'mistral.actions.action_factory.ScaleDownAction'>', attributes='{}', params='{u'nodes': [u'0c62ec51-e559-4674-aa6a-641cab341ffb'], u'container': u'overcloud', u'timeout': 240}']\n ERROR: u'u\\'0\\'\\n",
 u'status': u'FAILED'}
[stack@instack ~]$

Comment 15 Ana Krivokapic 2016-11-29 14:06:04 UTC
Mistral executor log: https://paste.fedoraproject.org/492949/28204148/
Heat engine log: https://paste.fedoraproject.org/492954/14804283/

Comment 16 Julie Pichon 2016-11-29 15:36:57 UTC
I was able to verify this, using the following packages:
openstack-tripleo-common-5.4.0-2
python-tripleoclient-5.4.0-1

The commands I used are slightly different due to working with a non-default plan, but they worked: I could delete both the original compute node and the scaled out ones.

Details:
--------
$ nova list
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                             | Status | Task State | Power State | Networks            |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| bfa87c11-d1ee-4e9e-bf39-cc93717747ec | overcloud-ssl-ipv6-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 1b8700f8-fb42-4307-b467-b75e04ae2631 | overcloud-ssl-ipv6-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| baa55e94-2bc9-4e94-b571-ab5f4f368f25 | overcloud-ssl-ipv6-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+

$ openstack action execution run tripleo.parameters.update '{"container":"overcloud-ssl-ipv6", "parameters":{"ComputeCount":3}}'
$ openstack overcloud plan deploy overcloud-ssl-ipv6

$ nova list
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                             | Status | Task State | Power State | Networks            |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| bfa87c11-d1ee-4e9e-bf39-cc93717747ec | overcloud-ssl-ipv6-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 1b8700f8-fb42-4307-b467-b75e04ae2631 | overcloud-ssl-ipv6-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| aeed24fd-14e3-4c8d-9f58-19d8a041e81e | overcloud-ssl-ipv6-compute-1     | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| 6a013bad-55c6-4381-bc17-054836abf747 | overcloud-ssl-ipv6-compute-2     | ACTIVE | -          | Running     | ctlplane=192.0.2.15 |
| baa55e94-2bc9-4e94-b571-ab5f4f368f25 | overcloud-ssl-ipv6-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+

$ openstack overcloud node delete --stack overcloud-ssl-ipv6 6a013bad-55c6-4381-bc17-054836abf747
$ openstack overcloud node delete --stack overcloud-ssl-ipv6 1b8700f8-fb42-4307-b467-b75e04ae2631

$ nova list
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                             | Status | Task State | Power State | Networks            |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+
| bfa87c11-d1ee-4e9e-bf39-cc93717747ec | overcloud-ssl-ipv6-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| aeed24fd-14e3-4c8d-9f58-19d8a041e81e | overcloud-ssl-ipv6-compute-1     | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| baa55e94-2bc9-4e94-b571-ab5f4f368f25 | overcloud-ssl-ipv6-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
+--------------------------------------+----------------------------------+--------+------------+-------------+---------------------+

$ openstack overcloud node delete --stack overcloud-ssl-ipv6 aeed24fd-14e3-4c8d-9f58-19d8a041e81e
$ openstack overcloud node delete --stack overcloud-ssl-ipv6 bfa87c11-d1ee-4e9e-bf39-cc93717747ec

$ nova list
+--------------------------------------+---------------------------------+--------+------------+-------------+--------------------+
| ID                                   | Name                            | Status | Task State | Power State | Networks           |
+--------------------------------------+---------------------------------+--------+------------+-------------+--------------------+
| baa55e94-2bc9-4e94-b571-ab5f4f368f25 | overcloud-ssl-ipv6-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.9 |
+--------------------------------------+---------------------------------+--------+------------+-------------+--------------------+

Comment 18 errata-xmlrpc 2016-12-14 16:03:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.