Description of problem: When trying to delete the overcloud stack, heat failed on RHELUnregistrationDeployment resource step. Version-Release number of selected component (if applicable): openstack-nova-scheduler-2015.1.2-18.el7ost.noarch openstack-heat-templates-0-0.8.20150605git.el7ost.noarch openstack-dashboard-2015.1.2-4.el7ost.noarch openstack-nova-cert-2015.1.2-18.el7ost.noarch openstack-neutron-common-2015.1.2-9.el7ost.noarch openstack-heat-common-2015.1.2-9.el7ost.noarch openstack-ceilometer-api-2015.1.2-1.el7ost.noarch openstack-swift-container-2.3.0-4.el7ost.noarch openstack-nova-console-2015.1.2-18.el7ost.noarch openstack-tripleo-common-0.0.1.dev6-6.git49b57eb.el7ost.noarch openstack-neutron-2015.1.2-9.el7ost.noarch openstack-puppet-modules-2015.1.8-51.el7ost.noarch openstack-nova-compute-2015.1.2-18.el7ost.noarch openstack-nova-conductor-2015.1.2-18.el7ost.noarch openstack-swift-proxy-2.3.0-4.el7ost.noarch openstack-tuskar-ui-0.4.0-5.el7ost.noarch openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch openstack-utils-2014.2-1.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.2-9.el7ost.noarch openstack-ceilometer-notification-2015.1.2-1.el7ost.noarch openstack-dashboard-theme-2015.1.2-4.el7ost.noarch openstack-tuskar-ui-extras-0.0.4-2.el7ost.noarch openstack-glance-2015.1.2-1.el7ost.noarch openstack-ironic-conductor-2015.1.2-2.el7ost.noarch openstack-selinux-0.6.46-1.el7ost.noarch openstack-ironic-api-2015.1.2-2.el7ost.noarch openstack-keystone-2015.1.2-2.el7ost.noarch openstack-swift-plugin-swift3-1.7-3.el7ost.noarch openstack-heat-api-2015.1.2-9.el7ost.noarch openstack-ceilometer-central-2015.1.2-1.el7ost.noarch openstack-tempest-kilo-20151020.1.el7ost.noarch openstack-swift-2.3.0-4.el7ost.noarch openstack-neutron-ml2-2015.1.2-9.el7ost.noarch openstack-nova-novncproxy-2015.1.2-18.el7ost.noarch openstack-swift-object-2.3.0-4.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-121.el7ost.noarch openstack-nova-common-2015.1.2-18.el7ost.noarch openstack-tripleo-image-elements-0.9.6-10.el7ost.noarch openstack-heat-api-cfn-2015.1.2-9.el7ost.noarch openstack-ironic-common-2015.1.2-2.el7ost.noarch openstack-ceilometer-alarm-2015.1.2-1.el7ost.noarch openstack-neutron-openvswitch-2015.1.2-9.el7ost.noarch openstack-nova-api-2015.1.2-18.el7ost.noarch openstack-heat-engine-2015.1.2-9.el7ost.noarch openstack-ceilometer-common-2015.1.2-1.el7ost.noarch openstack-ironic-discoverd-1.1.0-8.el7ost.noarch openstack-swift-account-2.3.0-4.el7ost.noarch openstack-tripleo-puppet-elements-0.0.1-5.el7ost.noarch openstack-ceilometer-collector-2015.1.2-1.el7ost.noarch openstack-tuskar-0.4.18-5.el7ost.noarch How reproducible: Deploy a standard overcloud stack with RHEL Portal Registration extra step based on official documentation. Try to delete the stack Steps to Reproduce: 1.Deploy a standard overcloud stack with RHEL Portal Registration extra step 2.Delete the stack 3. Actual results: Stack deletion failed when Heat try to unregistered nodes. Currently, I didn't found any workaround to clean Heat stack. Expected results: Delete the overcloud stack Additional info:
Heat doesn't unregister the server; a shell script triggered by Heat unregisters the server. We'd need a lot more information about why it is not doing so to diagnose this, starting with the output of the script (which is available from "heat deployment-show"). A good bet is a network connectivity problem though.
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
Hi, one scenario I have reproduced this is e.g. when a node fails for some reason, e.g. defect disc and try to remove the failed node: [stack@undercloud ~]$ nova list +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ | b4c62f25-7f12-4e1b-9fcd-3046631c295c | BALOSCOMPINT00 | ACTIVE | - | Running | ctlplane=10.254.17.37 | | 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 | BALOSCOMPINT01 | ACTIVE | - | Running | ctlplane=10.254.17.38 | | abf95736-75b5-4d48-a02b-f4f641eab188 | BALOSCTL00 | ACTIVE | - | Running | ctlplane=10.254.17.39 | +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ [stack@undercloud ~]$ ironic node-list +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | 955a408e-8cf1-41a5-a777-d630b6cbe533 | overcloud-node3 | 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 | power on | active | False | | 8aca0579-add8-40a7-947c-5e1b8171e465 | overcloud-node4 | None | power off | available | False | | a34cbd6d-6102-40ae-b20a-a6f886a53cc1 | overcloud-node5 | None | power off | available | False | | 8a604223-779a-496f-b374-3b7fe7ffe000 | overcloud-node2 | b4c62f25-7f12-4e1b-9fcd-3046631c295c | power on | active | False | | b5a50354-ac17-47dd-a8b7-aa8528d2070e | overcloud-node1 | abf95736-75b5-4d48-a02b-f4f641eab188 | power on | active | False | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ => let's use BALOSCOMPINT01 and fail the disk -> overcloud-node3 # virsh destroy overcloud-node3 Domain overcloud-node3 destroyed Modified HD source file: <source file='/var/lib/libvirt/images/overcloud-node3.qcow2'/> <source file='/var/lib/libvirt/images/overcloud-node3-disable.qcow2'/> Power state changed to "power off" in ironic: [stack@undercloud ~]$ ironic node-list +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | 955a408e-8cf1-41a5-a777-d630b6cbe533 | overcloud-node3 | 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 | power off | active | False | | 8aca0579-add8-40a7-947c-5e1b8171e465 | overcloud-node4 | None | power off | available | False | | a34cbd6d-6102-40ae-b20a-a6f886a53cc1 | overcloud-node5 | None | power off | available | False | | 8a604223-779a-496f-b374-3b7fe7ffe000 | overcloud-node2 | b4c62f25-7f12-4e1b-9fcd-3046631c295c | power on | active | False | | b5a50354-ac17-47dd-a8b7-aa8528d2070e | overcloud-node1 | abf95736-75b5-4d48-a02b-f4f641eab188 | power on | active | False | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ $ cat delete-node.sh cd /home/stack openstack overcloud node delete --stack overcloud --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/templates/network-environment.yaml \ -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \ -e /home/stack/templates/rhel-registration/rhel-registration-resource-registry.yaml \ 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 [stack@undercloud ~]$ sh ./delete-node.sh deleting nodes ['7f7296c5-faeb-42d6-82c7-c3eb144de8e8'] from stack overcloud 2016-08-25 05:54:45.616 11994 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy_resource) {SoftwareConfig "RHELRegistration" [895cf747-4185-4712-a2fe-6f434e2a67c1] Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba -b142-535a2ac40740]: {}, SoftwareDeployment "RHELRegistrationDeployment" [eed44451-e252-4dc7-a391-57fb1d11b54a] Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba-b142-535a2ac40740]: {SoftwareConfig "RHELRegistration" [895cf747-41 85-4712-a2fe-6f434e2a67c1] Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba-b142-535a2ac40740]}, SoftwareDeployment "RHELUnregistrationDeployment" Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba-b142-535a2ac40740]: {SoftwareConfig "RHELUnregistration" [a829d9d5-d2ed-411a-8e32-a064dca67ade] Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba-b142-535a2ac40740]}, SoftwareConfig "RHELUnregistration" [a829d9d5-d2ed-411a-8e32-a064dca67ade] Stack "overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc" [7022d299-7d4e-40ba-b142-535a2ac40740]: {}}) running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:214 After timeout update fails: [stack@undercloud rhel-registration]$ heat stack-list +--------------------------------------+------------+---------------+---------------------+---------------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+---------------+---------------------+---------------------+ | 1a53f935-d362-4a10-9230-e70b27a96fde | overcloud | UPDATE_FAILED | 2016-08-25T08:08:42 | 2016-08-25T09:27:16 | +--------------------------------------+------------+---------------+---------------------+---------------------+ We see that the unregister is still in progress: [stack@undercloud rhel-registration]$ heat resource-list -n 5 overcloud | grep -v COMPLE +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | f1023c5b-6b32-445c-98bd-8d61027eb43e | OS::TripleO::Compute | DELETE_FAILED | 2016-08-25T08:09:10 | overcloud-Compute-sogzfuruohoh | | NodeExtraConfig | 7022d299-7d4e-40ba-b142-535a2ac40740 | OS::TripleO::NodeExtraConfig | DELETE_IN_PROGRESS | 2016-08-25T08:09:11 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3 | | RHELUnregistrationDeployment | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | OS::Heat::SoftwareDeployment | DELETE_IN_PROGRESS | 2016-08-25T08:18:15 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc | | Compute | 1eb1bdad-5044-41bf-b52a-ef7ad34b4eb7 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-08-25T09:27:55 | overcloud | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ Restarted heat engine to get the nested actions in PROGRESS to fail, too [stack@undercloud rhel-registration]$ heat resource-list -n 5 overcloud | grep -v COMPLE +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | f1023c5b-6b32-445c-98bd-8d61027eb43e | OS::TripleO::Compute | DELETE_FAILED | 2016-08-25T08:09:10 | overcloud-Compute-sogzfuruohoh | | NodeExtraConfig | 7022d299-7d4e-40ba-b142-535a2ac40740 | OS::TripleO::NodeExtraConfig | DELETE_FAILED | 2016-08-25T08:09:11 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3 | | RHELUnregistrationDeployment | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | OS::Heat::SoftwareDeployment | DELETE_FAILED | 2016-08-25T08:18:15 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc | | Compute | 1eb1bdad-5044-41bf-b52a-ef7ad34b4eb7 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-08-25T09:27:55 | overcloud | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ Tried to remove the instance from nova and delete then the node from heat. This procedure worked e.g. when the systems IPMI interface is also not reachable due to defect or such: [stack@undercloud ~]$ nova delete 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 Request to delete server 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 has been accepted. [stack@undercloud ~]$ ironic node-delete 955a408e-8cf1-41a5-a777-d630b6cbe533 Deleted node 955a408e-8cf1-41a5-a777-d630b6cbe533 [stack@undercloud ~]$ nova list +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ | b4c62f25-7f12-4e1b-9fcd-3046631c295c | BALOSCOMPINT00 | ACTIVE | - | Running | ctlplane=10.254.17.37 | | abf95736-75b5-4d48-a02b-f4f641eab188 | BALOSCTL00 | ACTIVE | - | Running | ctlplane=10.254.17.39 | +--------------------------------------+----------------+--------+------------+-------------+-----------------------+ [stack@undercloud ~]$ ironic node-list +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ | 8aca0579-add8-40a7-947c-5e1b8171e465 | overcloud-node4 | None | power off | available | False | | a34cbd6d-6102-40ae-b20a-a6f886a53cc1 | overcloud-node5 | None | power off | available | False | | 8a604223-779a-496f-b374-3b7fe7ffe000 | overcloud-node2 | b4c62f25-7f12-4e1b-9fcd-3046631c295c | power on | active | False | | b5a50354-ac17-47dd-a8b7-aa8528d2070e | overcloud-node1 | abf95736-75b5-4d48-a02b-f4f641eab188 | power on | active | False | +--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+ [stack@undercloud ~]$ sh ./delete-node.sh deleting nodes ['7f7296c5-faeb-42d6-82c7-c3eb144de8e8'] from stack overcloud Two objects are equal when all of the attributes are equal, if you want to identify whether two objects are same one with same id, please use is_same_obj() function. => same result [stack@undercloud ~]$ heat stack-list +--------------------------------------+------------+---------------+---------------------+---------------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+---------------+---------------------+---------------------+ | 1a53f935-d362-4a10-9230-e70b27a96fde | overcloud | UPDATE_FAILED | 2016-08-25T08:08:42 | 2016-08-25T14:55:33 | +--------------------------------------+------------+---------------+---------------------+---------------------+ [stack@undercloud ~]$ heat resource-list -n 5 overcloud | grep -v COMPLE +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | f1023c5b-6b32-445c-98bd-8d61027eb43e | OS::TripleO::Compute | DELETE_FAILED | 2016-08-25T08:09:10 | overcloud-Compute-sogzfuruohoh | | NodeExtraConfig | 7022d299-7d4e-40ba-b142-535a2ac40740 | OS::TripleO::NodeExtraConfig | DELETE_FAILED | 2016-08-25T08:09:11 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3 | | RHELUnregistrationDeployment | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | OS::Heat::SoftwareDeployment | DELETE_IN_PROGRESS | 2016-08-25T08:18:15 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc | | Compute | 1eb1bdad-5044-41bf-b52a-ef7ad34b4eb7 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-08-25T14:56:08 | overcloud | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ It also seems that the delete of the resource do not time out. Delete of the unregister is still in progress when I came in the next morning: [stack@undercloud ~]$ date Fri Aug 26 03:05:35 EDT 2016 => 9:05 CEST [stack@undercloud ~]$ heat resource-list -n 5 overcloud | grep -v COMPLE +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | f1023c5b-6b32-445c-98bd-8d61027eb43e | OS::TripleO::Compute | DELETE_FAILED | 2016-08-25T08:09:10 | overcloud-Compute-sogzfuruohoh | | NodeExtraConfig | 7022d299-7d4e-40ba-b142-535a2ac40740 | OS::TripleO::NodeExtraConfig | DELETE_IN_PROGRESS | 2016-08-25T08:09:11 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3 | | RHELUnregistrationDeployment | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | OS::Heat::SoftwareDeployment | DELETE_IN_PROGRESS | 2016-08-25T08:18:15 | overcloud-Compute-sogzfuruohoh-1-hlpidtyp3gs3-NodeExtraConfig-4p4qqcxsjrzc | | Compute | 1eb1bdad-5044-41bf-b52a-ef7ad34b4eb7 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-08-25T14:59:25 | overcloud | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@undercloud ~]$ heat resource-list 7022d299-7d4e-40ba-b142-535a2ac40740 +------------------------------+--------------------------------------+------------------------------+--------------------+---------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +------------------------------+--------------------------------------+------------------------------+--------------------+---------------------+ | RHELRegistration | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2016-08-25T08:18:15 | | RHELRegistrationDeployment | | OS::Heat::SoftwareDeployment | DELETE_COMPLETE | 2016-08-25T08:18:15 | | RHELUnregistration | a829d9d5-d2ed-411a-8e32-a064dca67ade | OS::Heat::SoftwareConfig | CREATE_COMPLETE | 2016-08-25T08:18:15 | | RHELUnregistrationDeployment | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | OS::Heat::SoftwareDeployment | DELETE_IN_PROGRESS | 2016-08-25T08:18:15 | +------------------------------+--------------------------------------+------------------------------+--------------------+---------------------+ [stack@undercloud ~]$ heat deployment-list |grep -v COMPLETE +--------------------------------------+--------------------------------------+--------------------------------------+--------+-------------+---------------------+-----------------------+ | id | config_id | server_id | action | status | creation_time | status_reason | +--------------------------------------+--------------------------------------+--------------------------------------+--------+-------------+---------------------+-----------------------+ | 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 | 93cc2115-9f8f-4d43-8352-2d87903e0c3c | 7f7296c5-faeb-42d6-82c7-c3eb144de8e8 | DELETE | IN_PROGRESS | 2016-08-25T09:28:00 | Deploy data available | +--------------------------------------+--------------------------------------+--------------------------------------+--------+-------------+---------------------+-----------------------+ [stack@undercloud ~]$ heat deployment-show 47b05a76-c70b-4fdc-b17f-1dbcb3bd3372 { "status": "IN_PROGRESS", "server_id": "7f7296c5-faeb-42d6-82c7-c3eb144de8e8", "config_id": "93cc2115-9f8f-4d43-8352-2d87903e0c3c", "output_values": null, "creation_time": "2016-08-25T09:28:00", "updated_time": "2016-08-25T14:59:30", "input_values": {}, "action": "DELETE", "status_reason": "Deploy data available", "id": "47b05a76-c70b-4fdc-b17f-1dbcb3bd3372" } How could we cleanup the stack in case such a system is broken and the unregister script can not be run any longer?
BTW, steps done in comment4 were done on an OSP8 env
This seem to be a duplicate of 1313885 . The workaround mentioned there works for me.
*** This bug has been marked as a duplicate of bug 1313885 ***