Bug 1242796
| Summary: | Heat does not deletes failed stack at the first try | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> |
| Component: | openstack-heat | Assignee: | Steve Baker <sbaker> |
| Status: | CLOSED DUPLICATE | QA Contact: | Amit Ugol <augol> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | Director | CC: | mburns, oblaut, rybrown, sbaker, shardy, yeylon, zbitter |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream |
| Target Release: | 7.0 (Kilo) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-09-09 22:07:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It appears that you attempted to delete the stack before it had finished creating (or finished entering the CREATE_FAILED state). Since it had not completed its running action, the delete failed. When you tried again, the delete succeeded because the creation had finished failing. @Ryan: I think what actually can happen (I just hit a similar issue) is that if a stack ends up CREATE_FAILED because a nova server is in an ERROR state, we then try to delete it, but misinterpret the existing ERROR state as a failed delete. The server is still deleted, so you end up doing multiple deletes until all servers have actually been deleted. It's fixed upstream: https://bugs.launchpad.net/heat/+bug/1463716 *** Bug 1242983 has been marked as a duplicate of this bug. *** Actually I think the upstream fix is this one https://review.openstack.org/#/c/204301/ Without this fix, any stack-update or stack-delete on a CREATE_FAILED stack will likely fail @Ryan: I did not copy pasted all response in all cases, but the overcloud named stack was in *_FAILED, otherwise the delete would be rejected by the api. *** This bug has been marked as a duplicate of bug 1256564 *** |
Description of problem: I tried to deploy openstack with osp-director in virtual environment. My flavors does not matched the actual flavor size, due to this the boot was failed. I had issue with cleaning up the failed stack. Version-Release number of selected component (if applicable): rpm -qa |grep heat openstack-heat-engine-2015.1.0-4.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-35.el7ost.noarch openstack-heat-templates-0-0.6.20150605git.el7ost.noarch python-heatclient-0.6.0-1.el7ost.noarch openstack-heat-api-2015.1.0-4.el7ost.noarch heat-cfntools-1.2.8-2.el7.noarch openstack-heat-common-2015.1.0-4.el7ost.noarch openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch How reproducible: Seams reproducible (3 from 3 try), some cases even 2th delete attempt does not seams to work. Steps to Reproduce: [stack@instack ~]$ openstack overcloud deploy --plan-uuid `openstack management plan list | awk '/overcloud/{print $2}'` --control-scale 3 --ceph-storage-scale 3 --compute-scale 1 The following templates will be written: /tmp/tmp8qmb41/puppet/manifests/overcloud_volume.pp /tmp/tmp8qmb41/hieradata/object.yaml /tmp/tmp8qmb41/puppet/hieradata/common.yaml /tmp/tmp8qmb41/extraconfig/tasks/yum_update.yaml /tmp/tmp8qmb41/puppet/controller-config-pacemaker.yaml /tmp/tmp8qmb41/provider-Cinder-Storage-1.yaml /tmp/tmp8qmb41/provider-Compute-1.yaml /tmp/tmp8qmb41/network/noop.yaml /tmp/tmp8qmb41/puppet/bootstrap-config.yaml /tmp/tmp8qmb41/puppet/cinder-storage-post.yaml /tmp/tmp8qmb41/provider-Ceph-Storage-1.yaml /tmp/tmp8qmb41/puppet/controller-post-puppet.yaml /tmp/tmp8qmb41/puppet/cinder-storage-puppet.yaml /tmp/tmp8qmb41/puppet/manifests/overcloud_cephstorage.pp /tmp/tmp8qmb41/puppet/hieradata/object.yaml /tmp/tmp8qmb41/puppet/controller-puppet.yaml /tmp/tmp8qmb41/puppet/manifests/overcloud_controller_pacemaker.pp /tmp/tmp8qmb41/net-config-bridge.yaml /tmp/tmp8qmb41/puppet/swift-storage-post.yaml /tmp/tmp8qmb41/provider-Controller-1.yaml /tmp/tmp8qmb41/network/networks.yaml /tmp/tmp8qmb41/puppet/hieradata/RedHat.yaml /tmp/tmp8qmb41/puppet/manifests/overcloud_object.pp /tmp/tmp8qmb41/hieradata/controller.yaml /tmp/tmp8qmb41/network/ports/ctlplane_vip.yaml /tmp/tmp8qmb41/hieradata/volume.yaml /tmp/tmp8qmb41/puppet/compute-post-puppet.yaml /tmp/tmp8qmb41/provider-Swift-Storage-1.yaml /tmp/tmp8qmb41/puppet/swift-storage-puppet.yaml /tmp/tmp8qmb41/extraconfig/tasks/yum_update.sh /tmp/tmp8qmb41/puppet/swift-devices-and-proxy-config.yaml /tmp/tmp8qmb41/network/ports/net_ip_map.yaml /tmp/tmp8qmb41/puppet/compute-puppet.yaml /tmp/tmp8qmb41/puppet/hieradata/volume.yaml /tmp/tmp8qmb41/puppet/ceph-storage-post-puppet.yaml /tmp/tmp8qmb41/network/ports/noop.yaml /tmp/tmp8qmb41/puppet/extraconfig/pre_deploy/default.yaml /tmp/tmp8qmb41/network/ports/net_ip_list_map.yaml /tmp/tmp8qmb41/puppet/hieradata/ceph.yaml /tmp/tmp8qmb41/puppet/vip-config.yaml /tmp/tmp8qmb41/puppet/hieradata/controller.yaml /tmp/tmp8qmb41/plan.yaml /tmp/tmp8qmb41/environment.yaml /tmp/tmp8qmb41/puppet/ceph-storage-puppet.yaml /tmp/tmp8qmb41/hieradata/compute.yaml /tmp/tmp8qmb41/puppet/hieradata/compute.yaml /tmp/tmp8qmb41/hieradata/ceph.yaml /tmp/tmp8qmb41/puppet/manifests/overcloud_compute.pp /tmp/tmp8qmb41/hieradata/common.yaml /tmp/tmp8qmb41/puppet/manifests/ringbuilder.pp /tmp/tmp8qmb41/extraconfig/post_deploy/default.yaml /tmp/tmp8qmb41/net-config-noop.yaml /tmp/tmp8qmb41/puppet/ceph-cluster-config.yaml /tmp/tmp8qmb41/firstboot/userdata_default.yaml /tmp/tmp8qmb41/puppet/all-nodes-config.yaml ERROR: openstack Heat Stack create failed. [stack@instack ~]$ nova list +--------------------------------------+-------------------------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+----------+ | c3e05305-e0f4-4985-b544-eef998ca56cd | overcloud-cephstorage-0 | ERROR | - | NOSTATE | | | 3df15dda-c40a-463c-bdb3-c05fc8372fa9 | overcloud-cephstorage-1 | ERROR | - | NOSTATE | | | a34b345a-4e11-4b2e-a3ae-ee3edec0589e | overcloud-cephstorage-2 | ERROR | - | NOSTATE | | | d483abf9-d780-4c16-9c68-60f5987197bf | overcloud-compute-0 | ERROR | - | NOSTATE | | | 9aea2603-81f7-4fe4-9642-9b348fddd8fe | overcloud-controller-0 | ERROR | - | NOSTATE | | | d735c3e1-064c-40e4-865f-063be0ee8a3c | overcloud-controller-1 | ERROR | - | NOSTATE | | | 0b42cf62-5506-4596-8e14-5df70e80869e | overcloud-controller-2 | ERROR | - | NOSTATE | | +--------------------------------------+-------------------------+--------+------------+-------------+----------+ [stack@instack ~]$ heat stack-delete overcloud +--------------------------------------+------------+--------------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+--------------------+----------------------+ | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | overcloud | DELETE_IN_PROGRESS | 2015-07-14T08:07:36Z | +--------------------------------------+------------+--------------------+----------------------+ [stack@instack ~]$ heat stack-list -n +--------------------------------------+----------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+ | id | stack_name | stack_status | creation_time | parent | +--------------------------------------+----------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+ | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | overcloud | DELETE_FAILED | 2015-07-14T08:07:36Z | None | | 898d710a-c77e-4325-8881-15973d5d0edc | overcloud-Networks-shbyvlkq4pux | CREATE_COMPLETE | 2015-07-14T08:07:38Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | a19578fc-f555-4af7-a822-294030efd27b | overcloud-Networks-shbyvlkq4pux-ExternalNetwork-ecjm6bs7vaw2 | CREATE_COMPLETE | 2015-07-14T08:07:38Z | 898d710a-c77e-4325-8881-15973d5d0edc | | e4bc3b16-4627-4611-b881-67c5fd1a24ee | overcloud-Networks-shbyvlkq4pux-StorageMgmtNetwork-issgzlvwydke | CREATE_COMPLETE | 2015-07-14T08:07:38Z | 898d710a-c77e-4325-8881-15973d5d0edc | | 310e8ab2-7be4-464e-a3f5-f57b52a4c257 | overcloud-Networks-shbyvlkq4pux-TenantNetwork-6vaboytxeg3h | CREATE_COMPLETE | 2015-07-14T08:07:39Z | 898d710a-c77e-4325-8881-15973d5d0edc | | 37310850-9ff0-41c1-8cdb-f4567931cbd1 | overcloud-Networks-shbyvlkq4pux-StorageNetwork-i2ki7qck6fbw | CREATE_COMPLETE | 2015-07-14T08:07:39Z | 898d710a-c77e-4325-8881-15973d5d0edc | | 6a04326e-13fd-4e8c-894a-eba1b17f5d9b | overcloud-Networks-shbyvlkq4pux-InternalNetwork-cguzsthgw3p3 | CREATE_COMPLETE | 2015-07-14T08:07:40Z | 898d710a-c77e-4325-8881-15973d5d0edc | | 53b1765e-b45d-4c27-a3eb-6a3bb2874ae7 | overcloud-StorageVirtualIP-z6trxemtwgjf | CREATE_COMPLETE | 2015-07-14T08:07:43Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | a1b2aae4-5848-4e32-a86f-c6deca39afa3 | overcloud-InternalApiVirtualIP-wal32x6twhxs | CREATE_COMPLETE | 2015-07-14T08:07:43Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 0ee39cb3-f61d-4b2e-9d7c-818b0300c910 | overcloud-PublicVirtualIP-6wnbmm43vpjy | CREATE_COMPLETE | 2015-07-14T08:07:44Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 30266f32-8e64-48b3-883e-e4ffebc41b51 | overcloud-RedisVirtualIP-crpk2rvhp4qy | CREATE_COMPLETE | 2015-07-14T08:07:44Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 39e87d38-7cb3-486e-97a3-192d8be68e9c | overcloud-StorageMgmtVirtualIP-ya4he6zkdvj3 | CREATE_COMPLETE | 2015-07-14T08:07:44Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 59b56469-090c-45b1-baee-525b851f97a8 | overcloud-VipMap-bmk75dzms6ne | CREATE_COMPLETE | 2015-07-14T08:07:46Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 1b824898-c27a-4e7f-be55-b6fb7960f86d | overcloud-Controller-6l5tbquvm6m2 | CREATE_FAILED | 2015-07-14T08:07:50Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | b7d61bd3-8675-49cf-8a68-d994fb517e0e | overcloud-Compute-eadysguwsipm | DELETE_FAILED | 2015-07-14T08:07:51Z | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | | 0d5efd3f-ebfa-4899-aec9-15296be4bfee | overcloud-Controller-6l5tbquvm6m2-1-jvdysyoayonk | CREATE_FAILED | 2015-07-14T08:07:53Z | 1b824898-c27a-4e7f-be55-b6fb7960f86d | | 2c625ab1-a0c8-4365-ae91-f72bc46bfbe6 | overcloud-Controller-6l5tbquvm6m2-1-jvdysyoayonk-NodeUserData-3srmobfrsqi6 | CREATE_COMPLETE | 2015-07-14T08:07:53Z | 0d5efd3f-ebfa-4899-aec9-15296be4bfee | | 83793d9d-c355-49aa-bfbb-b0bea406453d | overcloud-Controller-6l5tbquvm6m2-1-jvdysyoayonk-UpdateConfig-7mqtuew34wbg | CREATE_COMPLETE | 2015-07-14T08:07:54Z | 0d5efd3f-ebfa-4899-aec9-15296be4bfee | | d14abd86-dcb7-4940-b1c1-98e66042d8ea | overcloud-Controller-6l5tbquvm6m2-0-nczizth2bw5a | CREATE_FAILED | 2015-07-14T08:07:56Z | 1b824898-c27a-4e7f-be55-b6fb7960f86d | | a0daa296-641b-4dc8-99e2-afdf72be498e | overcloud-Controller-6l5tbquvm6m2-0-nczizth2bw5a-UpdateConfig-cdeiefetpngi | CREATE_COMPLETE | 2015-07-14T08:07:57Z | d14abd86-dcb7-4940-b1c1-98e66042d8ea | | f1bd72b7-21f7-40ab-b925-8bbfe9fac3d3 | overcloud-Controller-6l5tbquvm6m2-0-nczizth2bw5a-NodeUserData-naaslk4akrhw | CREATE_COMPLETE | 2015-07-14T08:07:57Z | d14abd86-dcb7-4940-b1c1-98e66042d8ea | | 7fe6ed23-78d5-4d16-bf97-8148c614857e | overcloud-Controller-6l5tbquvm6m2-2-4wtpeobakbbe | CREATE_FAILED | 2015-07-14T08:08:00Z | 1b824898-c27a-4e7f-be55-b6fb7960f86d | | e2fb4855-72bb-4374-b95c-a8cd18ade95b | overcloud-Controller-6l5tbquvm6m2-2-4wtpeobakbbe-UpdateConfig-cjm3ospnta6r | CREATE_COMPLETE | 2015-07-14T08:08:00Z | 7fe6ed23-78d5-4d16-bf97-8148c614857e | | 139529fe-eb67-4657-8f04-da337db2f30a | overcloud-Controller-6l5tbquvm6m2-2-4wtpeobakbbe-NodeUserData-f66r2skonpct | CREATE_COMPLETE | 2015-07-14T08:08:01Z | 7fe6ed23-78d5-4d16-bf97-8148c614857e | +--------------------------------------+----------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+ [stack@instack ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+----------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+----------+ | 9aea2603-81f7-4fe4-9642-9b348fddd8fe | overcloud-controller-0 | ERROR | - | NOSTATE | | | d735c3e1-064c-40e4-865f-063be0ee8a3c | overcloud-controller-1 | ERROR | - | NOSTATE | | | 0b42cf62-5506-4596-8e14-5df70e80869e | overcloud-controller-2 | ERROR | - | NOSTATE | | +--------------------------------------+------------------------+--------+------------+-------------+----------+ [stack@instack ~]$ heat stack-delete overcloud +--------------------------------------+------------+--------------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+--------------------+----------------------+ | b01a1b97-c98d-4d49-b0fc-bcb3f74e0657 | overcloud | DELETE_IN_PROGRESS | 2015-07-14T08:07:36Z | +--------------------------------------+------------+--------------------+----------------------+ [stack@instack ~]$ heat stack-list -n +----+------------+--------------+---------------+--------+ | id | stack_name | stack_status | creation_time | parent | +----+------------+--------------+---------------+--------+ +----+------------+--------------+---------------+--------+ Additional info: Not too match related log.: 2015-07-14 04:10:36.805 29938 INFO heat.engine.environment [-] Registering OS::TripleO::BlockStorage::Net::SoftwareConfig -> file:///tmp/tmp8qmb41/net-config-noop.yaml 2015-07-14 04:10:36.805 29938 INFO heat.engine.environment [-] Registering OS::TripleO::ObjectStorage::Net::SoftwareConfig -> file:///tmp/tmp8qmb41/net-config-noop.yaml 2015-07-14 04:10:36.806 29938 INFO heat.engine.environment [-] Registering OS::TripleO::Network::Ports::NetIpMap -> file:///tmp/tmp8qmb41/network/ports/net_ip_map.yaml 2015-07-14 04:10:36.806 29938 INFO heat.engine.environment [-] Registering OS::TripleO::Network::Storage -> file:///tmp/tmp8qmb41/network/noop.yaml 2015-07-14 04:10:36.806 29938 INFO heat.engine.environment [-] Registering OS::TripleO::SoftwareDeployment -> OS::Heat::StructuredDeployment 2015-07-14 04:10:36.807 29938 INFO heat.engine.environment [-] Registering OS::TripleO::Controller::Ports::StorageMgmtPort -> file:///tmp/tmp8qmb41/network/ports/noop.yaml 2015-07-14 04:10:36.807 29938 INFO heat.engine.environment [-] Registering OS::TripleO::SwiftStorage::Ports::StoragePort -> file:///tmp/tmp8qmb41/network/ports/noop.yaml 2015-07-14 04:10:36.808 29938 INFO heat.engine.environment [-] Registering OS::TripleO::Compute::Ports::StoragePort -> file:///tmp/tmp8qmb41/network/ports/noop.yaml 2015-07-14 04:10:36.808 29938 INFO heat.engine.environment [-] Registering OS::TripleO::ControllerPostDeployment -> file:///tmp/tmp8qmb41/puppet/controller-post-puppet.yaml 2015-07-14 04:10:36.812 29938 INFO heat.engine.resource [-] DELETE: TemplateResource "0" [8f1eabb2-ff58-444a-80f5-48d8b33b3f7e] Stack "overcloud-Compute-eadysguwsipm" [b7d61bd3-8675-49cf-8a68-d994fb517e0e] 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource Traceback (most recent call last): 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 500, in _action_recorder 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource yield 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 930, in delete 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource yield self.action_handler_task(action, *action_args) 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 313, in wrapper 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource step = next(subtask) 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 544, in action_handler_task 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource while not check(handler_data): 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 419, in check_delete_complete 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource show_deleted=True) 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 332, in _check_status_complete 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource status_reason=nested.status_reason) 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceInError: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"" 2015-07-14 04:10:36.812 29938 TRACE heat.engine.resource