Bug 1259834 - Instances stuck in Status DELETED Task State deleting [NEEDINFO]
Instances stuck in Status DELETED Task State deleting
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
7.0 (Kilo)
Unspecified Linux
unspecified Severity high
: ---
: 8.0 (Liberty)
Assigned To: Eoghan Glynn
nlevinki
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-03 11:12 EDT by Jeremy
Modified: 2016-04-21 11:13 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-21 11:13:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
eglynn: needinfo? (jmelvin)


Attachments (Terms of Use)

  None (edit)
Description Jeremy 2015-09-03 11:12:33 EDT
Description of problem:Instances stuck in Status DELETED Task State deleting


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.try to delete an existing instance 
2.
3.

Actual results:
fails to delete instance 

Expected results:
instance deleted.

Additional info:



nova --debug delete 34d34415-5d74-436d-9523-b5a67981125e
REQ: curl -i 'http://87.44.1.163:35357/v2.0/tokens'; -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "{SHA1}13478233491b920a3ab2bd4f2405210a55d7449c"}}}'
INFO (connectionpool:187) Starting new HTTP connection (1): 87.44.1.163
DEBUG (connectionpool:357) "POST /v2.0/tokens HTTP/1.1" 200 11403
RESP: [200] {'date': 'Wed, 02 Sep 2015 14:34:21 GMT', 'content-type': 'application/json', 'content-length': '11403', 'vary': 'X-Auth-Token'}
RESP BODY: {"access": {"token": {"issued_at": "2015-09-02T14:34:21.015572", "expires": "2015-09-02T15:34:20Z", "id": "{SHA1}666453e3f46a0f3a2066ff317825ae422b4b81d4", "tenant": {"enabled": true, "description": "admin tenant", "name": "admin", "id": "205f244a7c9541d1b2423b057d17f062"}, "audit_ids": ["-Hf6EU1fQwaNvdczDF_arg"]}, "serviceCatalog": [{"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.166:8774/v2/205f244a7c9541d1b2423b057d17f062", "region": "RegionOne", "publicURL": "http://87.44.1.18:8774/v2/205f244a7c9541d1b2423b057d17f062", "internalURL": "http://87.44.1.166:8774/v2/205f244a7c9541d1b2423b057d17f062", "id": "065aac8de3f04545a9438555866507ee"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.165:9696/", "region": "RegionOne", "publicURL": "http://87.44.1.17:9696/", "internalURL": "http://87.44.1.165:9696/", "id": "592da301d6ca40cebaf8dbab36c92467"}], "type": "network", "name": "neutron"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.156:8776/v2/205f244a7c9541d1b2423b057d17f062", "region": "RegionOne", "publicURL": "http://87.44.1.9:8776/v2/205f244a7c9541d1b2423b057d17f062", "internalURL": "http://87.44.1.156:8776/v2/205f244a7c9541d1b2423b057d17f062", "id": "479146580112446bb4e0ec750d70895c"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.166:8774/v3", "region": "RegionOne", "publicURL": "http://87.44.1.18:8774/v3", "internalURL": "http://87.44.1.166:8774/v3", "id": "1eb4ac7afca3419a89e65e59e53f0d67"}], "type": "computev3", "name": "novav3"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.167:8080", "region": "RegionOne", "publicURL": "http://87.44.1.167:8080", "internalURL": "http://87.44.1.167:8080", "id": "06861cb219344c46bb65f6d78fb9b039"}], "type": "s3", "name": "swift_s3"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.159:9292", "region": "RegionOne", "publicURL": "http://87.44.1.11:9292", "internalURL": "http://87.44.1.159:9292", "id": "5e088515ce3444639f957dcd87028f8b"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.155:8777", "region": "RegionOne", "publicURL": "http://87.44.1.8:8777", "internalURL": "http://87.44.1.155:8777", "id": "01ab36015e5f451cad01a9c51d108a35"}], "type": "metering", "name": "ceilometer"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.161:8000/v1/", "region": "RegionOne", "publicURL": "http://87.44.1.13:8000/v1/", "internalURL": "http://87.44.1.161:8000/v1/", "id": "85f73f3947ec43c8b3b906793a8243db"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.156:8776/v1/205f244a7c9541d1b2423b057d17f062", "region": "RegionOne", "publicURL": "http://87.44.1.9:8776/v1/205f244a7c9541d1b2423b057d17f062", "internalURL": "http://87.44.1.156:8776/v1/205f244a7c9541d1b2423b057d17f062", "id": "26f3c3fd2a644f55bcd6fc1687b63c0d"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.166:8773/services/Admin", "region": "RegionOne", "publicURL": "http://87.44.1.18:8773/services/Cloud", "internalURL": "http://87.44.1.166:8773/services/Cloud", "id": "864fce2640844c09a0a4c7b8d6b4542a"}], "type": "ec2", "name": "nova_ec2"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.160:8004/v1/205f244a7c9541d1b2423b057d17f062", "region": "RegionOne", "publicURL": "http://87.44.1.12:8004/v1/205f244a7c9541d1b2423b057d17f062", "internalURL": "http://87.44.1.160:8004/v1/205f244a7c9541d1b2423b057d17f062", "id": "1564186189e94095a3a2d84961d11b1b"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.167:8080", "region": "RegionOne", "publicURL": "http://87.44.1.167:8080/v1/AUTH_205f244a7c9541d1b2423b057d17f062", "internalURL": "http://87.44.1.167:8080/v1/AUTH_205f244a7c9541d1b2423b057d17f062", "id": "1a07e89446f84923969a4ad79e96b01e"}], "type": "object-store", "name": "swift"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://87.44.1.163:35357/v2.0", "region": "RegionOne", "publicURL": "http://87.44.1.15:5000/v2.0", "internalURL": "http://87.44.1.163:5000/v2.0", "id": "53eb413647ea4ffc8512a97b65b1db49"}], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "1a25f8911ce048c1afc855a15c7cdcf5", "roles": [{"name": "admin"}, {"name": "heat_stack_owner"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["875a8dfae6ae425c947015cc60e9e702", "172b9af1012944c886ae15512257afa9"]}}}

REQ: curl -i 'http://87.44.1.18:8774/v2/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e'; -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: admin" -H "X-Auth-Token: {SHA1}666453e3f46a0f3a2066ff317825ae422b4b81d4"
INFO (connectionpool:187) Starting new HTTP connection (1): 87.44.1.18
DEBUG (connectionpool:357) "GET /v2/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e HTTP/1.1" 200 4541
RESP: [200] {'date': 'Wed, 02 Sep 2015 14:34:21 GMT', 'content-length': '4541', 'content-type': 'application/json', 'x-compute-request-id': 'req-52b934b4-9648-4f17-ab8a-0da395472d48'}
RESP BODY: {"server": {"OS-EXT-STS:task_state": "deleting", "addresses": {"admin-net": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:06:5e:62", "version": 4, "addr": "1.1.1.21", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://87.44.1.18:8774/v2/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e", "rel": "self"}, {"href": "http://87.44.1.18:8774/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e", "rel": "bookmark"}], "image": {"id": "19c020f9-bed7-4df3-a15c-3991a7f232cc", "links": [{"href": "http://87.44.1.18:8774/205f244a7c9541d1b2423b057d17f062/images/19c020f9-bed7-4df3-a15c-3991a7f232cc", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", "OS-EXT-SRV-ATTR:instance_name": "instance-00005cc3", "OS-SRV-USG:launched_at": "2015-09-01T07:32:55.000000", "flavor": {"id": "3", "links": [{"href": "http://87.44.1.18:8774/205f244a7c9541d1b2423b057d17f062/flavors/3", "rel": "bookmark"}]}, "id": "34d34415-5d74-436d-9523-b5a67981125e", "security_groups": [{"name": "default"}], "user_id": "1a25f8911ce048c1afc855a15c7cdcf5", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ERROR", "updated": "2015-09-02T14:29:34Z", "hostId": "f85bd20b2606c62e671565207639adc1c18b8665cd57943febaf7f56", "OS-EXT-SRV-ATTR:host": "osp1-compute03.osp.poc", "OS-SRV-USG:terminated_at": null, "key_name": "KR", "OS-EXT-SRV-ATTR:hypervisor_hostname": "osp1-compute03.osp.poc", "name": "KRSTRESS-00117", "created": "2015-09-01T07:32:43Z", "tenant_id": "205f244a7c9541d1b2423b057d17f062", "os-extended-volumes:volumes_attached": [], "fault": {"message": "Timed out waiting for a reply to message ID fa438dae28c04ae68e89316d22bfce13", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 325, in decorated_function\n    return function(self, context, *args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2567, in terminate_instance\n    do_terminate_instance(instance, bdms)\n  File \"/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\", line 272, in inner\n    return f(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2565, in do_terminate_instance\n    self._set_instance_error_state(context, instance)\n  File \"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 82, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2555, in do_terminate_instance\n    self._delete_instance(context, instance, bdms, quotas)\n  File \"/usr/lib/python2.7/site-packages/nova/hooks.py\", line 131, in inner\n    rv = f(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2524, in _delete_instance\n    quotas.rollback()\n  File \"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 82, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2498, in _delete_instance\n    instance.info_cache.delete()\n  File \"/usr/lib/python2.7/site-packages/nova/objects/base.py\", line 189, in wrapper\n    ctxt, self, fn.__name__, args, kwargs)\n  File \"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py\", line 364, in object_action\n    objmethod=objmethod, args=args, kwargs=kwargs)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py\", line 152, in call\n    retry=self.retry)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/transport.py\", line 90, in _send\n    timeout=timeout, retry=retry)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py\", line 408, in send\n    retry=retry)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py\", line 397, in _send\n    result = self._waiter.wait(msg_id, timeout)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py\", line 298, in wait\n    reply, ending, trylock = self._poll_queue(msg_id, timeout)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py\", line 238, in _poll_queue\n    message = self.waiters.get(msg_id, timeout)\n  File \"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py\", line 144, in get\n    'to message ID %s' % msg_id)\n", "created": "2015-09-02T11:53:03Z"}, "metadata": {}}}

REQ: curl -i 'http://87.44.1.18:8774/v2/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e'; -X DELETE -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: admin" -H "X-Auth-Token: {SHA1}666453e3f46a0f3a2066ff317825ae422b4b81d4"
INFO (connectionpool:187) Starting new HTTP connection (1): 87.44.1.18
DEBUG (connectionpool:357) "DELETE /v2/205f244a7c9541d1b2423b057d17f062/servers/34d34415-5d74-436d-9523-b5a67981125e HTTP/1.1" 204 0
RESP: [204] {'date': 'Wed, 02 Sep 2015 14:34:22 GMT', 'content-length': '0', 'content-type': 'application/json', 'x-compute-request-id': 'req-6d1ee08b-fd3b-4dd7-bf5e-d2de03e06f69'}
RESP BODY: null

Request to delete server 34d34415-5d74-436d-9523-b5a67981125e has been accepted.
[root@osp1-controller01 ~(openstack_admin)]#
Comment 6 Eoghan Glynn 2015-09-11 08:43:41 EDT
The standard operating procedure in such cases is to reset the nova state via:

  $ nova reset-state $INSTANCE_ID

then re-try the deletion action.

These kind of problems are usually associated with some infrastructure failure, and a admin-driven retry after a reset is generally the best course of action.

Please confirm if this resolves the issue.
Comment 7 Eoghan Glynn 2016-04-21 11:13:29 EDT
Closing due to a lack of response to the request for information on 2015-09-11.

Please feel free to reopen if you can provide that information.

Note You need to log in before you can comment on or make changes to this bug.