Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/

Bug 1093880

Summary:	Instances that fail due to missing Neutron notification configuration cannot be deleted.
Product:	[Community] RDO	Reporter:	Lars Kellogg-Stedman <lars>
Component:	openstack-nova	Assignee:	RHOS Maint <rhos-maint>
Status:	CLOSED UPSTREAM	QA Contact:	Ami Jeain <ajeain>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	kchamart, lars, ndipanov, rbryant, sgordon, yeylon
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-14 02:09:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1093877

Description Lars Kellogg-Stedman 2014-05-03 01:03:54 UTC

If someone attempts to boot an instance in Icehouse after an upgrade from Havana and they have not yet configured /etc/neutron/neutron.conf with the appropriate keystone credentials for sending notifications to Nova, the instance will fail to spawn.

Additionally, attempts to delete the instance will result in:

2014-05-03 00:50:45.757 7433 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: 
Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvob7f77a1e
-c5']

An administrator can work around this problem by running:

  nova reset-state <instance>

After this, the delete will proceed successfully.

Comment 1 Stephen Gordon 2014-05-22 16:07:08 UTC

Hi Lars,

I think the appropriate approach is to:

a) Document the workaround in the upgrade notes.

b) File an upstream bug to fix the fact that you can't delete it.

Thoughts?

Comment 2 Kashyap Chamarthy 2015-02-19 14:32:11 UTC

[Ping. Bug triaging here,  any updates here? No comments since about 9 months. Is this still reproducible with during an IceHouse -> Juno upgrade?.]

Comment 3 Lars Kellogg-Stedman 2015-02-20 15:39:16 UTC

This bug still exists in Juno.  That is, if you do not have Neutron configured appropriately and you start an instance, it will end up in the ERROR state.  If you try to delete that instance, you will see in the Nova logs:

2015-02-20 10:34:44.672 2214 AUDIT nova.compute.manager [req-a4b30d0b-e6d3-429f-8f7a-b7788b79c86c None] [instance: 1541a197-9f80-4ee5-a7d6-08e591aa83fd] Terminating instance
2015-02-20 10:34:44.680 2214 WARNING nova.virt.libvirt.driver [-] [instance: 1541a197-9f80-4ee5-a7d6-08e591aa83fd] During wait destroy, instance disappeared.
2015-02-20 10:34:44.904 2214 INFO nova.virt.libvirt.driver [req-a4b30d0b-e6d3-429f-8f7a-b7788b79c86c None] [instance: 1541a197-9f80-4ee5-a7d6-08e591aa83fd] Deletion of /var/lib/nova/instances/1541a197-9f80-4ee5-a7d6-08e591aa83fd_del complete
2015-02-20 10:34:45.079 2214 INFO nova.compute.manager [req-a4b30d0b-e6d3-429f-8f7a-b7788b79c86c None] [instance: 1541a197-9f80-4ee5-a7d6-08e591aa83fd] Instance disappeared during terminate

And "nova list" will show:

| 1541a197-9f80-4ee5-a7d6-08e591aa83fd | test0    | ERROR   | deleting   | NOSTATE     |                                  |

It looks like running "nova reset-state <instance>" does *not* work around this problem, and at this point even getting Neutron configured correctly will not permit you to delete the instance.

The only way I've found to recover from this is this SQL script:

  http://blog.oddbit.com/assets/2015/01/17/delete-deleting-instances.sql

And that's not pretty...