Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1139752

Summary: unable to delete two servers previously created
Product: Red Hat OpenStack Reporter: Kambiz Aghaiepour <kambiz>
Component: openstack-novaAssignee: Russell Bryant <rbryant>
Status: CLOSED NOTABUG QA Contact: Ami Jeain <ajeain>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 7)CC: dasmith, ndipanov, sgordon, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-08 15:49:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nova compute log where intance was running none

Description Kambiz Aghaiepour 2014-09-09 14:47:06 UTC 
Description of problem:

There are two phantom instances that I cannot delete.  A large number of VMs were launched (attempted to be launched) via rally (attempted to launch 1000 VMs, but openstack seemed to have a meltdown around 500 instances).  After making adjustments to some neutron settings, and attempted to get control of the environment back (as nova commands were timing out), managed to get back in to delete the instances left around by rally, in order to attempt the tests again.  However, two instances don't seem to go away (deleted close to 500 other instances though).  The environment was deployed using latest puddle staypuft (also see :  https://bugzilla.redhat.com/show_bug.cgi?id=1139732 ).

Here is what I see:

[root@macb8ca3a60bde8 ~(openstack_admin)]# nova list --all-tenants
+--------------------------------------+-----------------------------------+--------+------------+-------------+----------------------------------+
| ID                                   | Name                              | Status | Task State | Power State | Networks                         |
+--------------------------------------+-----------------------------------+--------+------------+-------------+----------------------------------+
| ddbb56a1-b378-43c3-af59-4ed8ea3d312c | F20                               | ACTIVE | -          | Running     | internal=192.168.0.2, 10.1.251.2 |
| c6e1ff68-0614-40c6-9ca0-0603ac76ba69 | rally_novaserver_cmesupvuevffuamq | ERROR  | deleting   | Running     | internal=192.168.1.20            |
| 78b3da4f-3db7-40e0-bbb8-04101829c8f0 | rally_novaserver_exxbymqewtatxtcq | ERROR  | deleting   | Running     | internal=192.168.1.78            |
+--------------------------------------+-----------------------------------+--------+------------+-------------+----------------------------------+

Attempting to reset the instances using reset-state, and deleting doesn't seem to have an effect.

How reproducible:

Uncertain if this is reproducible, as I have not attempted to rerun the rally scenario that broke the environment until I can ensure everything else is OK.  The rally scenario is attached to BZ found here:  https://bugzilla.redhat.com/show_bug.cgi?id=1139732
Actual results:
cannot delete the two instances lingering.

Expected results:
I should be able to delete the instances.

Additional info:
Comment 2 Kambiz Aghaiepour 2014-09-09 15:04:00 UTC 
Update:

 After restarting openstack on the compute nodes reportedly running the instances (though this was previously done), it appears the instances have now cleared up and are no longer showing up under the all-tenant listing.

I will attempt several more rally invocations to see if I can reproduce this problem.

Meanwhile, I have attached the nova-compute.log.gz file from the compute node which lists the errors being encountered during the time the delete operation was failing.  If the log file could indicate an explanation of what might have caused the error condition please let me know.
Comment 3 Kambiz Aghaiepour 2014-09-09 15:05:01 UTC 
Created attachment 935785 [details]
nova compute log where intance was running
Comment 4 Dan Smith 2014-10-08 15:49:23 UTC 
Looks like neutron is unhappy, preventing the delete of the port, which is preventing the delete of the instance:

2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api Traceback (most recent call last):
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 428, in deallocate_for_instance
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api     neutron.delete_port(port)
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api     ret = self.function(instance, *args, **kwargs)
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 326, in delete_port
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api     return self.delete(self.port_path % (port))
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1232, in delete
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api     headers=headers, params=params)
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1228, in retry_request
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api     raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
2014-09-05 15:10:57.936 2978 TRACE nova.network.neutronv2.api ConnectionFailed: Connection to neutron failed: Maximum attempts reached


Before this, there was a neutron internal server error, which is probably causing some hung connections and/or other madness.