Bug 1300693

Summary: neutron ports does not delete when delete instance on error status
Product: Red Hat OpenStack Reporter: Eran Kuris <ekuris>
Component: openstack-novaAssignee: Eoghan Glynn <eglynn>
Status: CLOSED UPSTREAM QA Contact: nlevinki <nlevinki>
Severity: low Docs Contact:
Priority: low    
Version: 8.0 (Liberty)CC: amuller, berrange, chrisw, dasmith, eglynn, ekuris, kchamart, mlopes, ndipanov, nyechiel, sbauza, sferdjao, sgordon, vromanso, yeylon
Target Milestone: ---   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-26 13:53:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1300680    
Bug Blocks:    
Attachments:
Description Flags
delete port none

Description Eran Kuris 2016-01-21 13:22:22 UTC
Created attachment 1116925 [details]
delete port

Description of problem:
On OVS-DPDK setup 1 controller & 1 compute with data tenant/network type vlan
I  delete instance that did not success to boot well and it is in error status 
when I delete this vm I expected that his neutron port will delete too from neutron port-list and from ovs-vsvctl
attached all log 
nic type: Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter 
driver: ixgbe
Version-Release number of selected component (if applicable):
[root@puma48 ~]# rpm -qa |grep neutro
openstack-neutron-openvswitch-7.0.1-6.el7ost.noarch
python-neutron-7.0.1-6.el7ost.noarch
openstack-neutron-common-7.0.1-6.el7ost.noarch
python-neutronclient-3.1.0-1.el7ost.noarch
openstack-neutron-7.0.1-6.el7ost.noarch
[root@puma48 ~]# rpm -qa |grep dpd
openvswitch-dpdk-2.4.0-0.10346.git97bab959.2.el7.x86_64
dpdk-2.1.0-5.el7.x86_64

How reproducible:
always 

Steps to Reproduce:
1.https://docs.google.com/document/d/1K_ku6_08ooq46dFLiE7fAJ0ByFdPCb0W_q6kKqF3Y0o/edit
2.create 2 vm's  - one of them will be in error status (https://bugzilla.redhat.com/show_bug.cgi?id=1300680)
3. check neutron port-list
4. remove vm in erro status 
5. check again neutron port-list the port didnt deleted .
Actual results:


Expected results:


Additional info:

Comment 1 Assaf Muller 2016-01-21 15:15:23 UTC
I suspect that this has nothing to do with DPDK but I want to make sure. Can you check this?

For example:
0) Install regular, non-DPDK openstack
1) Shut down OVS agent
2) Wait until it shows as dead on neutron agent-list (Will take 75 seconds)
3) Spawn a VM, it should enter error state
4) Observe it has a Neutron port
5) Delete the VM
6) Check if the Neutron port was deleted or not

It's possible that the behavior will be different if when you create the VM you:
1) First create a port and pass the port_id to the nova boot command
2) Or just specify a network_id when you boot the VM

Comment 3 Eran Kuris 2016-01-24 07:04:05 UTC
Assaf , 
I did what you ask me to run , In this scenario neutron port does not created when VM failed to boot (Error status). 

I also tried to spawn a VM and make it to enter to error status ,
when OVS-agent is enabled and than delete it.I found that the port is still exist.
This means that the issue is not related to DPDK

Comment 4 Assaf Muller 2016-01-24 15:54:23 UTC
(In reply to Eran Kuris from comment #3)
> Assaf , 
> I did what you ask me to run , In this scenario neutron port does not
> created when VM failed to boot (Error status). 
> 
> I also tried to spawn a VM and make it to enter to error status ,
> when OVS-agent is enabled and than delete it.I found that the port is still
> exist.
> This means that the issue is not related to DPDK

There's two ways for Nova to interact with Neutron.

1) neutron port-create, pass that port ID to nova boot command
2) nova boot and specify a network-id, and then Nova creates the port for you

Can you say if this bug happens with both scenarios 1 and 2?

Comment 5 Assaf Muller 2016-01-24 15:54:49 UTC
Switching to Nova component.

Comment 6 Eran Kuris 2016-01-25 05:25:16 UTC
yes happens in 2 scenarios

Comment 7 Nikola Dipanov 2016-02-26 13:53:16 UTC
(In reply to Assaf Muller from comment #4)
> (In reply to Eran Kuris from comment #3)
> > Assaf , 
> > I did what you ask me to run , In this scenario neutron port does not
> > created when VM failed to boot (Error status). 
> > 
> > I also tried to spawn a VM and make it to enter to error status ,
> > when OVS-agent is enabled and than delete it.I found that the port is still
> > exist.
> > This means that the issue is not related to DPDK
> 
> There's two ways for Nova to interact with Neutron.
> 
> 1) neutron port-create, pass that port ID to nova boot command
> 2) nova boot and specify a network-id, and then Nova creates the port for you
> 
> Can you say if this bug happens with both scenarios 1 and 2?

So in case this happens in scenario 1 above - that's the intended behavior. Nova will not delete ports that it did not itself create.

https://review.openstack.org/#/c/126309/

In scenario 2) (as can be seen from the above change), Nova should clean up the ports it did create unless there was some problem with delete.

This bug does not have enough information for me to make a guess as to what happened.

A plausible theory would be that in case of reschedules that are caused by failures to allocate networks, since neutron is broken, the instance goes to ERROR state after retries, meaning that the instance is not registered on any host (can be confirmed by running nova show).

In that case - when deleting the instance, Nova will only do a DB delete and never call out to the compute host or Neutron.

Normally I would ask the reporter for more details on this, but this seems like a minor nuance that has an easy workaround - manually delete ports (please correct me if my assumptions are wrong here). It is also likely the problem on upstream master as well, so considering the overall impact - I would recommend that the reporter tries to reproduce this with upstream master and reports the bug upstream (https://bugs.launchpad.net/nova).

Comment 8 Martin Lopes 2017-07-24 00:33:29 UTC
Note: the google doc guide mentioned in the description has been published here:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/network_functions_virtualization_configuration_guide/