Bug 1389185

Summary: fails to delete OpenStack network with Conflict (HTTP 409) error
Product: [Retired] Beaker Reporter: Dan Callaghan <dcallagh>
Component: generalAssignee: matt jia <mjia>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: dcallagh, dowang, mjia, rjoost
Target Milestone: 24.0Keywords: Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-21 18:50:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan Callaghan 2016-10-27 06:41:07 UTC
Description of problem:
I noticed this failure in the beakerd logs with some of my recipes which ran on OpenStack.

Version-Release number of selected component (if applicable):
unreleased develop ade77d7 

How reproducible:
not sure

Actual results:
bkr.server.model.scheduler DEBUG Releasing vm bfc35fa6-3a44-468e-b4e3-1bc3546cd835 for recipe 16747
bkr.server.model.scheduler ERROR Failed to destroy vm bfc35fa6-3a44-468e-b4e3-1bc3546cd835, leaked!
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/bkr/server/model/scheduler.py", line 4143, in release
     manager.destroy_vm(self)
   File "/usr/lib/python2.6/site-packages/bkr/server/dynamic_virt.py", line 143, in destroy_vm
     self.neutronclient.delete_network(str(vm.network_id))
[...]
 Conflict: Conflict (HTTP 409) (Request-ID: req-5527689c-6f31-475e-bb32-265137692352)

Comment 1 Dan Callaghan 2016-10-27 06:44:06 UTC
I noticed that the leftover network still had some ports attached -- not sure if that's expected, but it seems from the API docs that we actually need to find all remaining ports on the network and delete those, before we can delete the network.

Based on that, my theory about steps to reproduce (not sure about this):
1. Submit a job which will run on OpenStack (x86_64, single host, etc)
2. Wait for the instance to be created
3. When the instance is booting into iPXE and trying to download the kernel and initrd, hit Ctrl-Alt-Delete in the Console tab of Horizon
(note: in my case, iPXE was getting stuck downloading the kernel, that seems to be a separate issue but it probably helps to reproduce this)
4. The network will have a number of ports which say Attached Device is "network:dhcp", these will remain even after the instance itself is deleted.

Comment 2 matt jia 2016-11-02 04:44:25 UTC
(In reply to Dan Callaghan from comment #1)
> I noticed that the leftover network still had some ports attached -- not
> sure if that's expected, but it seems from the API docs that we actually
> need to find all remaining ports on the network and delete those, before we
> can delete the network. 
> Based on that, my theory about steps to reproduce (not sure about this):
> 1. Submit a job which will run on OpenStack (x86_64, single host, etc)
> 2. Wait for the instance to be created
> 3. When the instance is booting into iPXE and trying to download the kernel
> and initrd, hit Ctrl-Alt-Delete in the Console tab of Horizon
> (note: in my case, iPXE was getting stuck downloading the kernel, that seems
> to be a separate issue but it probably helps to reproduce this)
> 4. The network will have a number of ports which say Attached Device is
> "network:dhcp", these will remain even after the instance itself is deleted.

There should be 5 ports created when an instance is built. One is for the instance, one is for the interface of the router and the other three ports for the private network. The ports in the leftover networks seem okay for me as I can see the port of the instance and the port of the interface have been deleted. It seems like the problem here is the port of the instance was not released yet when deleting the network, so I think we should wait until the instance is deleted and the port is released before deleting the network.

Comment 3 matt jia 2016-11-02 05:56:56 UTC
On Gerrit:

  https://gerrit.beaker-project.org/#/c/5391/

Comment 6 Dan Callaghan 2017-02-21 18:50:22 UTC
Beaker 24.0 has been released.