Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1686271

Summary:	[OSP] Openshift-install got 409 for router deletion during destroy cluster
Product:	OpenShift Container Platform	Reporter:	weiwei jiang <wjiang>
Component:	Installer	Assignee:	Eric Duen <eduen>
Installer sub component:	openshift-installer	QA Contact:	weiwei jiang <wjiang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	bleanhar, dsanzmor
Version:	4.1.0
Target Milestone:	---
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	CLOSED / CURRENTRELEASE	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 06:27:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description weiwei jiang 2019-03-07 07:00:11 UTC

Description of problem:
When trying to deprevision cluster with OSP provider, got 409 error during delete router. 

Work around as following:
delete all instances -> delete all trunks -> delete all ports created -> clear gateway for the router -> delete all interfaces for the router -> run destroy cluster(this time, router will not got error) 

➜  installer git:(master) ✗ bin/openshift-install destroy cluster --dir ostest --log-level debug
DEBUG OpenShift Installer unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75 
DEBUG Deleting openstack containers                
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack servers                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting Trunk: ad1446bc-2259-447a-8b00-8cc054dca61e 
DEBUG Deleting Subnet: 3a3ba815-6699-4e83-abb7-b056392471d9 
DEBUG Deleting network: b1b00531-b96a-4ebe-a1c4-9ecc22c135aa 
DEBUG Deleting Security Group: 28ec9c3d-4982-40bf-aa2f-082037dededc 
DEBUG Deleting Port: 3120abde-bb8a-49b6-9c39-5bc2ad4a4551 
DEBUG Deleting Router: ca39ece0-0d4b-4c2b-b3b5-1e150f857361 
DEBUG Deleting Trunk: ed7fc1e2-2640-4802-b272-2eed1e646e1b 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Server: d8c7bb78-b061-4aa5-8a5c-bb55dc0fa122 
DEBUG Exiting deleting openstack networks          
DEBUG Exiting deleting openstack security-groups   
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting Server: 2eefff45-f031-4367-b491-c9134b5bad39 
FATAL Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/routers/ca39ece0-0d4b-4c2b-b3b5-1e150f857361], but got 409 instead
{"NeutronError": {"message": "Router ca39ece0-0d4b-4c2b-b3b5-1e150f857361 still has ports", "type": "RouterInUse", "detail": ""}} 


Version-Release number of the following components:
> bin/openshift-install version                                       
bin/openshift-install unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75

How reproducible:
Always

Steps to Reproduce:
1. Create a cluster with osp cluster
2. Try to destroy the cluster
3.

Actual results:
➜  installer git:(master) ✗ bin/openshift-install destroy cluster --dir ostest --log-level debug
DEBUG OpenShift Installer unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75 
DEBUG Deleting openstack containers                
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack servers                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting Trunk: ad1446bc-2259-447a-8b00-8cc054dca61e 
DEBUG Deleting Subnet: 3a3ba815-6699-4e83-abb7-b056392471d9 
DEBUG Deleting network: b1b00531-b96a-4ebe-a1c4-9ecc22c135aa 
DEBUG Deleting Security Group: 28ec9c3d-4982-40bf-aa2f-082037dededc 
DEBUG Deleting Port: 3120abde-bb8a-49b6-9c39-5bc2ad4a4551 
DEBUG Deleting Router: ca39ece0-0d4b-4c2b-b3b5-1e150f857361 
DEBUG Deleting Trunk: ed7fc1e2-2640-4802-b272-2eed1e646e1b 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Server: d8c7bb78-b061-4aa5-8a5c-bb55dc0fa122 
DEBUG Exiting deleting openstack networks          
DEBUG Exiting deleting openstack security-groups   
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting Server: 2eefff45-f031-4367-b491-c9134b5bad39 
FATAL Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/routers/ca39ece0-0d4b-4c2b-b3b5-1e150f857361], but got 409 instead
{"NeutronError": {"message": "Router ca39ece0-0d4b-4c2b-b3b5-1e150f857361 still has ports", "type": "RouterInUse", "detail": ""}} 


Expected results:
All resources should be delete successfully

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 weiwei jiang 2019-03-08 08:46:21 UTC

Currently the deprovision is using goroutines for each resources, this make router delete failed since router is still in use.

https://github.com/openshift/installer/blob/master/pkg/destroy/openstack/openstack_deprovision.go#L67

Comment 2 Flavio Percoco 2019-03-13 11:40:17 UTC

We're working on cleaning the destroy process. You can track this work here: https://jira.coreos.com/browse/OSASINFRA-64

Comment 3 weiwei jiang 2019-03-25 08:51:51 UTC

Another work around here is to use terraform directly to destroy all the resources which openshift-install created

#> cp ./data/data/config.tf ./data/data/openstack
#> ./terraform init -var-file ./ostest/terraform.tfvars -var-file ./ostest/terraform.openstack.auto.tfvars ./data/data/openstack
#> ./terraform destroy -var-file ./ostest/terraform.tfvars -var-file ./ostest/terraform.openstack.auto.tfvars -state ./ostest/terraform.tfstate  ./data/data/openstack

Comment 6 weiwei jiang 2019-07-29 05:04:55 UTC

Checked with openshift-install v4.2.0-201907281300-dirty
built from commit 6e2977c740853e842247b55c7cd08d9b350f3e93
release image registry.svc.ci.openshift.org/ocp/release@sha256:d81aeaf40eeee7a6f5c0da30540f3448b1f41f1d7ccb4b76e8e5bc445c929851

And this is not an issue any more.

Comment 7 weiwei jiang 2019-08-01 02:43:32 UTC

Checked with 4.2.0-0.nightly-2019-07-31-162901, this issue has been fixed.

["./openshift-install", "destroy", "cluster", "--dir=/home/jenkins/workspace/Remove VMs/cucushift/workdir/install-dir", "--log-level=debug"]
level=debug msg="OpenShift Installer v4.2.0-201907310219-dirty"
level=debug msg="Built from commit 5e2ea734a1cb1fb790d0cb00bce57f5d59d2c26a"
level=debug msg="Deleting openstack networks"
level=debug msg="Deleting openstack subnet-pools"
level=debug msg="Deleting openstack security-groups"
level=debug msg="Deleting openstack routers"
level=debug msg="Deleting openstack trunks"
level=debug msg="Deleting openstack containers"
level=debug msg="Deleting openstack subnets"
level=debug msg="Deleting openstack load balancers"
level=debug msg="Deleting openstack servers"
level=debug msg="Deleting openstack ports"
level=debug msg="No suitable endpoint could be found in the service catalog."
level=debug msg="Exiting deleting openstack load balancers"
level=debug msg="goroutine deleteLoadBalancers complete"

level=debug msg="Exiting deleting openstack subnet-pools"
level=debug msg="goroutine deleteSubnetPools complete"
level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db"
level=debug msg="Deleting Trunk: 2d7c25ed-1880-4f5c-9939-be631063d64d"
level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d"
level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65"
level=debug msg="Deleting Port: 2448472b-0231-4275-a9d2-bb9a194927ba"
level=debug msg="Deleting Trunk: 53f2a59c-41d2-454b-aff4-6e98e4398012"
level=debug msg="Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/52d8741a-8955-48d9-bf82-67c224f3ef65], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group 52d8741a-8955-48d9-bf82-67c224f3ef65 in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack security-groups"
level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack networks"
level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack subnets"
level=debug msg="Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/2448472b-0231-4275-a9d2-bb9a194927ba], but got 409 instead\n{\"NeutronError\": {\"message\": \"Port 2448472b-0231-4275-a9d2-bb9a194927ba is currently a parent port for trunk 718af8d5-3c17-45df-b020-63c8cc26bee7.\", \"type\": \"PortInUseAsTrunkParent\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack ports"
level=debug msg="Deleting Trunk: 718af8d5-3c17-45df-b020-63c8cc26bee7"
level=debug msg="Deleting Server: 5f197107-0e6a-4630-854f-4d72765f537b"

level=debug msg="Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/718af8d5-3c17-45df-b020-63c8cc26bee7], but got 409 instead\n{\"NeutronError\": {\"message\": \"Trunk 718af8d5-3c17-45df-b020-63c8cc26bee7 is currently in use.\", \"type\": \"TrunkInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack trunks"
level=debug msg="Deleting Server: 7d6a4d12-a98f-4b9c-baba-0458d4d27e1c"

level=debug msg="Deleting Server: aea2be71-32fd-4e28-a100-e4186ed79e58"
level=debug msg="Deleting Server: 4ea73981-37a7-45b6-af0f-03f9c021c6fa"
level=debug msg="Deleting Server: dacae271-1f56-483e-828f-bfe5ede1e755"

level=debug msg="Exiting deleting openstack servers"

level=debug msg="Deleting openstack security-groups"
level=debug msg="Deleting openstack networks"
level=debug msg="Deleting openstack subnets"

level=debug msg="Deleting openstack ports"
level=debug msg="Removing Subnet 3c395faa-d709-4cb0-b5ea-84ea87e8b9fb from Router 07793c20-6943-40d4-8201-5ec69116c516\n"
level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d"
level=debug msg="Deleting openstack trunks"
level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65"
level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db"
level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack networks"
level=debug msg="Deleting Port: 2448472b-0231-4275-a9d2-bb9a194927ba"
level=debug msg="Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/52d8741a-8955-48d9-bf82-67c224f3ef65], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group 52d8741a-8955-48d9-bf82-67c224f3ef65 in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack security-groups"
level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack subnets"
level=debug msg="Deleting Trunk: 718af8d5-3c17-45df-b020-63c8cc26bee7"

level=debug msg="Exiting deleting openstack trunks"

level=debug msg="Deleting openstack servers"
level=debug msg="Deleting Port: 427958e0-1af0-490a-b929-6f552816b108"

level=debug msg="Deleting Port: 9a62a991-0c23-4233-a05b-3e6ce35630c7"
level=debug msg="Exiting deleting openstack servers"
level=debug msg="goroutine deleteServers complete"

level=debug msg="Deleting Port: a8b9b253-958a-4e0a-ba14-41a1bea6dbfd"

level=debug msg="Deleting Port: b64cb03b-328a-4444-aac7-e5e2f2a2f672"

level=debug msg="Deleting openstack networks"
level=debug msg="Exiting deleting openstack ports"
level=debug msg="Deleting openstack security-groups"
level=debug msg="Deleting openstack subnets"
level=debug msg="Deleting object: bootstrap.ign\n"
level=debug msg="Deleting object: load-balancer.ign\n"
level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d"
level=debug msg="Deleting container: wjiangosp0801a-sjzcc\n"
level=debug msg="Deleting openstack trunks"

level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db"
level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65"
level=debug msg="Exiting deleting openstack containers"
level=debug msg="goroutine deleteContainers complete"
level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack networks"
level=debug msg="Exiting deleting openstack trunks"
level=debug msg="goroutine deleteTrunks complete"
level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack subnets"

level=debug msg="Deleting Security Group: abcd1421-405f-45f0-bdb7-9c89768b332c"

level=debug msg="Deleting Security Group: b0c8bd02-27ff-4fba-8cf4-3001743ed571"
level=debug msg="Exiting deleting openstack security-groups"

level=debug msg="Removing Subnet 04252c61-5baa-4806-9e59-513db6fb18db from Router 07793c20-6943-40d4-8201-5ec69116c516\n"

level=debug msg="Deleting openstack ports"

level=debug msg="Exiting deleting openstack ports"
level=debug msg="goroutine deletePorts complete"

level=debug msg="Deleting openstack networks"
level=debug msg="Deleting openstack subnets"
level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db"

level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d"
level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack subnets"
level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}"
level=debug msg="Exiting deleting openstack networks"
level=debug msg="Deleting openstack security-groups"

level=debug msg="Exiting deleting openstack security-groups"
level=debug msg="goroutine deleteSecurityGroups complete"

level=debug msg="Deleting Router: 07793c20-6943-40d4-8201-5ec69116c516\n"

level=debug msg="Exiting deleting openstack routers"

level=debug msg="Deleting openstack subnets"
level=debug msg="Deleting openstack networks"

level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db"
level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d"
level=debug msg="Deleting openstack routers"

level=debug msg="Exiting deleting openstack routers"
level=debug msg="goroutine deleteRouters complete"

level=debug msg="Deleting Subnet: 3c395faa-d709-4cb0-b5ea-84ea87e8b9fb"
level=debug msg="Deleting Subnet failed: Resource not found"
level=debug msg="Exiting deleting openstack subnets"

level=debug msg="Exiting deleting openstack networks"

level=debug msg="Deleting openstack subnets"
level=debug msg="Deleting openstack networks"
level=debug msg="Exiting deleting openstack subnets"
level=debug msg="goroutine deleteSubnets complete"
level=debug msg="Exiting deleting openstack networks"
level=debug msg="goroutine deleteNetworks complete"
level=debug msg="Purging asset \"Terraform Variables\" from disk"
level=debug msg="Purging asset \"Kubeconfig Admin Client\" from disk"
level=debug msg="Purging asset \"Kubeadmin Password\" from disk"
level=debug msg="Purging asset \"Certificate (journal-gatewayd)\" from disk"
level=debug msg="Purging asset \"Metadata\" from disk"
waiting for operation up to 36000 seconds..

Comment 8 Eric Duen 2019-08-05 18:56:06 UTC

This is an older bug and many things have been fixed since first identified.  Change status to ON_QA for validation this no longer happens.

Comment 9 David Sanz 2019-08-06 07:51:48 UTC

Verified on 4.2.0-0.nightly-2019-08-06-062019

Comment 10 errata-xmlrpc 2019-10-16 06:27:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922