Description of problem: When trying to deprevision cluster with OSP provider, got 409 error during delete router. Work around as following: delete all instances -> delete all trunks -> delete all ports created -> clear gateway for the router -> delete all interfaces for the router -> run destroy cluster(this time, router will not got error) ➜ installer git:(master) ✗ bin/openshift-install destroy cluster --dir ostest --log-level debug DEBUG OpenShift Installer unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75 DEBUG Deleting openstack containers DEBUG Deleting openstack routers DEBUG Deleting openstack ports DEBUG Deleting openstack security-groups DEBUG Deleting openstack subnets DEBUG Deleting openstack networks DEBUG Deleting openstack servers DEBUG Deleting openstack trunks DEBUG Deleting Trunk: ad1446bc-2259-447a-8b00-8cc054dca61e DEBUG Deleting Subnet: 3a3ba815-6699-4e83-abb7-b056392471d9 DEBUG Deleting network: b1b00531-b96a-4ebe-a1c4-9ecc22c135aa DEBUG Deleting Security Group: 28ec9c3d-4982-40bf-aa2f-082037dededc DEBUG Deleting Port: 3120abde-bb8a-49b6-9c39-5bc2ad4a4551 DEBUG Deleting Router: ca39ece0-0d4b-4c2b-b3b5-1e150f857361 DEBUG Deleting Trunk: ed7fc1e2-2640-4802-b272-2eed1e646e1b DEBUG Exiting deleting openstack subnets DEBUG Deleting Server: d8c7bb78-b061-4aa5-8a5c-bb55dc0fa122 DEBUG Exiting deleting openstack networks DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting openstack trunks DEBUG Deleting Server: 2eefff45-f031-4367-b491-c9134b5bad39 FATAL Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/routers/ca39ece0-0d4b-4c2b-b3b5-1e150f857361], but got 409 instead {"NeutronError": {"message": "Router ca39ece0-0d4b-4c2b-b3b5-1e150f857361 still has ports", "type": "RouterInUse", "detail": ""}} Version-Release number of the following components: > bin/openshift-install version bin/openshift-install unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75 How reproducible: Always Steps to Reproduce: 1. Create a cluster with osp cluster 2. Try to destroy the cluster 3. Actual results: ➜ installer git:(master) ✗ bin/openshift-install destroy cluster --dir ostest --log-level debug DEBUG OpenShift Installer unreleased-master-524-g13a752ea0fcae927cba6795782f87ffa332d5b75 DEBUG Deleting openstack containers DEBUG Deleting openstack routers DEBUG Deleting openstack ports DEBUG Deleting openstack security-groups DEBUG Deleting openstack subnets DEBUG Deleting openstack networks DEBUG Deleting openstack servers DEBUG Deleting openstack trunks DEBUG Deleting Trunk: ad1446bc-2259-447a-8b00-8cc054dca61e DEBUG Deleting Subnet: 3a3ba815-6699-4e83-abb7-b056392471d9 DEBUG Deleting network: b1b00531-b96a-4ebe-a1c4-9ecc22c135aa DEBUG Deleting Security Group: 28ec9c3d-4982-40bf-aa2f-082037dededc DEBUG Deleting Port: 3120abde-bb8a-49b6-9c39-5bc2ad4a4551 DEBUG Deleting Router: ca39ece0-0d4b-4c2b-b3b5-1e150f857361 DEBUG Deleting Trunk: ed7fc1e2-2640-4802-b272-2eed1e646e1b DEBUG Exiting deleting openstack subnets DEBUG Deleting Server: d8c7bb78-b061-4aa5-8a5c-bb55dc0fa122 DEBUG Exiting deleting openstack networks DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting openstack trunks DEBUG Deleting Server: 2eefff45-f031-4367-b491-c9134b5bad39 FATAL Expected HTTP response code [202 204] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/routers/ca39ece0-0d4b-4c2b-b3b5-1e150f857361], but got 409 instead {"NeutronError": {"message": "Router ca39ece0-0d4b-4c2b-b3b5-1e150f857361 still has ports", "type": "RouterInUse", "detail": ""}} Expected results: All resources should be delete successfully Additional info: Please attach logs from ansible-playbook with the -vvv flag
Currently the deprovision is using goroutines for each resources, this make router delete failed since router is still in use. https://github.com/openshift/installer/blob/master/pkg/destroy/openstack/openstack_deprovision.go#L67
We're working on cleaning the destroy process. You can track this work here: https://jira.coreos.com/browse/OSASINFRA-64
Another work around here is to use terraform directly to destroy all the resources which openshift-install created #> cp ./data/data/config.tf ./data/data/openstack #> ./terraform init -var-file ./ostest/terraform.tfvars -var-file ./ostest/terraform.openstack.auto.tfvars ./data/data/openstack #> ./terraform destroy -var-file ./ostest/terraform.tfvars -var-file ./ostest/terraform.openstack.auto.tfvars -state ./ostest/terraform.tfstate ./data/data/openstack
Checked with openshift-install v4.2.0-201907281300-dirty built from commit 6e2977c740853e842247b55c7cd08d9b350f3e93 release image registry.svc.ci.openshift.org/ocp/release@sha256:d81aeaf40eeee7a6f5c0da30540f3448b1f41f1d7ccb4b76e8e5bc445c929851 And this is not an issue any more.
Checked with 4.2.0-0.nightly-2019-07-31-162901, this issue has been fixed. ["./openshift-install", "destroy", "cluster", "--dir=/home/jenkins/workspace/Remove VMs/cucushift/workdir/install-dir", "--log-level=debug"] level=debug msg="OpenShift Installer v4.2.0-201907310219-dirty" level=debug msg="Built from commit 5e2ea734a1cb1fb790d0cb00bce57f5d59d2c26a" level=debug msg="Deleting openstack networks" level=debug msg="Deleting openstack subnet-pools" level=debug msg="Deleting openstack security-groups" level=debug msg="Deleting openstack routers" level=debug msg="Deleting openstack trunks" level=debug msg="Deleting openstack containers" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting openstack load balancers" level=debug msg="Deleting openstack servers" level=debug msg="Deleting openstack ports" level=debug msg="No suitable endpoint could be found in the service catalog." level=debug msg="Exiting deleting openstack load balancers" level=debug msg="goroutine deleteLoadBalancers complete" level=debug msg="Exiting deleting openstack subnet-pools" level=debug msg="goroutine deleteSubnetPools complete" level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db" level=debug msg="Deleting Trunk: 2d7c25ed-1880-4f5c-9939-be631063d64d" level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d" level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65" level=debug msg="Deleting Port: 2448472b-0231-4275-a9d2-bb9a194927ba" level=debug msg="Deleting Trunk: 53f2a59c-41d2-454b-aff4-6e98e4398012" level=debug msg="Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/52d8741a-8955-48d9-bf82-67c224f3ef65], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group 52d8741a-8955-48d9-bf82-67c224f3ef65 in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack security-groups" level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack networks" level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack subnets" level=debug msg="Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/2448472b-0231-4275-a9d2-bb9a194927ba], but got 409 instead\n{\"NeutronError\": {\"message\": \"Port 2448472b-0231-4275-a9d2-bb9a194927ba is currently a parent port for trunk 718af8d5-3c17-45df-b020-63c8cc26bee7.\", \"type\": \"PortInUseAsTrunkParent\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack ports" level=debug msg="Deleting Trunk: 718af8d5-3c17-45df-b020-63c8cc26bee7" level=debug msg="Deleting Server: 5f197107-0e6a-4630-854f-4d72765f537b" level=debug msg="Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/718af8d5-3c17-45df-b020-63c8cc26bee7], but got 409 instead\n{\"NeutronError\": {\"message\": \"Trunk 718af8d5-3c17-45df-b020-63c8cc26bee7 is currently in use.\", \"type\": \"TrunkInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack trunks" level=debug msg="Deleting Server: 7d6a4d12-a98f-4b9c-baba-0458d4d27e1c" level=debug msg="Deleting Server: aea2be71-32fd-4e28-a100-e4186ed79e58" level=debug msg="Deleting Server: 4ea73981-37a7-45b6-af0f-03f9c021c6fa" level=debug msg="Deleting Server: dacae271-1f56-483e-828f-bfe5ede1e755" level=debug msg="Exiting deleting openstack servers" level=debug msg="Deleting openstack security-groups" level=debug msg="Deleting openstack networks" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting openstack ports" level=debug msg="Removing Subnet 3c395faa-d709-4cb0-b5ea-84ea87e8b9fb from Router 07793c20-6943-40d4-8201-5ec69116c516\n" level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d" level=debug msg="Deleting openstack trunks" level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65" level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db" level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack networks" level=debug msg="Deleting Port: 2448472b-0231-4275-a9d2-bb9a194927ba" level=debug msg="Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/52d8741a-8955-48d9-bf82-67c224f3ef65], but got 409 instead\n{\"NeutronError\": {\"message\": \"Security Group 52d8741a-8955-48d9-bf82-67c224f3ef65 in use.\", \"type\": \"SecurityGroupInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack security-groups" level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack subnets" level=debug msg="Deleting Trunk: 718af8d5-3c17-45df-b020-63c8cc26bee7" level=debug msg="Exiting deleting openstack trunks" level=debug msg="Deleting openstack servers" level=debug msg="Deleting Port: 427958e0-1af0-490a-b929-6f552816b108" level=debug msg="Deleting Port: 9a62a991-0c23-4233-a05b-3e6ce35630c7" level=debug msg="Exiting deleting openstack servers" level=debug msg="goroutine deleteServers complete" level=debug msg="Deleting Port: a8b9b253-958a-4e0a-ba14-41a1bea6dbfd" level=debug msg="Deleting Port: b64cb03b-328a-4444-aac7-e5e2f2a2f672" level=debug msg="Deleting openstack networks" level=debug msg="Exiting deleting openstack ports" level=debug msg="Deleting openstack security-groups" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting object: bootstrap.ign\n" level=debug msg="Deleting object: load-balancer.ign\n" level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d" level=debug msg="Deleting container: wjiangosp0801a-sjzcc\n" level=debug msg="Deleting openstack trunks" level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db" level=debug msg="Deleting Security Group: 52d8741a-8955-48d9-bf82-67c224f3ef65" level=debug msg="Exiting deleting openstack containers" level=debug msg="goroutine deleteContainers complete" level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack networks" level=debug msg="Exiting deleting openstack trunks" level=debug msg="goroutine deleteTrunks complete" level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack subnets" level=debug msg="Deleting Security Group: abcd1421-405f-45f0-bdb7-9c89768b332c" level=debug msg="Deleting Security Group: b0c8bd02-27ff-4fba-8cf4-3001743ed571" level=debug msg="Exiting deleting openstack security-groups" level=debug msg="Removing Subnet 04252c61-5baa-4806-9e59-513db6fb18db from Router 07793c20-6943-40d4-8201-5ec69116c516\n" level=debug msg="Deleting openstack ports" level=debug msg="Exiting deleting openstack ports" level=debug msg="goroutine deletePorts complete" level=debug msg="Deleting openstack networks" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db" level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d" level=debug msg="Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/04252c61-5baa-4806-9e59-513db6fb18db], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on subnet 04252c61-5baa-4806-9e59-513db6fb18db: One or more ports have an IP allocation from this subnet.\", \"type\": \"SubnetInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack subnets" level=debug msg="Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/d3593314-6e04-4ce1-aba5-f1312bd0f72d], but got 409 instead\n{\"NeutronError\": {\"message\": \"Unable to complete operation on network d3593314-6e04-4ce1-aba5-f1312bd0f72d. There are one or more ports still in use on the network.\", \"type\": \"NetworkInUse\", \"detail\": \"\"}}" level=debug msg="Exiting deleting openstack networks" level=debug msg="Deleting openstack security-groups" level=debug msg="Exiting deleting openstack security-groups" level=debug msg="goroutine deleteSecurityGroups complete" level=debug msg="Deleting Router: 07793c20-6943-40d4-8201-5ec69116c516\n" level=debug msg="Exiting deleting openstack routers" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting openstack networks" level=debug msg="Deleting Subnet: 04252c61-5baa-4806-9e59-513db6fb18db" level=debug msg="Deleting network: d3593314-6e04-4ce1-aba5-f1312bd0f72d" level=debug msg="Deleting openstack routers" level=debug msg="Exiting deleting openstack routers" level=debug msg="goroutine deleteRouters complete" level=debug msg="Deleting Subnet: 3c395faa-d709-4cb0-b5ea-84ea87e8b9fb" level=debug msg="Deleting Subnet failed: Resource not found" level=debug msg="Exiting deleting openstack subnets" level=debug msg="Exiting deleting openstack networks" level=debug msg="Deleting openstack subnets" level=debug msg="Deleting openstack networks" level=debug msg="Exiting deleting openstack subnets" level=debug msg="goroutine deleteSubnets complete" level=debug msg="Exiting deleting openstack networks" level=debug msg="goroutine deleteNetworks complete" level=debug msg="Purging asset \"Terraform Variables\" from disk" level=debug msg="Purging asset \"Kubeconfig Admin Client\" from disk" level=debug msg="Purging asset \"Kubeadmin Password\" from disk" level=debug msg="Purging asset \"Certificate (journal-gatewayd)\" from disk" level=debug msg="Purging asset \"Metadata\" from disk" waiting for operation up to 36000 seconds..
This is an older bug and many things have been fixed since first identified. Change status to ON_QA for validation this no longer happens.
Verified on 4.2.0-0.nightly-2019-08-06-062019
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922