I observed this behavior while destroying an orphaned cluster on MOC. The cluster was crxf228b-3409b-7hsk2, created around 2021-01-14T21:52:49.000000. Destroy was looping over the deletion of volumes. However it was failing because they were still attached to running nodes. For some reason, the destroy module assumed the servers were deleted while they were not (DEBUG goroutine deleteServers complete). Destroying crxf228b-3409b-7hsk2 Destroying cluster using openshift-install DEBUG OpenShift Installer unreleased-master-4123-g69f0bbc18e8c6b1a6e278c54efa2def9b210033a DEBUG Built from commit 69f0bbc18e8c6b1a6e278c54efa2def9b210033a DEBUG Removing interfaces from custom router DEBUG Deleting openstack subnets DEBUG Deleting openstack base image DEBUG Deleting openstack containers DEBUG Deleting openstack load balancers DEBUG Deleting openstack routers DEBUG Deleting openstack servers DEBUG Deleting openstack networks DEBUG Deleting OpenStack volumes DEBUG Deleting openstack ports DEBUG Deleting openstack floating ips DEBUG Deleting openstack server groups DEBUG Deleting openstack subnet-pools DEBUG Deleting openstack trunks DEBUG Deleting openstack security-groups DEBUG Exiting deleting openstack base image DEBUG Deleting network: "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" DEBUG Exiting deleting openstack trunks DEBUG Deleting Server Group "eb65b20e-06ec-41bf-b406-f0ff528d3bb7" DEBUG Deleting Subnet: "04dafe23-baec-4f1d-8a25-ee3c4717d647" DEBUG Exiting deleting openstack floating ips DEBUG Deleting Security Group: "5fd19a64-077c-4c2c-b7d9-5cea9421bc35" DEBUG Exiting deleting openstack load balancers DEBUG Exiting removal of interfaces from custom router DEBUG goroutine deleteImages complete DEBUG goroutine deleteTrunks complete DEBUG goroutine deleteFloatingIPs complete DEBUG goroutine deleteLoadBalancers complete DEBUG Exiting deleting openstack subnet-pools DEBUG goroutine deleteSubnetPools complete DEBUG Deleting Network "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/c8b995ab-ff35-4da5-8aa6-7ef95f21aa00], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on network c8b995ab-ff35-4da5-8aa6-7ef95f21aa00. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} DEBUG Exiting deleting openstack networks DEBUG Exiting deleting openstack server groups DEBUG Deleting Port "080f8fb2-6bff-4e6b-893b-456599186ec2" DEBUG Deleting Subnet "04dafe23-baec-4f1d-8a25-ee3c4717d647" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/04dafe23-baec-4f1d-8a25-ee3c4717d647], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on subnet 04dafe23-baec-4f1d-8a25-ee3c4717d647: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Deleting Security Group "5fd19a64-077c-4c2c-b7d9-5cea9421bc35" failed with error: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/5fd19a64-077c-4c2c-b7d9-5cea9421bc35], but got 409 instead DEBUG {"NeutronError": {"message": "Security Group 5fd19a64-077c-4c2c-b7d9-5cea9421bc35 in use.", "type": "SecurityGroupInUse", "detail": ""}} DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting openstack routers DEBUG goroutine deleteRouters complete DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" DEBUG Dissociating Floating IP "5bd052b3-cad4-4fdf-a918-4cd52a899cfc" DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/95f84007-e2dc-43e6-bdbf-3c67e1832230], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} DEBUG Exiting deleting OpenStack volumes DEBUG Exiting deleting openstack containers DEBUG goroutine deleteContainers complete DEBUG Exiting deleting openstack servers DEBUG goroutine deleteServers complete DEBUG Deleting Port "1412cc26-d336-4c67-a0c1-f72e17a2fa24" DEBUG Deleting Port "2e3bb6b5-97f2-4d9b-a07d-c1723e94f00b" DEBUG Deleting Port "3bd4c43a-24dd-46b2-8bfd-7c26d898dc31" DEBUG Deleting Port "3bdb130b-ae49-4362-8832-3ba720728f01" DEBUG Dissociating Floating IP "4d9efb18-9656-4baf-a9b2-f726d5d299af" DEBUG Deleting Port "5f87f663-9aa6-4b8a-a3aa-3d7b6095ee42" DEBUG Deleting Port "9fdf364c-8cf4-4be1-8581-8f6a02a22cd1" DEBUG Deleting Port "dd8084df-711a-4eb4-a5df-c01d40a6d05c" DEBUG Exiting deleting openstack ports DEBUG Deleting openstack networks DEBUG Deleting openstack server groups DEBUG Deleting openstack subnets DEBUG Deleting openstack security-groups DEBUG Deleting OpenStack volumes DEBUG Deleting network: "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" DEBUG Exiting deleting openstack server groups DEBUG goroutine deleteServerGroups complete DEBUG Deleting Network "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/c8b995ab-ff35-4da5-8aa6-7ef95f21aa00], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on network c8b995ab-ff35-4da5-8aa6-7ef95f21aa00. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} DEBUG Exiting deleting openstack networks DEBUG Deleting Subnet: "04dafe23-baec-4f1d-8a25-ee3c4717d647" DEBUG Deleting Security Group: "5fd19a64-077c-4c2c-b7d9-5cea9421bc35" DEBUG Deleting Subnet "04dafe23-baec-4f1d-8a25-ee3c4717d647" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/04dafe23-baec-4f1d-8a25-ee3c4717d647], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on subnet 04dafe23-baec-4f1d-8a25-ee3c4717d647: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Deleting Security Group: "c31e1737-2823-4737-b5f9-b6de26a0a695" DEBUG Exiting deleting openstack security-groups DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/95f84007-e2dc-43e6-bdbf-3c67e1832230], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} DEBUG Exiting deleting OpenStack volumes DEBUG Deleting openstack ports DEBUG Exiting deleting openstack ports DEBUG goroutine deletePorts complete DEBUG Deleting openstack networks DEBUG Deleting openstack subnets DEBUG Deleting openstack security-groups DEBUG Deleting network: "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" DEBUG Deleting Network "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/c8b995ab-ff35-4da5-8aa6-7ef95f21aa00], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on network c8b995ab-ff35-4da5-8aa6-7ef95f21aa00. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} DEBUG Exiting deleting openstack networks DEBUG Deleting Subnet: "04dafe23-baec-4f1d-8a25-ee3c4717d647" DEBUG Deleting Subnet "04dafe23-baec-4f1d-8a25-ee3c4717d647" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/04dafe23-baec-4f1d-8a25-ee3c4717d647], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on subnet 04dafe23-baec-4f1d-8a25-ee3c4717d647: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Exiting deleting openstack security-groups DEBUG goroutine deleteSecurityGroups complete DEBUG Deleting OpenStack volumes DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/95f84007-e2dc-43e6-bdbf-3c67e1832230], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} DEBUG Exiting deleting OpenStack volumes DEBUG Deleting openstack networks DEBUG Deleting openstack subnets DEBUG Deleting network: "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" DEBUG Deleting Network "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/c8b995ab-ff35-4da5-8aa6-7ef95f21aa00], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on network c8b995ab-ff35-4da5-8aa6-7ef95f21aa00. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} DEBUG Exiting deleting openstack networks DEBUG Deleting Subnet: "04dafe23-baec-4f1d-8a25-ee3c4717d647" DEBUG Deleting Subnet "04dafe23-baec-4f1d-8a25-ee3c4717d647" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/04dafe23-baec-4f1d-8a25-ee3c4717d647], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on subnet 04dafe23-baec-4f1d-8a25-ee3c4717d647: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Deleting OpenStack volumes DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/95f84007-e2dc-43e6-bdbf-3c67e1832230], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} DEBUG Exiting deleting OpenStack volumes DEBUG Deleting openstack networks DEBUG Deleting openstack subnets DEBUG Deleting network: "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" DEBUG Deleting Network "c8b995ab-ff35-4da5-8aa6-7ef95f21aa00" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/c8b995ab-ff35-4da5-8aa6-7ef95f21aa00], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on network c8b995ab-ff35-4da5-8aa6-7ef95f21aa00. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} DEBUG Exiting deleting openstack networks DEBUG Deleting Subnet: "04dafe23-baec-4f1d-8a25-ee3c4717d647" DEBUG Deleting Subnet "04dafe23-baec-4f1d-8a25-ee3c4717d647" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/04dafe23-baec-4f1d-8a25-ee3c4717d647], but got 409 instead DEBUG {"NeutronError": {"message": "Unable to complete operation on subnet 04dafe23-baec-4f1d-8a25-ee3c4717d647: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Deleting OpenStack volumes DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" DEBUG Deleting volume "95f84007-e2dc-43e6-bdbf-3c67e1832230" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/95f84007-e2dc-43e6-bdbf-3c67e1832230], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}}
For the same cluster, although the installer thought it deleted the router (goroutine deleteRouters complete), it also failed to delete it: moc-ci ❯ openstack router show crxf228b-3409b-7hsk2-external-router +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2021-01-14T21:14:05Z | | description | | | external_gateway_info | {"network_id": "71b97520-69af-4c35-8153-cdf827d96e60", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "3a43b38a-0a5e-40ad-bf98-27f9a57e3df5", "ip_address": "128.31.27.233"}]} | | flavor_id | None | | id | ae2d3a62-af94-41f4-904d-244ae00c1183 | | interfaces_info | [{"port_id": "91d53f0a-8199-4d97-99ce-a579f46c398b", "ip_address": "10.0.128.1", "subnet_id": "04dafe23-baec-4f1d-8a25-ee3c4717d647"}] | | location | cloud='moc-ci', project.domain_id='default', project.domain_name=, project.id='593227d1d5d04cba8847d5b6b742e0a7', project.name='rh-openshift-ci', region_name='moc-kzn', zone= | | name | crxf228b-3409b-7hsk2-external-router | | project_id | 593227d1d5d04cba8847d5b6b742e0a7 | | revision_number | 8 | | routes | | | status | ACTIVE | | tags | | | updated_at | 2021-01-14T21:14:12Z | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Looking at this again, the router is missing the tags that should identify it with the cluster. It seems like we fail to correctly apply the tag in case of UPI clusters.
Verified with 4.8.0-0.nightly-2021-02-24-041731 DEBUG OpenShift Installer 4.8.0-0.nightly-2021-02-24-041731 DEBUG Built from commit 5f40b052af9d455d7a5cc03ee739a81ef1f10f23 DEBUG Removing interfaces from custom router DEBUG Exiting removal of interfaces from custom router DEBUG Deleting OpenStack volumes DEBUG Deleting openstack floating ips DEBUG Deleting openstack routers DEBUG Deleting openstack load balancers DEBUG Deleting openstack server groups DEBUG Deleting openstack subnet-pools DEBUG Deleting openstack base image DEBUG Deleting openstack servers DEBUG Deleting openstack ports DEBUG Deleting openstack subnets DEBUG Deleting openstack containers DEBUG Deleting openstack trunks DEBUG Deleting openstack networks DEBUG Deleting openstack security-groups DEBUG Deleting Subnet: "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" DEBUG Exiting deleting openstack subnet-pools DEBUG goroutine deleteSubnetPools complete DEBUG Exiting deleting openstack base image DEBUG goroutine deleteImages complete DEBUG Exiting deleting openstack load balancers DEBUG goroutine deleteLoadBalancers complete DEBUG Exiting deleting openstack floating ips DEBUG goroutine deleteFloatingIPs complete DEBUG Deleting Server Group "b42d6ba0-7095-4515-9e1f-02ab08e0d705" DEBUG Deleting Subnet "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7], but got 409 instead DEBUG {"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7: One or more ports have an IP allocation from this subnet.", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Exiting deleting openstack trunks DEBUG goroutine deleteTrunks complete DEBUG Exiting deleting openstack server groups DEBUG Deleting network: "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" DEBUG Deleting Port "05ba7bfc-a28c-4be3-90bb-bed0f709d6e3" DEBUG Deleting Network "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/7c2f2f32-a1f7-42b3-be6f-b8f6b147757f], but got 409 instead DEBUG {"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network 7c2f2f32-a1f7-42b3-be6f-b8f6b147757f. There are one or more ports still in use on the network.", "detail": ""}} DEBUG Deleting Security Group: "10b08367-cb8f-4ac6-b507-b4f20b31d812" DEBUG Deleting Security Group "10b08367-cb8f-4ac6-b507-b4f20b31d812" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/10b08367-cb8f-4ac6-b507-b4f20b31d812], but got 409 instead DEBUG {"NeutronError": {"type": "SecurityGroupInUse", "message": "Security Group 10b08367-cb8f-4ac6-b507-b4f20b31d812 in use.", "detail": ""}} DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting OpenStack volumes DEBUG goroutine deleteVolumes complete DEBUG Exiting deleting openstack networks DEBUG Deleting Port "2c0a91e1-690f-4be6-8860-0e9cb48f0b49" DEBUG Deleting Server "c6c5f77a-89a4-4766-8c56-0aa9fe5fb814" DEBUG Deleting Server "fb2dbc6c-30c9-4c01-8aaf-a6d17907a21e" DEBUG Deleting Server "5c7011d9-05d4-4aa2-80af-da6a672a2c4f" DEBUG Deleting Server "43724117-226d-4d67-8d2a-6ea96172ba49" DEBUG Deleting Server "688833e3-3608-4e81-b1f0-0cd1057ef49d" DEBUG Deleting Port "61dadf60-06e3-4565-9609-579fb569f73c" DEBUG Exiting deleting openstack servers DEBUG Deleting Port "67b14ec8-e712-42fb-bef5-6acc1c8a6c28" DEBUG Removing Subnet "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" from Router "696a96c0-25b1-44cc-9e0c-a524fd15866a" DEBUG Deleting Port "8d637b98-b261-4d7c-8677-209950f64c90" DEBUG Deleting Port "ab658e8b-689c-4de1-bda7-815ceabf00e8" DEBUG Bulk deleting container "wj47uos224az-dgtq6-image-registry-vqrlegrlmegqliygddsckajgbddp" objects DEBUG Deleting container "wj47uos224az-dgtq6-image-registry-vqrlegrlmegqliygddsckajgbddp" DEBUG Deleting openstack subnets DEBUG Exiting deleting openstack containers DEBUG goroutine deleteContainers complete DEBUG Deleting openstack server groups DEBUG Deleting Port "da7d499a-5a04-4fe9-8d7c-dca46ae73cde" DEBUG Deleting Router "696a96c0-25b1-44cc-9e0c-a524fd15866a" DEBUG Deleting openstack security-groups DEBUG Deleting Subnet: "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" DEBUG Exiting deleting openstack routers DEBUG Exiting deleting openstack server groups DEBUG goroutine deleteServerGroups complete DEBUG Exiting deleting openstack ports DEBUG Deleting openstack networks DEBUG Deleting Security Group: "10b08367-cb8f-4ac6-b507-b4f20b31d812" DEBUG Deleting network: "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" DEBUG Deleting Security Group: "22949acb-960f-4aa0-930b-e0a41dfbb7fe" DEBUG Exiting deleting openstack subnets DEBUG Deleting openstack servers DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting openstack networks DEBUG Exiting deleting openstack servers DEBUG goroutine deleteServers complete DEBUG Deleting openstack routers DEBUG Deleting openstack ports DEBUG Exiting deleting openstack routers DEBUG goroutine deleteRouters complete DEBUG Exiting deleting openstack ports DEBUG goroutine deletePorts complete DEBUG Deleting openstack subnets DEBUG Deleting openstack security-groups DEBUG Exiting deleting openstack subnets DEBUG goroutine deleteSubnets complete DEBUG Deleting openstack networks DEBUG Exiting deleting openstack security-groups DEBUG goroutine deleteSecurityGroups complete DEBUG Exiting deleting openstack networks DEBUG goroutine deleteNetworks complete DEBUG Removing tag wj47uos224az-dgtq6-primaryClusterNetwork from openstack networks DEBUG Exiting untagging openstack networks INFO Time elapsed: 54s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438