Bug 1916593
| Summary: | Destroy cluster sometimes stuck in a loop | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Martin André <m.andre> |
| Component: | Installer | Assignee: | Martin André <m.andre> |
| Installer sub component: | OpenShift on OpenStack | QA Contact: | Gaoyun Pei <gpei> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | pprinett |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: OpenStack UPI playbooks did not tag all resources they created.
Consequence: The "openshift-install destroy" command fails to properly identify all of the cluster resources and loops over resource deletion until it reaches a timeout, leaving resources behind.
Fix: Add missing tag instructions to OpenStack UPI playbooks.
Result: The "openshift-install destroy" command can fully deprovision cluster provisioned via OpenStack UPI playbooks.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:36:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Martin André
2021-01-15 08:13:45 UTC
For the same cluster, although the installer thought it deleted the router (goroutine deleteRouters complete), it also failed to delete it:
moc-ci ❯ openstack router show crxf228b-3409b-7hsk2-external-router
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2021-01-14T21:14:05Z |
| description | |
| external_gateway_info | {"network_id": "71b97520-69af-4c35-8153-cdf827d96e60", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "3a43b38a-0a5e-40ad-bf98-27f9a57e3df5", "ip_address": "128.31.27.233"}]} |
| flavor_id | None |
| id | ae2d3a62-af94-41f4-904d-244ae00c1183 |
| interfaces_info | [{"port_id": "91d53f0a-8199-4d97-99ce-a579f46c398b", "ip_address": "10.0.128.1", "subnet_id": "04dafe23-baec-4f1d-8a25-ee3c4717d647"}] |
| location | cloud='moc-ci', project.domain_id='default', project.domain_name=, project.id='593227d1d5d04cba8847d5b6b742e0a7', project.name='rh-openshift-ci', region_name='moc-kzn', zone= |
| name | crxf228b-3409b-7hsk2-external-router |
| project_id | 593227d1d5d04cba8847d5b6b742e0a7 |
| revision_number | 8 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2021-01-14T21:14:12Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Looking at this again, the router is missing the tags that should identify it with the cluster. It seems like we fail to correctly apply the tag in case of UPI clusters. Verified with 4.8.0-0.nightly-2021-02-24-041731 DEBUG OpenShift Installer 4.8.0-0.nightly-2021-02-24-041731 DEBUG Built from commit 5f40b052af9d455d7a5cc03ee739a81ef1f10f23 DEBUG Removing interfaces from custom router DEBUG Exiting removal of interfaces from custom router DEBUG Deleting OpenStack volumes DEBUG Deleting openstack floating ips DEBUG Deleting openstack routers DEBUG Deleting openstack load balancers DEBUG Deleting openstack server groups DEBUG Deleting openstack subnet-pools DEBUG Deleting openstack base image DEBUG Deleting openstack servers DEBUG Deleting openstack ports DEBUG Deleting openstack subnets DEBUG Deleting openstack containers DEBUG Deleting openstack trunks DEBUG Deleting openstack networks DEBUG Deleting openstack security-groups DEBUG Deleting Subnet: "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" DEBUG Exiting deleting openstack subnet-pools DEBUG goroutine deleteSubnetPools complete DEBUG Exiting deleting openstack base image DEBUG goroutine deleteImages complete DEBUG Exiting deleting openstack load balancers DEBUG goroutine deleteLoadBalancers complete DEBUG Exiting deleting openstack floating ips DEBUG goroutine deleteFloatingIPs complete DEBUG Deleting Server Group "b42d6ba0-7095-4515-9e1f-02ab08e0d705" DEBUG Deleting Subnet "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7], but got 409 instead DEBUG {"NeutronError": {"type": "SubnetInUse", "message": "Unable to complete operation on subnet dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7: One or more ports have an IP allocation from this subnet.", "detail": ""}} DEBUG Exiting deleting openstack subnets DEBUG Exiting deleting openstack trunks DEBUG goroutine deleteTrunks complete DEBUG Exiting deleting openstack server groups DEBUG Deleting network: "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" DEBUG Deleting Port "05ba7bfc-a28c-4be3-90bb-bed0f709d6e3" DEBUG Deleting Network "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/7c2f2f32-a1f7-42b3-be6f-b8f6b147757f], but got 409 instead DEBUG {"NeutronError": {"type": "NetworkInUse", "message": "Unable to complete operation on network 7c2f2f32-a1f7-42b3-be6f-b8f6b147757f. There are one or more ports still in use on the network.", "detail": ""}} DEBUG Deleting Security Group: "10b08367-cb8f-4ac6-b507-b4f20b31d812" DEBUG Deleting Security Group "10b08367-cb8f-4ac6-b507-b4f20b31d812" failed with error: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/10b08367-cb8f-4ac6-b507-b4f20b31d812], but got 409 instead DEBUG {"NeutronError": {"type": "SecurityGroupInUse", "message": "Security Group 10b08367-cb8f-4ac6-b507-b4f20b31d812 in use.", "detail": ""}} DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting OpenStack volumes DEBUG goroutine deleteVolumes complete DEBUG Exiting deleting openstack networks DEBUG Deleting Port "2c0a91e1-690f-4be6-8860-0e9cb48f0b49" DEBUG Deleting Server "c6c5f77a-89a4-4766-8c56-0aa9fe5fb814" DEBUG Deleting Server "fb2dbc6c-30c9-4c01-8aaf-a6d17907a21e" DEBUG Deleting Server "5c7011d9-05d4-4aa2-80af-da6a672a2c4f" DEBUG Deleting Server "43724117-226d-4d67-8d2a-6ea96172ba49" DEBUG Deleting Server "688833e3-3608-4e81-b1f0-0cd1057ef49d" DEBUG Deleting Port "61dadf60-06e3-4565-9609-579fb569f73c" DEBUG Exiting deleting openstack servers DEBUG Deleting Port "67b14ec8-e712-42fb-bef5-6acc1c8a6c28" DEBUG Removing Subnet "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" from Router "696a96c0-25b1-44cc-9e0c-a524fd15866a" DEBUG Deleting Port "8d637b98-b261-4d7c-8677-209950f64c90" DEBUG Deleting Port "ab658e8b-689c-4de1-bda7-815ceabf00e8" DEBUG Bulk deleting container "wj47uos224az-dgtq6-image-registry-vqrlegrlmegqliygddsckajgbddp" objects DEBUG Deleting container "wj47uos224az-dgtq6-image-registry-vqrlegrlmegqliygddsckajgbddp" DEBUG Deleting openstack subnets DEBUG Exiting deleting openstack containers DEBUG goroutine deleteContainers complete DEBUG Deleting openstack server groups DEBUG Deleting Port "da7d499a-5a04-4fe9-8d7c-dca46ae73cde" DEBUG Deleting Router "696a96c0-25b1-44cc-9e0c-a524fd15866a" DEBUG Deleting openstack security-groups DEBUG Deleting Subnet: "dc3ecd01-9b28-4a0a-8545-ef6641a6d9a7" DEBUG Exiting deleting openstack routers DEBUG Exiting deleting openstack server groups DEBUG goroutine deleteServerGroups complete DEBUG Exiting deleting openstack ports DEBUG Deleting openstack networks DEBUG Deleting Security Group: "10b08367-cb8f-4ac6-b507-b4f20b31d812" DEBUG Deleting network: "7c2f2f32-a1f7-42b3-be6f-b8f6b147757f" DEBUG Deleting Security Group: "22949acb-960f-4aa0-930b-e0a41dfbb7fe" DEBUG Exiting deleting openstack subnets DEBUG Deleting openstack servers DEBUG Exiting deleting openstack security-groups DEBUG Exiting deleting openstack networks DEBUG Exiting deleting openstack servers DEBUG goroutine deleteServers complete DEBUG Deleting openstack routers DEBUG Deleting openstack ports DEBUG Exiting deleting openstack routers DEBUG goroutine deleteRouters complete DEBUG Exiting deleting openstack ports DEBUG goroutine deletePorts complete DEBUG Deleting openstack subnets DEBUG Deleting openstack security-groups DEBUG Exiting deleting openstack subnets DEBUG goroutine deleteSubnets complete DEBUG Deleting openstack networks DEBUG Exiting deleting openstack security-groups DEBUG goroutine deleteSecurityGroups complete DEBUG Exiting deleting openstack networks DEBUG goroutine deleteNetworks complete DEBUG Removing tag wj47uos224az-dgtq6-primaryClusterNetwork from openstack networks DEBUG Exiting untagging openstack networks INFO Time elapsed: 54s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |