Bug 1862044 - [OSP] Installer cannot destroy clusters created with older versions of the installer
Summary: [OSP] Installer cannot destroy clusters created with older versions of the in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Mike Fedosin
QA Contact: David Sanz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-30 09:35 UTC by Martin André
Modified: 2020-10-27 16:21 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:21:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4032 0 None closed Bug 1862044: deleting servers using metadata-based filtering 2020-10-01 08:10:26 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:21:49 UTC

Description Martin André 2020-07-30 09:35:33 UTC
When trying to destroy a cluster created with the installer < 4.6, the 4.6 installer is stuck in a loop:

DEBUG Deleting openstack trunks 
DEBUG Deleting openstack security-groups 
DEBUG Deleting openstack networks 
DEBUG Deleting openstack subnets 
DEBUG Deleting openstack ports 
DEBUG Deleting Trunk "2690e2ab-acd6-45a2-b796-7ac1af684833" 
DEBUG Deleting Subnet: "beca4736-8bd5-4462-96f5-05564ba11a38" 
DEBUG Deleting network: "e384f8b6-3be5-40d9-8a1b-10d711a1544d" 
DEBUG Deleting Security Group: "15d30b83-4b3e-4566-adfa-50af14f01a19" 
DEBUG Deleting Port "1cff6212-8a80-489e-b573-e471bc075b15" 
DEBUG Deleting Trunk "2690e2ab-acd6-45a2-b796-7ac1af684833" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/trunks/2690e2ab-acd6-45a2-b796-7ac1af684833], but got 409 instead
{"NeutronError": {"message": "Trunk 2690e2ab-acd6-45a2-b796-7ac1af684833 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks 
DEBUG Deleting Network "e384f8b6-3be5-40d9-8a1b-10d711a1544d" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/e384f8b6-3be5-40d9-8a1b-10d711a1544d], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network e384f8b6-3be5-40d9-8a1b-10d711a1544d. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks 
DEBUG Deleting Security Group "15d30b83-4b3e-4566-adfa-50af14f01a19" failed with error: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/15d30b83-4b3e-4566-adfa-50af14f01a19], but got 409 instead
{"NeutronError": {"message": "Security Group 15d30b83-4b3e-4566-adfa-50af14f01a19 in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups 
DEBUG Deleting Subnet "beca4736-8bd5-4462-96f5-05564ba11a38" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/beca4736-8bd5-4462-96f5-05564ba11a38], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet beca4736-8bd5-4462-96f5-05564ba11a38: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets 
DEBUG Deleting Port "1cff6212-8a80-489e-b573-e471bc075b15" failed with error: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/ports/1cff6212-8a80-489e-b573-e471bc075b15], but got 409 instead
{"NeutronError": {"message": "Port 1cff6212-8a80-489e-b573-e471bc075b15 is currently a parent port for trunk 50501c62-7e85-4b9f-9003-e1da4d1bcc8b.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports 
DEBUG Deleting OpenStack volumes 
DEBUG Deleting volume "c532fb84-b342-4c58-91a0-da0098759325" 
DEBUG Deleting volume "c532fb84-b342-4c58-91a0-da0098759325" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/c532fb84-b342-4c58-91a0-da0098759325], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error
_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} 
DEBUG Exiting deleting OpenStack volumes 

We can see in the logs:

DEBUG goroutine deleteServers complete

which obviously was erroneous since the master nodes were still present. Manually deleting the master nodes allowed the installer to proceed with the destroy of other resources.

This is because we switched to tag filtering for server deletion, and the older clusters do not set the tags.

Comment 3 David Sanz 2020-08-13 11:49:45 UTC
Cluster installed on 4.5.0-0.nightly-2020-08-13-101206

Cluster updated to 4.6.0-0.nightly-2020-08-13-091737

Deleted using 4.5.0-0.nightly-2020-08-13-101206

No server left behind

Comment 5 errata-xmlrpc 2020-10-27 16:21:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.