Bug 1862044

Summary: [OSP] Installer cannot destroy clusters created with older versions of the installer
Product: OpenShift Container Platform Reporter: Martin André <m.andre>
Component: InstallerAssignee: Mike Fedosin <mfedosin>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: pprinett
Version: 4.6   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:21:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin André 2020-07-30 09:35:33 UTC
When trying to destroy a cluster created with the installer < 4.6, the 4.6 installer is stuck in a loop:

DEBUG Deleting openstack trunks 
DEBUG Deleting openstack security-groups 
DEBUG Deleting openstack networks 
DEBUG Deleting openstack subnets 
DEBUG Deleting openstack ports 
DEBUG Deleting Trunk "2690e2ab-acd6-45a2-b796-7ac1af684833" 
DEBUG Deleting Subnet: "beca4736-8bd5-4462-96f5-05564ba11a38" 
DEBUG Deleting network: "e384f8b6-3be5-40d9-8a1b-10d711a1544d" 
DEBUG Deleting Security Group: "15d30b83-4b3e-4566-adfa-50af14f01a19" 
DEBUG Deleting Port "1cff6212-8a80-489e-b573-e471bc075b15" 
DEBUG Deleting Trunk "2690e2ab-acd6-45a2-b796-7ac1af684833" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/trunks/2690e2ab-acd6-45a2-b796-7ac1af684833], but got 409 instead
{"NeutronError": {"message": "Trunk 2690e2ab-acd6-45a2-b796-7ac1af684833 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks 
DEBUG Deleting Network "e384f8b6-3be5-40d9-8a1b-10d711a1544d" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/e384f8b6-3be5-40d9-8a1b-10d711a1544d], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network e384f8b6-3be5-40d9-8a1b-10d711a1544d. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks 
DEBUG Deleting Security Group "15d30b83-4b3e-4566-adfa-50af14f01a19" failed with error: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/15d30b83-4b3e-4566-adfa-50af14f01a19], but got 409 instead
{"NeutronError": {"message": "Security Group 15d30b83-4b3e-4566-adfa-50af14f01a19 in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups 
DEBUG Deleting Subnet "beca4736-8bd5-4462-96f5-05564ba11a38" failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/beca4736-8bd5-4462-96f5-05564ba11a38], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet beca4736-8bd5-4462-96f5-05564ba11a38: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets 
DEBUG Deleting Port "1cff6212-8a80-489e-b573-e471bc075b15" failed with error: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/ports/1cff6212-8a80-489e-b573-e471bc075b15], but got 409 instead
{"NeutronError": {"message": "Port 1cff6212-8a80-489e-b573-e471bc075b15 is currently a parent port for trunk 50501c62-7e85-4b9f-9003-e1da4d1bcc8b.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports 
DEBUG Deleting OpenStack volumes 
DEBUG Deleting volume "c532fb84-b342-4c58-91a0-da0098759325" 
DEBUG Deleting volume "c532fb84-b342-4c58-91a0-da0098759325" failed: Bad request with: [DELETE https://kaizen.massopen.cloud:13776/v2/593227d1d5d04cba8847d5b6b742e0a7/volumes/c532fb84-b342-4c58-91a0-da0098759325], error message: {"badRequest": {"message": "Invalid volume: Volume status must be available or error or error_restoring or error
_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer.", "code": 400}} 
DEBUG Exiting deleting OpenStack volumes 

We can see in the logs:

DEBUG goroutine deleteServers complete

which obviously was erroneous since the master nodes were still present. Manually deleting the master nodes allowed the installer to proceed with the destroy of other resources.

This is because we switched to tag filtering for server deletion, and the older clusters do not set the tags.

Comment 3 David Sanz 2020-08-13 11:49:45 UTC
Cluster installed on 4.5.0-0.nightly-2020-08-13-101206

Cluster updated to 4.6.0-0.nightly-2020-08-13-091737

Deleted using 4.5.0-0.nightly-2020-08-13-101206

No server left behind

Comment 5 errata-xmlrpc 2020-10-27 16:21:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196