Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1754542

Summary: [IPI][Openstack]: Destroy Script Fails when run before bootstrap completes
Product: OpenShift Container Platform Reporter: egarcia
Component: InstallerAssignee: Mike Fedosin <mfedosin>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: calfonso, dsanzmor
Version: 4.2.zFlags: egarcia: needinfo-
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-23 11:06:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description egarcia 2019-09-23 14:00:38 UTC 
Description of problem: When you run the destroy script for the openstack IPI installer before bootstrapping completes, it fails.

How reproducible: always

Steps to Reproduce:
1. run installer
2. stop before bootstrap complete phase
3. run destroy script

Actual results:
[egarcia@emilio shiftstack-ci]$ ./destroy_cluster.sh 
Destroying cluster using openshift-install
DEBUG OpenShift Installer unreleased-master-1780-gf96afb99f1ce4f8976ce62f7df44acb24d2062d6 
DEBUG Built from commit f96afb99f1ce4f8976ce62f7df44acb24d2062d6 
DEBUG Deleting openstack floating ips              
DEBUG Deleting OpenStack volumes                   
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack containers                
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack servers                   
DEBUG Deleting Trunk: c8220484-eb3a-43e3-9d1e-6904ee69735f 
DEBUG Deleting Subnet: 9b628661-5edc-4cdc-a6fd-b8569c4dec27 
DEBUG Deleting Security Group: 7c1f847b-897d-4efd-8d57-8b206efcc1d4 
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Deleting network: 49a678d4-c53c-40ae-9704-bbf526a6e3e8 
DEBUG Deleting Floating IP: b2ebd91a-f04a-426a-a44d-ca45537d19fa 
DEBUG Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/trunks/c8220484-eb3a-43e3-9d1e-6904ee69735f], but got 409 instead
{"NeutronError": {"message": "Trunk c8220484-eb3a-43e3-9d1e-6904ee69735f is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting object: bootstrap.ign               
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/7c1f847b-897d-4efd-8d57-8b206efcc1d4], but got 409 instead
{"NeutronError": {"message": "Security Group 7c1f847b-897d-4efd-8d57-8b206efcc1d4 in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/9b628661-5edc-4cdc-a6fd-b8569c4dec27], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet 9b628661-5edc-4cdc-a6fd-b8569c4dec27: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting container: emilio-xtkk8             
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/49a678d4-c53c-40ae-9704-bbf526a6e3e8], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 49a678d4-c53c-40ae-9704-bbf526a6e3e8. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Deleting Floating IP: 18d409f0-9865-4366-b7f2-a05c77bdc0e6 
DEBUG Exiting deleting OpenStack volumes           
DEBUG goroutine deleteVolumes complete             
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Port: 1d782824-6b29-402d-b2ed-71c00c620fc2 
DEBUG Deleting Server: 59e31bd9-8dcd-4a0f-b78b-dba909c48363 
DEBUG Exiting deleting openstack floating ips      
DEBUG Deleting Server: a23f9c9b-abb9-433c-aa2c-9ad3ca85237a 
FATAL Resource not found        

Expected results:
[egarcia@emilio shiftstack-ci]$ ./destroy_cluster.sh 
Destroying cluster using openshift-install
DEBUG OpenShift Installer unreleased-master-1780-gf96afb99f1ce4f8976ce62f7df44acb24d2062d6 
DEBUG Built from commit f96afb99f1ce4f8976ce62f7df44acb24d2062d6 
DEBUG Deleting OpenStack volumes                   
DEBUG Deleting openstack containers                
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack servers                   
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack floating ips              
DEBUG Deleting Trunk: c8220484-eb3a-43e3-9d1e-6904ee69735f 
DEBUG Deleting Security Group: 7c1f847b-897d-4efd-8d57-8b206efcc1d4 
DEBUG Deleting Subnet: 9b628661-5edc-4cdc-a6fd-b8569c4dec27 
DEBUG Deleting network: 49a678d4-c53c-40ae-9704-bbf526a6e3e8 
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Exiting deleting openstack floating ips      
DEBUG goroutine deleteFloatingIPs complete         
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/7c1f847b-897d-4efd-8d57-8b206efcc1d4], but got 409 instead
{"NeutronError": {"message": "Security Group 7c1f847b-897d-4efd-8d57-8b206efcc1d4 in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Trunk: d38e1021-7b36-42a8-9d5f-12703b50c8c7 
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/9b628661-5edc-4cdc-a6fd-b8569c4dec27], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet 9b628661-5edc-4cdc-a6fd-b8569c4dec27: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/49a678d4-c53c-40ae-9704-bbf526a6e3e8], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 49a678d4-c53c-40ae-9704-bbf526a6e3e8. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Port: 3a38db03-dd02-43b9-b287-4f40c56abebd 
DEBUG Exiting deleting OpenStack volumes           
DEBUG goroutine deleteVolumes complete             
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Trunk: d700f52c-2971-4cd2-8b67-b4f8bcb7f108 
DEBUG Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/trunks/d700f52c-2971-4cd2-8b67-b4f8bcb7f108], but got 409 instead
{"NeutronError": {"message": "Trunk d700f52c-2971-4cd2-8b67-b4f8bcb7f108 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting Server: aef3a736-35d8-4e7e-bf00-f919eaeedcbc 
iDEBUG Deleting Server: 398d819b-ebb7-4577-b01b-bd2399d33ecc 
DEBUG Exiting deleting openstack servers           
DEBUG Deleting Port: 77621eed-d716-41c1-9a9f-9fe8bfc177bf 
DEBUG Deleting Port: 87ed62f6-e527-4bd7-942e-0c72a5ac59a6 
DEBUG Removing Subnet 9b628661-5edc-4cdc-a6fd-b8569c4dec27 from Router 995c2fa4-f9e9-46e9-b4eb-731f0ed69351 
DEBUG Deleting Port: ca65d068-406a-4faf-bb27-7cd28827833b 
DEBUG Deleting Port: fb6eee81-35f5-4156-854b-d0db6ba0e7a5 
DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/ports/fb6eee81-35f5-4156-854b-d0db6ba0e7a5], but got 409 instead
{"NeutronError": {"message": "Port fb6eee81-35f5-4156-854b-d0db6ba0e7a5 is currently a parent port for trunk d700f52c-2971-4cd2-8b67-b4f8bcb7f108.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports             
DEBUG Deleting Router: 995c2fa4-f9e9-46e9-b4eb-731f0ed69351 
DEBUG Exiting deleting openstack routers           
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack trunks                    
DEBUG Deleting Security Group: 7c1f847b-897d-4efd-8d57-8b206efcc1d4 
DEBUG Deleting Subnet: 9b628661-5edc-4cdc-a6fd-b8569c4dec27 
DEBUG Deleting network: 49a678d4-c53c-40ae-9704-bbf526a6e3e8 
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/security-groups/7c1f847b-897d-4efd-8d57-8b206efcc1d4], but got 409 instead
{"NeutronError": {"message": "Security Group 7c1f847b-897d-4efd-8d57-8b206efcc1d4 in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/networks/49a678d4-c53c-40ae-9704-bbf526a6e3e8], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 49a678d4-c53c-40ae-9704-bbf526a6e3e8. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting openstack servers                   
DEBUG Deleting Trunk: d700f52c-2971-4cd2-8b67-b4f8bcb7f108 
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://kaizen.massopen.cloud:13696/v2.0/subnets/9b628661-5edc-4cdc-a6fd-b8569c4dec27], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet 9b628661-5edc-4cdc-a6fd-b8569c4dec27: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Exiting deleting openstack trunks            
DEBUG Exiting deleting openstack servers           
DEBUG goroutine deleteServers complete             
DEBUG Deleting openstack ports                     
DEBUG Deleting Port: fb6eee81-35f5-4156-854b-d0db6ba0e7a5 
DEBUG Deleting openstack routers                   
DEBUG Exiting deleting openstack ports             
DEBUG Exiting deleting openstack routers           
DEBUG goroutine deleteRouters complete             
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting Security Group: 7c1f847b-897d-4efd-8d57-8b206efcc1d4 
DEBUG Deleting network: 49a678d4-c53c-40ae-9704-bbf526a6e3e8 
DEBUG Deleting Subnet: 9b628661-5edc-4cdc-a6fd-b8569c4dec27 
DEBUG Deleting Security Group: aa8c789a-5e6c-4eff-b900-3cbad12f1c0e 
DEBUG Exiting deleting openstack trunks            
DEBUG goroutine deleteTrunks complete              
DEBUG Exiting deleting openstack security-groups   
DEBUG Cannot find network 49a678d4-c53c-40ae-9704-bbf526a6e3e8. It's probably already been deleted. 
DEBUG Exiting deleting openstack networks          
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting openstack ports                     
DEBUG Exiting deleting openstack ports             
DEBUG goroutine deletePorts complete               
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack subnets                   
DEBUG Exiting deleting openstack security-groups   
DEBUG goroutine deleteSecurityGroups complete      
DEBUG Exiting deleting openstack subnets           
DEBUG goroutine deleteSubnets complete             
DEBUG Deleting network: 49a678d4-c53c-40ae-9704-bbf526a6e3e8 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting openstack networks                  
DEBUG Exiting deleting openstack networks          
DEBUG goroutine deleteNetworks complete            
DEBUG Purging asset "Terraform Variables" from disk 
DEBUG Purging asset "Kubeconfig Admin Client" from disk 
DEBUG Purging asset "Kubeadmin Password" from disk 
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk 
DEBUG Purging asset "Metadata" from disk           
DEBUG Purging asset "Cluster" from disk
   
Additional info:
Comment 1 David Sanz 2019-09-24 13:15:03 UTC 
*** Bug 1754856 has been marked as a duplicate of this bug. ***
Comment 3 egarcia 2019-11-13 18:56:52 UTC 
I have not backported it yet, the fix is merged in master.
Comment 4 egarcia 2019-11-13 19:08:02 UTC 
Should we backport this at all? We aren't backporting auto image upload or self signed certs, so this issue cant occur in the first place.
Comment 5 Mike Fedosin 2019-11-14 06:29:00 UTC 
Should be fixed by https://github.com/openshift/installer/pull/2561
Comment 6 Mike Fedosin 2019-11-14 06:30:08 UTC 
Imho it makes sense to backport the patch, since it doesn't affect much code
Comment 7 David Sanz 2019-11-14 15:39:16 UTC 
Verified on 4.2.0-0.nightly-2019-11-13-203727
Comment 9 errata-xmlrpc 2020-01-23 11:06:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062