1746748 – [IPI] [OSP] After a correct installation, destroy command fails because of "FATAL Resource not found"

Bug 1746748 - [IPI] [OSP] After a correct installation, destroy command fails because of "FATAL Resource not found"

Summary: [IPI] [OSP] After a correct installation, destroy command fails because of "F...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Installer
Sub Component:
Version:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.2.0
Assignee:	Martin André
QA Contact:	David Sanz
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-08-29 08:17 UTC by David Sanz
Modified:	2019-10-16 06:38 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-10-16 06:38:33 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift installer issues 1985	None	closed	[openstack] FATAL Resource not found in destroy cluster	2020-07-15 03:58:45 UTC
Github	openshift installer pull 2322	None	closed	Bug 1746748: OpenStack: idempotent cluster deletion	2020-07-15 03:58:46 UTC
Red Hat Product Errata	RHBA-2019:2922	None	None	None	2019-10-16 06:38:41 UTC

Description David Sanz 2019-08-29 08:17:07 UTC

Description of problem:

First execution of destroy cluster fails trying to delete a server two times:

# ./openshift-install destroy cluster --log-level debug
DEBUG OpenShift Installer v4.2.0-201908282219-dirty 
DEBUG Built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b 
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack containers                
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack servers                   
DEBUG No suitable endpoint could be found in the service catalog. 
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Deleting Trunk: 3ef22ff8-65b4-4753-88ec-fa3c64425a04 
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting container: morenod-ocp-tfjfc-image-registry-knvcadtdkdxqfmnahmodhucsjatfr 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/3ef22ff8-65b4-4753-88ec-fa3c64425a04], but got 409 instead
{"NeutronError": {"message": "Trunk 3ef22ff8-65b4-4753-88ec-fa3c64425a04 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/0f5f5851-f7d4-47f0-b8f4-618f4210d666], but got 409 instead
{"NeutronError": {"message": "Port 0f5f5851-f7d4-47f0-b8f4-618f4210d666 is currently a parent port for trunk 9b780c87-88e4-4607-9188-967870a13a0e.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports             
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Server: 06ed1305-c21d-4fb9-93e6-019c2c8bbd89 
DEBUG Deleting Server: 93c41ee0-416d-4a07-8388-fa9150b9a350 
DEBUG Deleting Server: 8260640a-ac7e-4dd4-be58-12d33c22d60d 
DEBUG Deleting Server: a85b8720-fa1e-4118-ad6b-bc5b143c7f20 
DEBUG Deleting Server: 4b8a4c4d-09bf-40da-be49-8c19f98bd0cb 
DEBUG Deleting Server: b101dd8d-fd5a-48e7-88ad-2db992bb8ccb 
DEBUG Exiting deleting openstack servers           
DEBUG Removing Subnet fe20faeb-39fc-449d-a288-38839d0496a9 from Router 2383330d-718b-4b43-99cc-f45c21861286 
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack security-groups           
DEBUG Deleting Trunk: 3ef22ff8-65b4-4753-88ec-fa3c64425a04 
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack subnets                   
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting openstack ports                     
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Trunk: 63a61c12-eb10-4eec-b02e-d869eb2c2ca7 
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Trunk: 6c052b0b-adb0-4227-9904-ce17af093c0f 
DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/0f5f5851-f7d4-47f0-b8f4-618f4210d666], but got 409 instead
{"NeutronError": {"message": "Port 0f5f5851-f7d4-47f0-b8f4-618f4210d666 is currently a parent port for trunk 9b780c87-88e4-4607-9188-967870a13a0e.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports             
DEBUG Deleting Trunk: 94f6d36f-b76d-4932-9779-60d8cfa8dc0e 
DEBUG Deleting Trunk: 9b780c87-88e4-4607-9188-967870a13a0e 
DEBUG Deleting Trunk: c22d262a-b030-477a-9853-d6ad22241ea4 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting openstack servers                   
DEBUG Deleting Server: 93c41ee0-416d-4a07-8388-fa9150b9a350 
FATAL Resource not found 


After that, if you execute destroy again, it finish correctly:

DEBUG OpenShift Installer v4.2.0-201908282219-dirty 
DEBUG Built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b 
DEBUG Deleting openstack servers                   
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack containers                
DEBUG No suitable endpoint could be found in the service catalog. 
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Exiting deleting openstack trunks            
DEBUG goroutine deleteTrunks complete              
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Router: 2383330d-718b-4b43-99cc-f45c21861286 
DEBUG Deleting Port: 4fa49efd-ddaa-477f-9bfa-407a2f485711 
DEBUG Exiting deleting openstack routers           
DEBUG Exiting deleting openstack servers           
DEBUG goroutine deleteServers complete             
DEBUG Deleting Port: 5083cdd0-d7a6-4c47-a034-c04293fed722 
DEBUG Deleting Port: 55803b59-a0f2-436b-862b-a211245ef4c8 
DEBUG Deleting Port: 55c8b6cb-d003-49e8-a983-2e42ad8ad07b 
DEBUG Deleting Port: 84f5c149-8417-49e3-871d-bb9b0bc3655a 
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Port: 87a066b3-678a-4df1-a05a-6ae69eb01572 
DEBUG Deleting openstack routers                   
DEBUG Exiting deleting openstack routers           
DEBUG goroutine deleteRouters complete             
DEBUG Deleting Port: 8f49cf53-6ee8-4ac4-82b4-fb6bdb3e8c4a 
DEBUG Exiting deleting openstack ports             
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Exiting deleting openstack ports             
DEBUG goroutine deletePorts complete               
DEBUG Deleting openstack networks                  
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting openstack subnets                   
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group: e93f78f3-9ef6-4d95-8217-913805614061 
DEBUG Exiting deleting openstack security-groups   
DEBUG Exiting deleting openstack subnets           
DEBUG Exiting deleting openstack networks          
DEBUG Deleting openstack security-groups           
DEBUG Exiting deleting openstack security-groups   
DEBUG goroutine deleteSecurityGroups complete      
DEBUG Deleting openstack subnets                   
DEBUG Exiting deleting openstack subnets           
DEBUG goroutine deleteSubnets complete             
DEBUG Deleting openstack networks                  
DEBUG Exiting deleting openstack networks          
DEBUG goroutine deleteNetworks complete            
DEBUG Purging asset "Terraform Variables" from disk 
DEBUG Purging asset "Kubeconfig Admin Client" from disk 
DEBUG Purging asset "Kubeadmin Password" from disk 
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk 
DEBUG Purging asset "Metadata" from disk           
DEBUG Purging asset "Cluster" from disk            



Version-Release number of the following components:
4.2.0-0.nightly-2019-08-29-062233

How reproducible:

Steps to Reproduce:
1.Install cluster using IPI on OSP
2.Execute cluster deletion command: openshift-install destroy cluster
3.First execution fails with error: FATAL Resource not found                           

Actual results:
First execution of destroy command fails, second works fine

Expected results:
Destroy command works on the first try

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Martin André 2019-08-29 15:25:22 UTC

Reported upstream at https://github.com/openshift/installer/issues/1985.

At first glance, I'd say that's because https://github.com/openshift/installer/blob/d1b272a8ee8c428afc22ae274df0264384da5131/pkg/destroy/openstack/openstack.go#L98 is not thread safe.

Getting the list of resources to delete and effectively deleting them should be an atomic operation.
https://github.com/openshift/installer/blob/d1b272a8ee8c428afc22ae274df0264384da5131/pkg/destroy/openstack/openstack.go#L204-L226

Comment 3 David Sanz 2019-09-09 06:57:40 UTC

Verified on 4.2.0-0.nightly-2019-09-08-232045

Comment 4 errata-xmlrpc 2019-10-16 06:38:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922

Note You need to log in before you can comment on or make changes to this bug.