Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1746748

Summary: [IPI] [OSP] After a correct installation, destroy command fails because of "FATAL Resource not found"
Product: OpenShift Container Platform Reporter: David Sanz <dsanzmor>
Component: InstallerAssignee: Martin André <m.andre>
Installer sub component: openshift-installer QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: dprince, m.andre, ppitonak
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:38:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Sanz 2019-08-29 08:17:07 UTC 
Description of problem:

First execution of destroy cluster fails trying to delete a server two times:

# ./openshift-install destroy cluster --log-level debug
DEBUG OpenShift Installer v4.2.0-201908282219-dirty 
DEBUG Built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b 
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack containers                
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack servers                   
DEBUG No suitable endpoint could be found in the service catalog. 
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Deleting Trunk: 3ef22ff8-65b4-4753-88ec-fa3c64425a04 
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting container: morenod-ocp-tfjfc-image-registry-knvcadtdkdxqfmnahmodhucsjatfr 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/3ef22ff8-65b4-4753-88ec-fa3c64425a04], but got 409 instead
{"NeutronError": {"message": "Trunk 3ef22ff8-65b4-4753-88ec-fa3c64425a04 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/0f5f5851-f7d4-47f0-b8f4-618f4210d666], but got 409 instead
{"NeutronError": {"message": "Port 0f5f5851-f7d4-47f0-b8f4-618f4210d666 is currently a parent port for trunk 9b780c87-88e4-4607-9188-967870a13a0e.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports             
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Server: 06ed1305-c21d-4fb9-93e6-019c2c8bbd89 
DEBUG Deleting Server: 93c41ee0-416d-4a07-8388-fa9150b9a350 
DEBUG Deleting Server: 8260640a-ac7e-4dd4-be58-12d33c22d60d 
DEBUG Deleting Server: a85b8720-fa1e-4118-ad6b-bc5b143c7f20 
DEBUG Deleting Server: 4b8a4c4d-09bf-40da-be49-8c19f98bd0cb 
DEBUG Deleting Server: b101dd8d-fd5a-48e7-88ad-2db992bb8ccb 
DEBUG Exiting deleting openstack servers           
DEBUG Removing Subnet fe20faeb-39fc-449d-a288-38839d0496a9 from Router 2383330d-718b-4b43-99cc-f45c21861286 
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack security-groups           
DEBUG Deleting Trunk: 3ef22ff8-65b4-4753-88ec-fa3c64425a04 
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack subnets                   
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting openstack ports                     
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Trunk: 63a61c12-eb10-4eec-b02e-d869eb2c2ca7 
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Trunk: 6c052b0b-adb0-4227-9904-ce17af093c0f 
DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/0f5f5851-f7d4-47f0-b8f4-618f4210d666], but got 409 instead
{"NeutronError": {"message": "Port 0f5f5851-f7d4-47f0-b8f4-618f4210d666 is currently a parent port for trunk 9b780c87-88e4-4607-9188-967870a13a0e.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
DEBUG Exiting deleting openstack ports             
DEBUG Deleting Trunk: 94f6d36f-b76d-4932-9779-60d8cfa8dc0e 
DEBUG Deleting Trunk: 9b780c87-88e4-4607-9188-967870a13a0e 
DEBUG Deleting Trunk: c22d262a-b030-477a-9853-d6ad22241ea4 
DEBUG Exiting deleting openstack trunks            
DEBUG Deleting openstack servers                   
DEBUG Deleting Server: 93c41ee0-416d-4a07-8388-fa9150b9a350 
FATAL Resource not found 


After that, if you execute destroy again, it finish correctly:

DEBUG OpenShift Installer v4.2.0-201908282219-dirty 
DEBUG Built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b 
DEBUG Deleting openstack servers                   
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack load balancers            
DEBUG Deleting openstack subnets                   
DEBUG Deleting openstack routers                   
DEBUG Deleting openstack subnet-pools              
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack trunks                    
DEBUG Deleting openstack containers                
DEBUG No suitable endpoint could be found in the service catalog. 
DEBUG Exiting deleting openstack load balancers    
DEBUG goroutine deleteLoadBalancers complete       
DEBUG Exiting deleting openstack subnet-pools      
DEBUG goroutine deleteSubnetPools complete         
DEBUG Exiting deleting openstack trunks            
DEBUG goroutine deleteTrunks complete              
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Port: 0f5f5851-f7d4-47f0-b8f4-618f4210d666 
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Exiting deleting openstack containers        
DEBUG goroutine deleteContainers complete          
DEBUG Deleting Router: 2383330d-718b-4b43-99cc-f45c21861286 
DEBUG Deleting Port: 4fa49efd-ddaa-477f-9bfa-407a2f485711 
DEBUG Exiting deleting openstack routers           
DEBUG Exiting deleting openstack servers           
DEBUG goroutine deleteServers complete             
DEBUG Deleting Port: 5083cdd0-d7a6-4c47-a034-c04293fed722 
DEBUG Deleting Port: 55803b59-a0f2-436b-862b-a211245ef4c8 
DEBUG Deleting Port: 55c8b6cb-d003-49e8-a983-2e42ad8ad07b 
DEBUG Deleting Port: 84f5c149-8417-49e3-871d-bb9b0bc3655a 
DEBUG Deleting openstack networks                  
DEBUG Deleting openstack security-groups           
DEBUG Deleting openstack subnets                   
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/b79427db-c971-4cd4-bc15-87579ffa956b], but got 409 instead
{"NeutronError": {"message": "Security Group b79427db-c971-4cd4-bc15-87579ffa956b in use.", "type": "SecurityGroupInUse", "detail": ""}} 
DEBUG Exiting deleting openstack security-groups   
DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/0bec0912-80f6-4707-9605-58c3adaf6940], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on network 0bec0912-80f6-4707-9605-58c3adaf6940. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
DEBUG Exiting deleting openstack networks          
DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/fe20faeb-39fc-449d-a288-38839d0496a9], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet fe20faeb-39fc-449d-a288-38839d0496a9: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
DEBUG Exiting deleting openstack subnets           
DEBUG Deleting Port: 87a066b3-678a-4df1-a05a-6ae69eb01572 
DEBUG Deleting openstack routers                   
DEBUG Exiting deleting openstack routers           
DEBUG goroutine deleteRouters complete             
DEBUG Deleting Port: 8f49cf53-6ee8-4ac4-82b4-fb6bdb3e8c4a 
DEBUG Exiting deleting openstack ports             
DEBUG Deleting openstack ports                     
DEBUG Deleting openstack security-groups           
DEBUG Exiting deleting openstack ports             
DEBUG goroutine deletePorts complete               
DEBUG Deleting openstack networks                  
DEBUG Deleting Security Group: b79427db-c971-4cd4-bc15-87579ffa956b 
DEBUG Deleting openstack subnets                   
DEBUG Deleting network: 0bec0912-80f6-4707-9605-58c3adaf6940 
DEBUG Deleting Subnet: fe20faeb-39fc-449d-a288-38839d0496a9 
DEBUG Deleting Security Group: e93f78f3-9ef6-4d95-8217-913805614061 
DEBUG Exiting deleting openstack security-groups   
DEBUG Exiting deleting openstack subnets           
DEBUG Exiting deleting openstack networks          
DEBUG Deleting openstack security-groups           
DEBUG Exiting deleting openstack security-groups   
DEBUG goroutine deleteSecurityGroups complete      
DEBUG Deleting openstack subnets                   
DEBUG Exiting deleting openstack subnets           
DEBUG goroutine deleteSubnets complete             
DEBUG Deleting openstack networks                  
DEBUG Exiting deleting openstack networks          
DEBUG goroutine deleteNetworks complete            
DEBUG Purging asset "Terraform Variables" from disk 
DEBUG Purging asset "Kubeconfig Admin Client" from disk 
DEBUG Purging asset "Kubeadmin Password" from disk 
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk 
DEBUG Purging asset "Metadata" from disk           
DEBUG Purging asset "Cluster" from disk            



Version-Release number of the following components:
4.2.0-0.nightly-2019-08-29-062233

How reproducible:

Steps to Reproduce:
1.Install cluster using IPI on OSP
2.Execute cluster deletion command: openshift-install destroy cluster
3.First execution fails with error: FATAL Resource not found                           

Actual results:
First execution of destroy command fails, second works fine

Expected results:
Destroy command works on the first try

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 1 Martin André 2019-08-29 15:25:22 UTC 
Reported upstream at https://github.com/openshift/installer/issues/1985.

At first glance, I'd say that's because https://github.com/openshift/installer/blob/d1b272a8ee8c428afc22ae274df0264384da5131/pkg/destroy/openstack/openstack.go#L98 is not thread safe.

Getting the list of resources to delete and effectively deleting them should be an atomic operation.
https://github.com/openshift/installer/blob/d1b272a8ee8c428afc22ae274df0264384da5131/pkg/destroy/openstack/openstack.go#L204-L226
Comment 3 David Sanz 2019-09-09 06:57:40 UTC 
Verified on 4.2.0-0.nightly-2019-09-08-232045
Comment 4 errata-xmlrpc 2019-10-16 06:38:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922