Bug 1750636

Summary: [IPI] [OSP] Destroy cluster will not prune bootstrap fip if installation failed to wait for bootstrap to complete
Product: OpenShift Container Platform Reporter: weiwei jiang <wjiang>
Component: InstallerAssignee: Mike Fedosin <mfedosin>
Installer sub component: openshift-installer QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: eduen
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:40:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description weiwei jiang 2019-09-10 06:34:29 UTC 
Description of problem:
After failed to wait for bootstrap to complete due to some reason, 
trying to destroy cluster but it will not prune bootstrap fip.
And bootstrap fip will be orphan resource.

Version-Release number of the following components:
./openshift-install v4.2.0
built from commit a42ba5bd8ee70f0bfbc593af2938fe629916a4db
release image registry.svc.ci.openshift.org/ocp/release@sha256:8dfcda91fc74010a764855cf632dce2b5a4bdf8eb92aed01a6c464b5e972fbd7

How reproducible:
Always

Steps to Reproduce:
1. Try to install IPI on OSP and make it failed to bootstrap
2. Then destroy cluster
3.

Actual results:
bootstrap-fip is left after successful deprovision.


Expected results:
bootstrap fip should be pruned.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 4 weiwei jiang 2019-09-12 09:02:47 UTC 
    Verified on 4.2.0-0.nightly-2019-09-12-072817

    ➜  installer git:(master) ✗ export CLUSTER_NAME=wj0912a
    ➜  installer git:(master) ✗ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-09-12-072817 
    ➜  installer git:(master) ✗ ./openshift-install-${OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE#r*:} destroy cluster --dir ${CLUSTER_NAME} --log-level debug 
    DEBUG OpenShift Installer v4.2.0                   
    DEBUG Built from commit 073ee39e7bb99e6bc09245a9a877ab6915251dc5 
    DEBUG Deleting openstack subnets                   
    DEBUG Deleting openstack servers                   
    DEBUG Deleting OpenStack volumes                   
    DEBUG Deleting openstack subnet-pools              
    DEBUG Deleting openstack networks                  
    DEBUG Deleting openstack floating ips              
    DEBUG Deleting openstack containers                
    DEBUG Deleting openstack load balancers            
    DEBUG Deleting openstack trunks                    
    DEBUG Deleting openstack security-groups           
    DEBUG Deleting openstack routers                   
    DEBUG Deleting openstack ports                     
    DEBUG No suitable endpoint could be found in the service catalog. 
    DEBUG Exiting deleting openstack load balancers    
    DEBUG goroutine deleteLoadBalancers complete       
    DEBUG Deleting network: ac5b8188-ba2e-4862-8231-5cb1648e4708 
    DEBUG Exiting deleting openstack subnet-pools      
    DEBUG goroutine deleteSubnetPools complete         
    DEBUG Deleting Trunk: a7d646d9-51a5-4576-89c7-cbeea809dba3 
    DEBUG Exiting deleting openstack floating ips      
    DEBUG goroutine deleteFloatingIPs complete         
    DEBUG Deleting Subnet: efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6 
    DEBUG Deleting Security Group: 7c41d672-0a95-45d2-a5ce-98ce80b8aafe 
    DEBUG Deleting Trunk failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/trunks/a7d646d9-51a5-4576-89c7-cbeea809dba3], but got 409 instead
    {"NeutronError": {"message": "Trunk a7d646d9-51a5-4576-89c7-cbeea809dba3 is currently in use.", "type": "TrunkInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack trunks            
    DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/ac5b8188-ba2e-4862-8231-5cb1648e4708], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on network ac5b8188-ba2e-4862-8231-5cb1648e4708. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack networks          
    DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/7c41d672-0a95-45d2-a5ce-98ce80b8aafe], but got 409 instead
    {"NeutronError": {"message": "Security Group 7c41d672-0a95-45d2-a5ce-98ce80b8aafe in use.", "type": "SecurityGroupInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack security-groups   
    DEBUG Exiting deleting OpenStack volumes           
    DEBUG goroutine deleteVolumes complete             
    DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on subnet efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack subnets           
    DEBUG Deleting Port: 5782184f-c9b1-4989-b2c1-f3027a9a8d37 
    DEBUG Deleting Server: 486213bc-7ca2-4fe1-abab-9b5b8f4ab7ce 
    DEBUG Deleting Server: 84a131ad-5cba-4869-a54b-28b9a65cf388 
    DEBUG Deleting Port: 6811346a-5fb0-49ba-add7-5e00afb5acb7 
    DEBUG Deleting Server: eacc6a9e-b9d3-4c7e-85bf-7eace9a89410 
    DEBUG Deleting Server: add02bec-91b2-48d5-80a3-fb157a73c787 
    DEBUG Exiting deleting openstack servers           
    DEBUG Removing Subnet efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6 from Router a11f3f82-c9b0-493e-9f0d-c780a7730334 
    DEBUG Deleting openstack trunks                    
    DEBUG Deleting openstack networks                  
    DEBUG Deleting openstack security-groups           
    DEBUG Deleting openstack subnets                   
    DEBUG Deleting Port: 80677634-8d6f-4760-8e2f-81c93f3b603e 
    DEBUG Deleting Port failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/ports/80677634-8d6f-4760-8e2f-81c93f3b603e], but got 409 instead
    {"NeutronError": {"message": "Port 80677634-8d6f-4760-8e2f-81c93f3b603e is currently a parent port for trunk a7d646d9-51a5-4576-89c7-cbeea809dba3.", "type": "PortInUseAsTrunkParent", "detail": ""}} 
    DEBUG Exiting deleting openstack ports             
    DEBUG Deleting Trunk: a7d646d9-51a5-4576-89c7-cbeea809dba3 
    DEBUG Deleting network: ac5b8188-ba2e-4862-8231-5cb1648e4708 
    DEBUG Deleting Subnet: efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6 
    DEBUG Deleting Security Group: 7c41d672-0a95-45d2-a5ce-98ce80b8aafe 
    DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/ac5b8188-ba2e-4862-8231-5cb1648e4708], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on network ac5b8188-ba2e-4862-8231-5cb1648e4708. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack networks          
    DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on subnet efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack subnets           
    DEBUG Deleting Trunk: c5bb1027-13a7-442c-a151-499123473a1a 
    DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/7c41d672-0a95-45d2-a5ce-98ce80b8aafe], but got 409 instead
    {"NeutronError": {"message": "Security Group 7c41d672-0a95-45d2-a5ce-98ce80b8aafe in use.", "type": "SecurityGroupInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack security-groups   
    DEBUG Deleting Trunk: e10e1ac5-2868-42c6-8742-56bddb481dfd 
    DEBUG Deleting openstack servers                   
    DEBUG Exiting deleting openstack trunks            
    DEBUG Deleting object: bootstrap.ign               
    DEBUG Deleting container: wj0912a-5qwrk            
    DEBUG Deleting openstack ports                     
    DEBUG Deleting Router: a11f3f82-c9b0-493e-9f0d-c780a7730334 
    DEBUG Exiting deleting openstack servers           
    DEBUG goroutine deleteServers complete             
    DEBUG Exiting deleting openstack routers           
    DEBUG Deleting Port: 80677634-8d6f-4760-8e2f-81c93f3b603e 
    DEBUG Deleting openstack networks                  
    DEBUG Deleting openstack subnets                   
    DEBUG Exiting deleting openstack containers        
    DEBUG goroutine deleteContainers complete          
    DEBUG Deleting openstack security-groups           
    DEBUG Deleting Subnet: efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6 
    DEBUG Deleting network: ac5b8188-ba2e-4862-8231-5cb1648e4708 
    DEBUG Deleting Port: 8cde71d3-97ad-458d-8b9a-0d84d87c4be1 
    DEBUG Deleting openstack trunks                    
    DEBUG Deleting Network failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/networks/ac5b8188-ba2e-4862-8231-5cb1648e4708], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on network ac5b8188-ba2e-4862-8231-5cb1648e4708. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack networks          
    DEBUG Deleting Security Group: 7c41d672-0a95-45d2-a5ce-98ce80b8aafe 
    DEBUG Deleting Subnet failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/subnets/efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6], but got 409 instead
    {"NeutronError": {"message": "Unable to complete operation on subnet efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack subnets           
    DEBUG Deleting Security Group failed: Expected HTTP response code [] when accessing [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13696/v2.0/security-groups/7c41d672-0a95-45d2-a5ce-98ce80b8aafe], but got 409 instead
    {"NeutronError": {"message": "Security Group 7c41d672-0a95-45d2-a5ce-98ce80b8aafe in use.", "type": "SecurityGroupInUse", "detail": ""}} 
    DEBUG Exiting deleting openstack security-groups   
    DEBUG Exiting deleting openstack trunks            
    DEBUG goroutine deleteTrunks complete              
    DEBUG Deleting Port: 93eb6684-b71a-4578-beb2-7b6fce07d589 
    DEBUG Deleting openstack routers                   
    DEBUG Deleting Port: ee4ce053-ed25-453b-9dd2-b0bc50da3069 
    DEBUG Exiting deleting openstack routers           
    DEBUG goroutine deleteRouters complete             
    DEBUG Exiting deleting openstack ports             
    DEBUG Deleting openstack networks                  
    DEBUG Deleting openstack subnets                   
    DEBUG Deleting openstack security-groups           
    DEBUG Deleting Subnet: efdcdbbc-c953-4f25-ba9b-eaee0e6f6ac6 
    DEBUG Deleting network: ac5b8188-ba2e-4862-8231-5cb1648e4708 
    DEBUG Deleting Security Group: 7c41d672-0a95-45d2-a5ce-98ce80b8aafe 
    DEBUG Deleting openstack ports                     
    DEBUG Deleting Security Group: c136cc05-7cda-4183-b358-a1a98d3a2e63 
    DEBUG Exiting deleting openstack subnets           
    DEBUG Exiting deleting openstack networks          
    DEBUG Exiting deleting openstack security-groups   
    DEBUG Exiting deleting openstack ports             
    DEBUG goroutine deletePorts complete               
    DEBUG Deleting openstack subnets                   
    DEBUG Deleting openstack networks                  
    DEBUG Deleting openstack security-groups           
    DEBUG Exiting deleting openstack subnets           
    DEBUG goroutine deleteSubnets complete             
    DEBUG Exiting deleting openstack networks          
    DEBUG goroutine deleteNetworks complete            
    DEBUG Exiting deleting openstack security-groups   
    DEBUG goroutine deleteSecurityGroups complete      
    DEBUG Purging asset "Terraform Variables" from disk 
    DEBUG Purging asset "Kubeconfig Admin Client" from disk 
    DEBUG Purging asset "Kubeadmin Password" from disk 
    DEBUG Purging asset "Certificate (journal-gatewayd)" from disk 
    DEBUG Purging asset "Metadata" from disk           
    DEBUG Purging asset "Cluster" from disk
Comment 5 errata-xmlrpc 2019-10-16 06:40:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922