Bug 1772477

Summary: [IPI][Openstack] First attempt to destroy cluster sometimes fails with msg="Resource not found"
Product: OpenShift Container Platform Reporter: Filip Brychta <fbrychta>
Component: InstallerAssignee: Pierre Prinetti <pprinett>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: itbrown, m.andre, mfedosin
Version: 4.2.z   
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1773909 (view as bug list) Environment:
Last Closed: 2020-01-23 11:12:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1773909    
Attachments:
Description Flags
installation dir
none
installation log
none
removal log from failed attempt
none
removal log from second successful attempt none

Description Filip Brychta 2019-11-14 12:48:26 UTC
Created attachment 1636125 [details]
installation dir

Description of problem:
First attempt to destroy cluster in OpenStack sometimes fail with:
level=fatal msg="Resource not found"
Second attempt works fine and the cluster is removed.

Following error is shown:
level=debug msg="Deleting Floating IP: d1f931dd-6df1-4f56-ae0c-8095d64e5a7a"
level=fatal msg="Resource not found"


Version-Release number of the following components:
OCP 4.2.2

How reproducible:
Sometimes

Steps to Reproduce:
1. Install OCP 4.2.2 in OpenStack: openshift-install --log-level debug create cluster --dir /home/jenkins/workspace/install-ocp4-2-plus/openshift/cluster-installation/psi/fbr-42-t1
2. wait for installation to finish
3. try to destroy the cluster: openshift-install --log-level debug destroy cluster --dir /home/jenkins/workspace/destroy-ocp4-2-plus/openshift/cluster-installation/psi/fbr-42-t1

Actual results:
Removal fails.
Full logs attached but here is the error msg:
level=debug msg="Deleting Floating IP: d1f931dd-6df1-4f56-ae0c-8095d64e5a7a"
level=fatal msg="Resource not found"

Expected results:
Cluster is removed without any errors.

Additional info:
Also saw following error from different cluster:
level=debug msg="Deleting container: ocp-jqe2-lzc2g-image-registry-ptmfjgligviifudscepaghxhcthtggmu\n"
level=fatal msg="Resource not found"

I don't know if the err:
level=fatal msg="Resource not found" 
is related to previous message:
level=debug msg="Deleting Floating IP: d1f931dd-6df1-4f56-ae0c-8095d64e5a7a"
but if yes, I can see in install log that this IP was created by installer:
level=debug msg="module.topology.openstack_networking_floatingip_associate_v2.api_fip[0]: Creation complete after 5s [id=d1f931dd-6df1-4f56-ae0c-8095d64e5a7a]"

Full debug log from installation, from failed removal and from second successful removal are attached. Installation dir is attached too.

Comment 1 Filip Brychta 2019-11-14 12:49:01 UTC
Created attachment 1636126 [details]
installation log

Comment 2 Filip Brychta 2019-11-14 12:49:29 UTC
Created attachment 1636127 [details]
removal log from failed attempt

Comment 3 Filip Brychta 2019-11-14 12:50:09 UTC
Created attachment 1636128 [details]
removal log from second successful attempt

Comment 4 Martin André 2019-11-18 14:53:04 UTC
We've fixed a number of issues with the destroy command on Openstack platform in the master branch and I'm confident this is fixed in the 4.3 builds. We should consider backporting those fixes to 4.2.z.

Comment 5 Martin André 2019-11-18 14:57:11 UTC
*** Bug 1773294 has been marked as a duplicate of this bug. ***

Comment 6 Martin André 2019-11-18 15:45:55 UTC
This should have been fixed with https://github.com/openshift/installer/pull/2408

Targeting the bug to 4.3.0 so that QE can confirm the issue is fixed and we'll backport to 4.2.z after cloning the BZ.

Comment 7 Mike Fedosin 2019-11-21 12:36:14 UTC
https://github.com/openshift/installer/pull/2408 doesn't completely solve the issue and therefore https://github.com/openshift/installer/pull/2561 is also required

Comment 8 David Sanz 2019-12-02 12:00:09 UTC
Verified on 4.3.0-0.nightly-2019-12-02-055401

Comment 10 errata-xmlrpc 2020-01-23 11:12:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062