Bug 1746459 - destroy-cluster sometimes fails to delete AWS snapshot
Summary: destroy-cluster sometimes fails to delete AWS snapshot
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.2.0
Assignee: Matthew Staebler
QA Contact: Mike Fiedler
Depends On:
TreeView+ depends on / blocked
Reported: 2019-08-28 14:01 UTC by Matthew Staebler
Modified: 2019-10-16 06:40 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-10-16 06:38:14 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift installer pull 2284 0 None None None 2019-08-28 16:29:28 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:40:23 UTC

Description Matthew Staebler 2019-08-28 14:01:24 UTC
When destroying a cluster, the installer sometimes gets stuck trying to delete a snapshot. I have seen this happen on two separate occasions. In both cases, the installer completed the terraform apply during create-cluster.

This is a snippet of what gets repeated in the installer logs.

time="2019-08-28T13:48:36Z" level=debug msg="search for IAM roles" installID=mfs5wxtd
time="2019-08-28T13:48:36Z" level=debug msg="search for IAM users" installID=mfs5wxtd
time="2019-08-28T13:48:46Z" level=debug msg="search for and delete matching resources by tag in us-east-1 matching aws.Filter{\"kubernetes.io/cluster/ci-cluster-v4-1-vdgsk\":\"owned\"}" installID=mfs5wxtd
time="2019-08-28T13:48:46Z" level=debug msg="InvalidSnapshot.NotFound: \n\tstatus code: 400, request id: 9c2e706d-faec-4272-8f56-e683c882397a" arn="arn:aws:ec2:us-east-1:462175581547:snapshot/snap-0e711a4f51bdce4be" installID=mfs5wxtd

Version-Release number of selected component (if applicable):

Comment 1 Devan Goodwin 2019-08-28 14:53:50 UTC
Hitting this as well today, the snapshot in question does not appear to exist in the AWS UI. Unclear how the filtering can be finding it indefinitely.

Comment 2 Naveen Malik 2019-08-28 18:24:55 UTC
Same problem.  There are no snapshots in us-east-1 and it seems to find something and loop for 2+ hours at this point trying to delete it:

time="2019-08-28T18:18:17Z" level=debug msg="search for and delete matching resources by tag in us-east-1 matching aws.Filter{\"kubernetes.io/cluster/nmalik2-sm54x\":\"owned\"}"
time="2019-08-28T18:18:17Z" level=debug msg="InvalidSnapshot.NotFound: \n\tstatus code: 400, request id: 552fc3fe-1369-40f2-9cc1-2e2de111cc7d" arn="arn:aws:ec2:us-east-1:278307472902:snapshot/snap-020a90089e802252c"

Comment 4 Mike Fiedler 2019-09-12 01:14:38 UTC
Verified on 4.2.0-0.nightly-2019-09-10-074025

Installed, deleted master AMI, deleted master AMI snapshot, destroyed
Installed, deleted master AMI, destroyed while deleting AMI snapshot
Installed, destroyed without touching AMI or snapshot

All cluster destroys were successful.

Comment 5 errata-xmlrpc 2019-10-16 06:38:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.