Bug 1876935

Summary: AWS volume snapshot is not deleted after the cluster is destroyed
Product: OpenShift Container Platform Reporter: Qin Ping <piqin>
Component: StorageAssignee: Fabio Bertinatto <fbertina>
Storage sub component: Kubernetes External Components QA Contact: Wei Duan <wduan>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: medium CC: aos-bugs, fbertina, hekumar, jsafrane
Version: 4.6   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:17:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Qin Ping 2020-09-08 14:39:18 UTC
Description of problem:
volume snapshot is not deleted after the cluster is destroyed

Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-09-07-210448

How reproducible:
Always

Steps to Reproduce:
1. Launched a 4.6 OCP cluster on AWS
2. Created a volumesnapshot for a PVC
3. Didn't delete the volumesnapshotcontent
4. Destroy cluster

Actual results:
Checked from the AWS UI, the volume snapshot is not deleted.

Expected results:
Deleted the volume snapshot when cluster is destroyed.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 4 Jan Safranek 2020-09-09 13:40:57 UTC
We need to take some holistic approach what to do with volumes / snapshots when a cluster is destroyed. Volumes of some in-tree volume plugins are destroyed (AWS), some are not (vSphere?).

Comment 5 Jan Safranek 2020-09-17 15:53:57 UTC
Result: 'openshift-install destroy cluster' should delete volumes & snapshots provisioned in the cluster.

Therefore we need:
1. The CSI driver to label snapshots accordingly
2. openshift-install to list & delete them (this may be already implemented).

Comment 6 Fabio Bertinatto 2020-10-01 09:41:07 UTC
CSI Driver patch (in review): https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/568

The library-go patch (in review): https://github.com/openshift/library-go/pull/909

Once those are merged we need to merge the operator patch here: https://github.com/openshift/aws-ebs-csi-driver-operator/pull/83

Comment 9 Wei Duan 2020-10-21 04:38:48 UTC
Please ignore my previous comments (https://bugzilla.redhat.com/show_bug.cgi?id=1876935#c8), I mixed with another bug.
Verified pass on 4.7.0-0.nightly-2020-10-17-034503, after deleteing the cluster, the previously created snapshot on aws UI are removed accordingly, changed the status to VERIFIED.

Comment 12 errata-xmlrpc 2021-02-24 15:17:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633