Description of problem:
The volume created by aws ebs csi driver can not be deleted when the cluster is destroyed
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Deploy aws ebs csi driver with the operator
2. Create a PVC and Pod using ebs.csi.aws.com provisioner(aws ebs csi driver)
3. Create a PVC and Pod using kubernetes.io/aws-ebs provisioner(in-tree plug-in)
4. Destroy the cluster
The volume created by the kubernetes.io/aws-ebs provisioner can be deleted when the cluster is destroyed.
The volume created by the ebs.csi.aws.com provisioner can not be deleted.
The volume created by the ebs.csi.aws.com provisioner should be deleted.
Node Log (of failed PODs):
StorageClass Dump (if StorageClass used by PV/PVC):
Found the volume created by in-tree plug-in is deleted in the installer log.
level=info msg=Deleted arn="arn:aws:ec2:us-east-2:301721915996:volume/vol-0de4f771de91394ef" id=vol-0de4f771de91394ef
Checked the volume tags from the amazon web console, they are different.
vol-07d8c287ced08f952(aws ebs csi driver) CSIVolumeName:pvc-4ec09014-19cf-430e-83b5-ccf4317ba956
> The volume created by the ebs.csi.aws.com provisioner can not be deleted.
What does it mean? What blocks the deletion? What error message does it show?
In-tree provisioner creates volumes with these tags:
The first tag seems to be the most important.
The current version of AWS EBS CSI driver creates only this tag:
I tried to pass "--extra-volume-tags=kubernetes.io/cluster/<cluster id>=owned" to the CSI driver, however, this gets blocked by the driver with "Invalid driver options: Invalid extra volume tags: Volume tag key prefix 'kubernetes.io' is reserved". I need to fix the driver first.
Upstream PR to fix the driver part: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/530
The driver operator still needs to be fixed to pass the --cluster-id to the driver!
PR openshift/aws-ebs-csi-driver-operator/pull/83 waiting for review. Once it's OK, I'll submit the library-go changes to its repo.
Also, I'd like to start a discussion about how we're going to solve this problem in other CSI drivers.
Should this behavior (volumes deleted when the cluster is deleted) be the same for all volumes created by the CSI drivers shipped with OpenShift?
What's the current behavior with oVirt CSI Driver volumes?
Apparently, non-attached volumes created by oVirt CSI driver are NOT deleted when the cluster is destroyed (CC @firstname.lastname@example.org). Created a ticket here: https://bugzilla.redhat.com/show_bug.cgi?id=1874065
Manila ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1820238
Moving back to ASSIGNED until we discuss what's the right approach for all CSI drivers.
This requires changes in the CSI driver (done), in library-go and in the AWS EBS CSI Driver Operator.
The library-go patch is here: https://github.com/openshift/library-go/pull/909
Once that's merged we need to merge the operator patch here: https://github.com/openshift/aws-ebs-csi-driver-operator/pull/83
Verified with: 4.7.0-0.nightly-2020-10-17-034503
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.