Bug 1840759
Summary: | [aws-ebs-csi-driver] The volume created by aws ebs csi driver can not be deleted when the cluster is destroyed | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Qin Ping <piqin> |
Component: | Storage | Assignee: | Fabio Bertinatto <fbertina> |
Storage sub component: | Operators | QA Contact: | Qin Ping <piqin> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | medium | ||
Priority: | medium | CC: | aos-bugs, fbertina, jsafrane |
Version: | 4.5 | ||
Target Milestone: | --- | ||
Target Release: | 4.7.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | No Doc Update | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-02-24 15:12:15 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Qin Ping
2020-05-27 14:46:23 UTC
> The volume created by the ebs.csi.aws.com provisioner can not be deleted.
What does it mean? What blocks the deletion? What error message does it show?
In-tree provisioner creates volumes with these tags: kubernetes.io/cluster/jsafrane-10235-vgdv6: owned Name: jsafrane-10235-vgdv6-dynamic-pvc-447cc711-bb65-4b4d-836d-a822e4e77e43 kubernetes.io/created-for/pv/name: pvc-447cc711-bb65-4b4d-836d-a822e4e77e43 kubernetes.io/created-for/pvc/name: myclaim kubernetes.io/created-for/pvc/namespace: default The first tag seems to be the most important. The current version of AWS EBS CSI driver creates only this tag: CSIVolumeName: pvc-4e4cb311-3907-4192-bc58-cde8ea112392 I tried to pass "--extra-volume-tags=kubernetes.io/cluster/<cluster id>=owned" to the CSI driver, however, this gets blocked by the driver with "Invalid driver options: Invalid extra volume tags: Volume tag key prefix 'kubernetes.io' is reserved". I need to fix the driver first. Upstream PR to fix the driver part: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/530 The driver operator still needs to be fixed to pass the --cluster-id to the driver! PR openshift/aws-ebs-csi-driver-operator/pull/83 waiting for review. Once it's OK, I'll submit the library-go changes to its repo. Also, I'd like to start a discussion about how we're going to solve this problem in other CSI drivers. Should this behavior (volumes deleted when the cluster is deleted) be the same for all volumes created by the CSI drivers shipped with OpenShift? What's the current behavior with oVirt CSI Driver volumes? Apparently, non-attached volumes created by oVirt CSI driver are NOT deleted when the cluster is destroyed (CC @bzlotnik). Created a ticket here: https://bugzilla.redhat.com/show_bug.cgi?id=1874065 Manila ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1820238 Moving back to ASSIGNED until we discuss what's the right approach for all CSI drivers. This requires changes in the CSI driver (done), in library-go and in the AWS EBS CSI Driver Operator. The library-go patch is here: https://github.com/openshift/library-go/pull/909 Once that's merged we need to merge the operator patch here: https://github.com/openshift/aws-ebs-csi-driver-operator/pull/83 Verified with: 4.7.0-0.nightly-2020-10-17-034503 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |