Bug 1802467 - Could not delete volumesnapshotcontent if volumesnapshotclass policy is Retain
Summary: Could not delete volumesnapshotcontent if volumesnapshotclass policy is Retain
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Jan Safranek
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-13 08:41 UTC by Chao Yang
Modified: 2020-07-13 17:15 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:14:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift csi-external-snapshotter pull 15 0 None closed Bug 1802467: Fix deletion of snapshots with Retain policy 2020-07-01 09:12:26 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:15:14 UTC

Description Chao Yang 2020-02-13 08:41:23 UTC
Description of problem:
Could not delete volumesnapshotcontent if volumesnapshotclass policy is Retain

Version-Release number of selected component (if applicable):
version   4.4.0-0.nightly-2020-02-11-200858   True        False         24h     Cluster version is 4.4.0-0.nightly-2020-02-11-200858

How reproducible:
Always

Steps to Reproduce:
1.Create csi pvc/pod
2.Create volumesnapshotclass with policy Retain
3.Create volumesnapshot
4.Check volumesnapshotcontent is created
5.Delete volumesnapshot
oc delete volumesnapshot --all
volumesnapshot.snapshot.storage.k8s.io "new-snapshot-test" deleted
6.oc delete volumesnapshotcontent snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7


oc delete volumesnapshotcontent snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7 --loglevel=9
I0213 15:12:39.150076   18678 loader.go:375] Config loaded from file:  /home/chaoyang/kubeconfig
I0213 15:12:39.157780   18678 request.go:1017] Request Body: {"propagationPolicy":"Background"}
I0213 15:12:39.157847   18678 round_trippers.go:423] curl -k -v -XDELETE  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: oc/v4.4.0 (linux/amd64) kubernetes/7097318" 'https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7'
I0213 15:12:39.907047   18678 round_trippers.go:443] DELETE https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7 200 OK in 749 milliseconds
I0213 15:12:39.907078   18678 round_trippers.go:449] Response Headers:
I0213 15:12:39.907085   18678 round_trippers.go:452]     Content-Length: 1055
I0213 15:12:39.907090   18678 round_trippers.go:452]     Date: Thu, 13 Feb 2020 07:12:39 GMT
I0213 15:12:39.907094   18678 round_trippers.go:452]     Audit-Id: e2760d2a-c8d9-404f-aecc-31b9ac9f1e95
I0213 15:12:39.907101   18678 round_trippers.go:452]     Content-Type: application/json
I0213 15:12:39.907132   18678 request.go:1017] Response Body: {"apiVersion":"snapshot.storage.k8s.io/v1beta1","kind":"VolumeSnapshotContent","metadata":{"creationTimestamp":"2020-02-13T06:07:11Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2020-02-13T06:10:17Z","finalizers":["snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection"],"generation":2,"name":"snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7","resourceVersion":"624813","selfLink":"/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7","uid":"b2216651-6229-45ac-8c8d-97f669339e4c"},"spec":{"deletionPolicy":"Retain","driver":"ebs.csi.aws.com","source":{"volumeHandle":"vol-006c8773352ad03ec"},"volumeSnapshotClassName":"ebs-snap","volumeSnapshotRef":{"apiVersion":"snapshot.storage.k8s.io/v1beta1","kind":"VolumeSnapshot","name":"new-snapshot-test","namespace":"333","resourceVersion":"623717","uid":"64c6a4e1-11b7-492f-bfbe-33dddbf493c7"}},"status":{"creationTime":1581574031000000000,"readyToUse":true,"restoreSize":2147483648,"snapshotHandle":"snap-09cd4ce29836c6a99"}}
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7" deleted
I0213 15:12:39.907394   18678 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: oc/v4.4.0 (linux/amd64) kubernetes/7097318" 'https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents?fieldSelector=metadata.name%3Dsnapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7'
I0213 15:12:40.156323   18678 round_trippers.go:443] GET https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents?fieldSelector=metadata.name%3Dsnapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7 200 OK in 248 milliseconds
I0213 15:12:40.156349   18678 round_trippers.go:449] Response Headers:
I0213 15:12:40.156357   18678 round_trippers.go:452]     Date: Thu, 13 Feb 2020 07:12:40 GMT
I0213 15:12:40.156365   18678 round_trippers.go:452]     Audit-Id: 6c158dc9-77b0-4505-8927-d080081034d2
I0213 15:12:40.156371   18678 round_trippers.go:452]     Content-Type: application/json
I0213 15:12:40.156379   18678 round_trippers.go:452]     Content-Length: 1277
I0213 15:12:40.156413   18678 request.go:1017] Response Body: {"apiVersion":"snapshot.storage.k8s.io/v1beta1","items":[{"apiVersion":"snapshot.storage.k8s.io/v1beta1","kind":"VolumeSnapshotContent","metadata":{"creationTimestamp":"2020-02-13T06:07:11Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2020-02-13T06:10:17Z","finalizers":["snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection"],"generation":2,"name":"snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7","resourceVersion":"624813","selfLink":"/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7","uid":"b2216651-6229-45ac-8c8d-97f669339e4c"},"spec":{"deletionPolicy":"Retain","driver":"ebs.csi.aws.com","source":{"volumeHandle":"vol-006c8773352ad03ec"},"volumeSnapshotClassName":"ebs-snap","volumeSnapshotRef":{"apiVersion":"snapshot.storage.k8s.io/v1beta1","kind":"VolumeSnapshot","name":"new-snapshot-test","namespace":"333","resourceVersion":"623717","uid":"64c6a4e1-11b7-492f-bfbe-33dddbf493c7"}},"status":{"creationTime":1581574031000000000,"readyToUse":true,"restoreSize":2147483648,"snapshotHandle":"snap-09cd4ce29836c6a99"}}],"kind":"VolumeSnapshotContentList","metadata":{"continue":"","resourceVersion":"646226","selfLink":"/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents"}}
I0213 15:12:40.156706   18678 round_trippers.go:423] curl -k -v -XGET  -H "User-Agent: oc/v4.4.0 (linux/amd64) kubernetes/7097318" -H "Accept: application/json" 'https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents?fieldSelector=metadata.name%3Dsnapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7&resourceVersion=646226&watch=true'
I0213 15:12:40.401969   18678 round_trippers.go:443] GET https://api.chaoyang-new.qe.devcluster.openshift.com:6443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents?fieldSelector=metadata.name%3Dsnapcontent-64c6a4e1-11b7-492f-bfbe-33dddbf493c7&resourceVersion=646226&watch=true 200 OK in 245 milliseconds
I0213 15:12:40.401998   18678 round_trippers.go:449] Response Headers:
I0213 15:12:40.402027   18678 round_trippers.go:452]     Content-Type: application/json
I0213 15:12:40.402033   18678 round_trippers.go:452]     Date: Thu, 13 Feb 2020 07:12:40 GMT


Actual results:
Volumesnapshotcontent is not deleted

Expected results:
Volumesnapshotcontent should be deleted

Master Log:
Some logs from controller
I0213 05:44:55.236647       1 snapshot_controller.go:209] syncSnapshot: VolumeSnapshot new-snapshot-test is bound to volumeSnapshotContent [snapcontent-9d93c33a-4305-4992-a02d-20c9ab08ee26]
I0213 05:53:29.098780       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:53:29.098784       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:53:29.098824       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:53:29.098946       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:57:41.547726       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:57:41.547719       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:57:41.547784       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
I0213 05:57:41.547882       1 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Tomas Smetana 2020-02-13 10:19:58 UTC
This is expected behaviour I'd say:

https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/

"If the deletionPolicy is Retain, then both the underlying snapshot and VolumeSnapshotContent remain."

Comment 2 Tomas Smetana 2020-02-13 11:17:05 UTC
Reopening. Sorry for the noise... The object itself should be deletable.

Comment 7 Chao Yang 2020-03-23 09:03:12 UTC
Passed on 4.5.0-0.nightly-2020-03-22-211241
oc get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT      AGE
snapcontent-7cd4d9f6-d289-4cb2-837a-5b8edfcf25cf   true         2147483648    Delete           ebs.csi.aws.com   ebs-snap              new-snapshot-test   128m
snapcontent-e7d1938c-e7b2-4199-87d1-4859b4d8636b   true         2147483648    Retain           ebs.csi.aws.com   ebs-snap              new-snapshot-test   88s
# oc delete volumesnapshotcontent snapcontent-e7d1938c-e7b2-4199-87d1-4859b4d8636b
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-e7d1938c-e7b2-4199-87d1-4859b4d8636b" deleted
# oc get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT      AGE
snapcontent-7cd4d9f6-d289-4cb2-837a-5b8edfcf25cf   true         2147483648    Delete           ebs.csi.aws.com   ebs-snap              new-snapshot-test   128m

Comment 9 errata-xmlrpc 2020-07-13 17:14:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.