Bug 1902711

Summary: Tracker for Bug #1903078 Deleting VolumeSnapshotClass makes VolumeSnapshot not Ready
Product: Red Hat OpenShift Container Storage Reporter: Jilju Joy <jijoy>
Component: csi-driverAssignee: Humble Chirammal <hchiramm>
Status: CLOSED ERRATA QA Contact: Jilju Joy <jijoy>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.6CC: asriram, dwalveka, edonnell, hchiramm, madam, mrajanna, muagarwa, nberry, ocs-bugs, tdesala
Target Milestone: ---Keywords: Tracking
Target Release: OCS 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
.You cannot create a PVC from a volume snapshot in the absence of `volumesnapshotclass` A PVC can not be created from a volume snapshot in the absence of `volumesnapshotclass`. This issue is caused because the status of the volume snapshot changes to a *not ready* state on deleting the `volumesnapshotclass`. This issue has been fixed in OCP 4.7.0 and higher.
Story Points: ---
Clone Of:
: 1903078 (view as bug list) Environment:
Last Closed: 2021-05-19 09:16:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1882359, 1903078, 1938134    

Description Jilju Joy 2020-11-30 13:32:06 UTC
Description of problem (please be detailed as possible and provide log
snippests):
Volume snapshot status in UI turns to 'Error' after deleting the volume snapshot class. In the yaml output, the value of 'readyToUse' parameter became false. Volume snapshot content remains in Ready state.

$ oc describe volumesnapshot pvccephfs-snapshot -n test-snapshot-delete
Name:         pvccephfs-snapshot
Namespace:    test-snapshot-delete
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-11-30T11:27:34Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:      Mozilla
    Operation:    Update
    Time:         2020-11-30T11:27:34Z
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:status:
        .:
        f:boundVolumeSnapshotContentName:
        f:creationTime:
        f:error:
          .:
          f:message:
          f:time:
        f:readyToUse:
        f:restoreSize:
    Manager:         snapshot-controller
    Operation:       Update
    Time:            2020-11-30T11:33:24Z
  Resource Version:  174170
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/namespaces/test-snapshot-delete/volumesnapshots/pvccephfs-snapshot
  UID:               6c38f430-3202-435b-b464-bb7f90476761
Spec:
  Source:
    Persistent Volume Claim Name:  pvccephfs
  Volume Snapshot Class Name:      ocs-storagecluster-cephfsplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-6c38f430-3202-435b-b464-bb7f90476761
  Creation Time:                       2020-11-30T11:27:36Z
  Error:
    Message:     Failed to get snapshot class with error volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
    Time:        2020-11-30T11:33:24Z
  Ready To Use:  false
  Restore Size:  5Gi
Events:
  Type     Reason                  Age   From                 Message
  ----     ------                  ----  ----                 -------
  Normal   CreatingSnapshot        21m   snapshot-controller  Waiting for a snapshot test-snapshot-delete/pvccephfs-snapshot to be created by the CSI driver.
  Warning  GetSnapshotClassFailed  15m   snapshot-controller  Failed to get snapshot class with error volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found



$ oc describe volumesnapshot pvcrbd-snapshot -n test-snapshot-delete
Name:         pvcrbd-snapshot
Namespace:    test-snapshot-delete
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-11-30T11:28:06Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:      Mozilla
    Operation:    Update
    Time:         2020-11-30T11:28:06Z
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:status:
        .:
        f:boundVolumeSnapshotContentName:
        f:creationTime:
        f:error:
          .:
          f:message:
          f:time:
        f:readyToUse:
        f:restoreSize:
    Manager:         snapshot-controller
    Operation:       Update
    Time:            2020-11-30T11:33:24Z
  Resource Version:  174169
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/namespaces/test-snapshot-delete/volumesnapshots/pvcrbd-snapshot
  UID:               6830107d-03cb-4547-9939-4e346c63f2fb
Spec:
  Source:
    Persistent Volume Claim Name:  pvcrbd
  Volume Snapshot Class Name:      ocs-storagecluster-rbdplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-6830107d-03cb-4547-9939-4e346c63f2fb
  Creation Time:                       2020-11-30T11:28:07Z
  Error:
    Message:     Failed to get snapshot class with error volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
    Time:        2020-11-30T11:33:24Z
  Ready To Use:  false
  Restore Size:  5Gi
Events:
  Type     Reason                  Age   From                 Message
  ----     ------                  ----  ----                 -------
  Normal   CreatingSnapshot        21m   snapshot-controller  Waiting for a snapshot test-snapshot-delete/pvcrbd-snapshot to be created by the CSI driver.
  Warning  GetSnapshotClassFailed  16m   snapshot-controller  Failed to get snapshot class with error volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found



$ oc -n test-snapshot-delete get volumesnapshot
NAME                 READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                               SNAPSHOTCONTENT                                    CREATIONTIME   AGE
pvccephfs-snapshot   false        pvccephfs                           5Gi           ocs-storagecluster-cephfsplugin-snapclass   snapcontent-6c38f430-3202-435b-b464-bb7f90476761   6m37s          6m39s
pvcrbd-snapshot      false        pvcrbd                              5Gi           ocs-storagecluster-rbdplugin-snapclass      snapcontent-6830107d-03cb-4547-9939-4e346c63f2fb   6m6s           6m7s



$ oc get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                                  VOLUMESNAPSHOTCLASS                         VOLUMESNAPSHOT       AGE
snapcontent-6830107d-03cb-4547-9939-4e346c63f2fb   true         5368709120    Delete           openshift-storage.rbd.csi.ceph.com      ocs-storagecluster-rbdplugin-snapclass      pvcrbd-snapshot      6m34s
snapcontent-6c38f430-3202-435b-b464-bb7f90476761   true         5368709120    Delete           openshift-storage.cephfs.csi.ceph.com   ocs-storagecluster-cephfsplugin-snapclass   pvccephfs-snapshot   7m6s


$ oc get volumesnapshotclass
No resources found


Logs will be shared shortly.
==========================================================================
Version of all relevant components (if applicable):
ocs-operator.v4.6.0-160.ci
Cluster version is 4.7.0-0.nightly-2020-11-29-133728
ceph version 14.2.8-111.el8cp (2e6029d57bc594eceba4751373da6505028c2650) nautilus (stable)

rook: 4.6-73.15d47331.release_4.6

Snapshotter Version: v4.6.0-202011041933.p0-0-gef587a1-dirty

CSI Driver version: release-4.6 and Git version: 9ae5f5cee7774048f65fdc48e07115e1c4f7b502


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Cannot delete volume snapshot


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes


Can this issue reproduce from the UI?
Yes


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create RBD or CephFs PVC and create a snapshot from it. Wait until the volume snapshot is Ready.
2. Delete the volume snapshot class used to create the above volume snapshot.
 (If using the default volume snapshot class, update reconcileStrategy of 
  snapshotClasses to 'ignore' in the storagecluster to avoid reconciling )
3. Check the status of volume snapshot.


Actual results:
Snapshot is not Ready


Expected results:
Volume snapshot should remain in Ready state.


Additional info:

Comment 6 Jilju Joy 2020-12-07 07:41:57 UTC
This title "You cannot create PVC or clones from `snapshot`" should be changed. This title in release notes will give an impression that a PVC cannot be created from snapshot. This is applicable only in a situation where the snapshot class is not present.
Why do we need to mention about clone here ?

Comment 8 Mudit Agarwal 2020-12-07 08:06:43 UTC
Changing the doc_text back from '+' to '?' as the content is changed. Erin, PTAL

Thanks Jiliju/Madhu.

Comment 9 Neha Berry 2020-12-08 15:12:49 UTC
Hi Erin

I added the known issue statement as the current one is very misleading and people may think snapshot restore does not work at all

You cannot restore PVC from `snapshot` 

to 

You cannot restore PVC from `snapshot` in absence of volume snapshotclass

Comment 10 Mudit Agarwal 2021-01-07 12:34:59 UTC
OCP BZ is already verified, moving this one to ON_QA

Comment 13 Jilju Joy 2021-01-15 10:09:08 UTC
Verified in version:
OCP 4.7.0-0.nightly-2021-01-14-211319
ocs-operator.v4.7.0-230.ci


List of Cephfs and RBD volume snapshots
$ oc -n namespace-test-e9c5276f746b419fae7eeb463 get volumesnapshot
NAME                                                READYTOUSE   SOURCEPVC                                  SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT                                    CREATIONTIME   AGE
pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   true         pvc-test-5b560b0d48a2447ebf69fdb91eda7a4                           3Gi           cephfssnapshotclass   snapcontent-86391466-211a-487f-aa27-141910a5f50d   70s            72s
pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   true         pvc-test-6aade12d84234b6f8e88649b0f9f41a                           3Gi           rbdsnapshotclass      snapcontent-83ba7737-569a-4049-a868-134bb6a1b637   52s            53s


Delete the volume snapshot classes
$ oc delete volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
volumesnapshotclass.snapshot.storage.k8s.io "cephfssnapshotclass" deleted
volumesnapshotclass.snapshot.storage.k8s.io "rbdsnapshotclass" deleted


Verify the volume snapshotclasses are deleted
$ oc get volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "cephfssnapshotclass" not found
Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "rbdsnapshotclass" not found


List of Cephfs and RBD volume snapshots. Snapshots are in Ready state.
$ oc -n namespace-test-e9c5276f746b419fae7eeb463 get volumesnapshot
NAME                                                READYTOUSE   SOURCEPVC                                  SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT                                    CREATIONTIME   AGE
pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   true         pvc-test-5b560b0d48a2447ebf69fdb91eda7a4                           3Gi           cephfssnapshotclass   snapcontent-86391466-211a-487f-aa27-141910a5f50d   2m29s          2m31s
pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   true         pvc-test-6aade12d84234b6f8e88649b0f9f41a                           3Gi           rbdsnapshotclass      snapcontent-83ba7737-569a-4049-a868-134bb6a1b637   2m11s          2m12s


Verify the status of volume snapshot contents. Snapshot contents are in Ready state.
$ oc get volumesnapshotcontent snapcontent-86391466-211a-487f-aa27-141910a5f50d snapcontent-83ba7737-569a-4049-a868-134bb6a1b637
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                                  VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT                                      AGE
snapcontent-86391466-211a-487f-aa27-141910a5f50d   true         3221225472    Delete           openshift-storage.cephfs.csi.ceph.com   cephfssnapshotclass   pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   3m21s
snapcontent-83ba7737-569a-4049-a868-134bb6a1b637   true         3221225472    Delete           openshift-storage.rbd.csi.ceph.com      rbdsnapshotclass      pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   3m3s


Verify volume snapshot class are still absent.
$ oc get volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "cephfssnapshotclass" not found
Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io "rbdsnapshotclass" not found

Comment 14 Jilju Joy 2021-01-15 10:13:14 UTC
(In reply to Jilju Joy from comment #13)
> Verified in version:
> OCP 4.7.0-0.nightly-2021-01-14-211319
> ocs-operator.v4.7.0-230.ci
> 
> 
> List of Cephfs and RBD volume snapshots
> $ oc -n namespace-test-e9c5276f746b419fae7eeb463 get volumesnapshot
> NAME                                                READYTOUSE   SOURCEPVC  
> SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT 
> CREATIONTIME   AGE
> pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   true        
> pvc-test-5b560b0d48a2447ebf69fdb91eda7a4                           3Gi      
> cephfssnapshotclass   snapcontent-86391466-211a-487f-aa27-141910a5f50d   70s
> 72s
> pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   true        
> pvc-test-6aade12d84234b6f8e88649b0f9f41a                           3Gi      
> rbdsnapshotclass      snapcontent-83ba7737-569a-4049-a868-134bb6a1b637   52s
> 53s
> 
> 
> Delete the volume snapshot classes
> $ oc delete volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
> volumesnapshotclass.snapshot.storage.k8s.io "cephfssnapshotclass" deleted
> volumesnapshotclass.snapshot.storage.k8s.io "rbdsnapshotclass" deleted
> 
> 
> Verify the volume snapshotclasses are deleted
> $ oc get volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
> Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io
> "cephfssnapshotclass" not found
> Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io
> "rbdsnapshotclass" not found
> 
> 
> List of Cephfs and RBD volume snapshots. Snapshots are in Ready state.
> $ oc -n namespace-test-e9c5276f746b419fae7eeb463 get volumesnapshot
> NAME                                                READYTOUSE   SOURCEPVC  
> SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT 
> CREATIONTIME   AGE
> pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   true        
> pvc-test-5b560b0d48a2447ebf69fdb91eda7a4                           3Gi      
> cephfssnapshotclass   snapcontent-86391466-211a-487f-aa27-141910a5f50d  
> 2m29s          2m31s
> pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   true        
> pvc-test-6aade12d84234b6f8e88649b0f9f41a                           3Gi      
> rbdsnapshotclass      snapcontent-83ba7737-569a-4049-a868-134bb6a1b637  
> 2m11s          2m12s
> 
> 
> Verify the status of volume snapshot contents. Snapshot contents are in
> Ready state.
> $ oc get volumesnapshotcontent
> snapcontent-86391466-211a-487f-aa27-141910a5f50d
> snapcontent-83ba7737-569a-4049-a868-134bb6a1b637
> NAME                                               READYTOUSE   RESTORESIZE 
> DELETIONPOLICY   DRIVER                                  VOLUMESNAPSHOTCLASS
> VOLUMESNAPSHOT                                      AGE
> snapcontent-86391466-211a-487f-aa27-141910a5f50d   true         3221225472  
> Delete           openshift-storage.cephfs.csi.ceph.com   cephfssnapshotclass
> pvc-test-5b560b0d48a2447ebf69fdb91eda7a4-snapshot   3m21s
> snapcontent-83ba7737-569a-4049-a868-134bb6a1b637   true         3221225472  
> Delete           openshift-storage.rbd.csi.ceph.com      rbdsnapshotclass   
> pvc-test-6aade12d84234b6f8e88649b0f9f41a-snapshot   3m3s
> 
> 
> Verify volume snapshot class are still absent.
> $ oc get volumesnapshotclass cephfssnapshotclass rbdsnapshotclass
> Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io
> "cephfssnapshotclass" not found
> Error from server (NotFound): volumesnapshotclasses.snapshot.storage.k8s.io
> "rbdsnapshotclass" not found

Snapshot restore succeeded in the absence of volume snapshotclass.

Comment 20 Humble Chirammal 2021-04-05 08:28:17 UTC
>> This issue is caused because the status of the volume snapshot changes to
> Error on deleting the volumesnapshotclass.

Disha, the status of volumesnapshot changes to "not ready" in this case and not really "error" state. so, that part of the text can be corrected. also, better avoid space between `volume` and `snapshot` above.

Comment 21 Disha Walvekar 2021-04-05 08:40:52 UTC
(In reply to Humble Chirammal from comment #20)
> >> This issue is caused because the status of the volume snapshot changes to
> > Error on deleting the volumesnapshotclass.
> 
> Disha, the status of volumesnapshot changes to "not ready" in this case and
> not really "error" state. so, that part of the text can be corrected. also,
> better avoid space between `volume` and `snapshot` above.

Thank you for the inputs, Humble. Changed to 

A PVC can not be created from a volume snapshot in the absence of volumesnapshotclass.
This issue is caused because the status of the volume snapshot changes to a `not ready` state on deleting the volumesnapshotclass.
To work around this issue, recreate the volumesnapshotclass.

Comment 30 errata-xmlrpc 2021-05-19 09:16:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2041